software testing services
Rex Black Consulting Services | software testing experts providing consulting, outsourcing, and software training
CALL US TODAY
(866) 438-4830
ISTQB certification testingISTQB certification testing ISTQB certification testing
PMI

Archive for the ‘software test skills’ Category

Interviewing Software Testers

I spend a lot of time traveling the world and talking to IT executives, managers, and staff members.  Even with the economy struggling, a surprising number of managers are hiring new staff.  I did a webinar on hiring in November, and over 300 people registered to attend.  You can listen to the recorded version of the webinar here.

Hiring is great, and it’s sure better to hire than to fire.  However, with hiring comes the possibility of the dreaded hiring mistake, the opening scene of one of the manager’s worst nightmares.  Since interviews are so important in selecting the right people to hire—and, ideally, deflecting the hiring mistakes—you need to be able to conduct effective interviews.  What are the important elements of an effective interviewing process?

One element is a good job description.  In addition to the obvious sections of this document, it should answer for following questions:

  • What tasks and responsibilities are involved?
  • What and how much experience?
  • What specific skills are needed, and what is the career path?
  • What training, education, certification, or licenses are required?
  • What is the start date?
  • If unusual, what are the hours, the dress code, and the travel requirements?

The first four of these sections are easy to complete if you have done a task analysis and skills inventory for your team.  Finally, avoid a classic worst-practice of job descriptions by distinguishing between required and desirable qualifications. 

With the job description in place, you can start interviewing candidates.  (I assume that your HR department will handle the actual recruiting activities that bring candidates and their resumes your way.)  To make the process efficient—to be blunt, to minimize wasted time on in-person interviews with unqualified candidates—I recommend using a phone interview (perhaps more accurately called a “phone screen”) to start.

This brings us to an important point that applies to the interview hiring process.  While we were all taught to be polite when growing up—and you should be polite in this entire process—we do need to turn off the politeness instinct that causes us to pull back and redirect questions when we sense that the other person is uncomfortable.  Remember, the objective of the hiring process is to hire the most qualified candidate, not to make all the interviewees completely comfortable. 

So, in the phone screen, you should explore the person’s experience and qualifications in a polite but incisive way.  In particular, weed out people who pad their resume or inflate their experience.  If a buzzword or acronym is on a resume, check that the candidate has meaningful mastery of the subject. Carefully evaluate all claimed expertise and experience in the phone interview.  Be especially skeptical if a skill is listed without any description of a particular job where the skill was applied.  Also, you may want to verify degrees, certifications, and licenses if these are important.  If the candidate passes the phone screen, then you can schedule an in-person interview with yourself and others on the team in which the person will work.  Key managers who will work with the candidate are often included.  Again, I assume your HR team can help you set up the interview participants and schedule.

In the in-person interview, include a mix of qualification questions, behavioral interviewing, and audition interviewing.  Qualification questions are those with correct and incorrect answers; e.g., “What programming language is primarily used to write the Linux operating system?” Pick skills and knowledge that relates to the actual work the successful candidate will perform, then develop a set of good qualification questions for them.  Don’t make these questions so hard that no one gets them right; the objective is not to pose the riddle of the sphinx to the candidates, but to measure their level of skill.

Behavioral interviewing is concerned, obviously enough, with how a person will behave on the job.  Behavioral questions are open-ended, and often require candidates to relate their past experience to the job you are considering them for. For example, here are three possible behavioral interview questions:

  • Tell me about ways that past managers have enabled you to do your best work.
  • How will what you learned on project XYZ help us here at our company?
  • Of all the jobs you’ve had, what was your most enjoyable job, and what did like the most?

Depending on the culture, workstyles and values of your company, the right answer for any of these questions could be the wrong answer for other companies.

You should also include audition interviews.  An audition interview is where you set up an actual work task—or a scaled down version of it—and ask the candidate to perform it.  For example, if you are hiring a test engineer, you could ask the candidate to create a test based on a requirements specification or user story from a past or current project.  As another example, if you are hiring a test technician, you could give the candidate a real test case–written to the expected level of detail as most test cases–and have the candidate test a real system.  While audition interviews might sound complicated, they’re actually easy and fun once you get the hang of them.  I have hired some really good people—and not hired some people I might otherwise have mistakenly hired—based on their audition interviews.

Let me conclude with some cautionary notes on avoiding classic, all-too-common interviewing mistakes.  One of these mistakes is scaring off the candidates.  This can happen when people deliberately intimidate, stress out, or just plain weird out candidates in interviews.  It can also happen when interviewers who are having a bad day vent their frustrations; you should be honest about the good and bad aspects of the company, but don’t painting a bleak and bitter picture.  Another classic mistake, which I mentioned earlier, is being afraid to ask tough questions.  Probe for weakness in the skills and experienced claimed in the resume and the interview.  Keep your ears open for fudging, vagueness, attempts to redirect the question, and using incorrect technical terms.  Another classic mistake is to break the law.  Be sure you know what questions and topics you can’t bring up.  Again, turn to your HR department to help you (and others who will be involved in interviewing) understand what topics are acceptable in an interview.

Successful teams are built by smart managers who have mastered the art of effective interviewing.  Effective interviewing starts with a good job description, as that document defines clear requirements for the position.  Effective interviewing should also include a good phone screen.  The interview process should include qualification questions, behavioral interviewing, and an audition interview. Don’t scare off the candidate or break the law, but do ask polite yet challenging questions. By including these essential elements of effective interviewing, you can be a smarter hiring manager, too.

McAfee Online Systems, Including Financials, Not Tested?

Some of  the readers of this blog have perhaps read or heard me say that the state of the common practice of software testing lags about 25 years behind the state of the art.  Here’s a situation which either is a case study in why I say that, or it’s something worse.

We (RBCS) used McAfee AntiVirus software for years on many of the PCs in our office as well as on company laptops.  This was mostly due to laziness, not a high level of satisfaction.  McAfee often came pre-installed on computers we bought, so we would simply renew the update subscriptions.  However, repeated problems with McAfee’s virus protection–especially its aggressive interference with e-mail programs–led us to switch to Trend Micro, which now runs on all our PCs.

On September 30, we received an e-mail from McAfee (specifically, an account called “subscriptions” at “mcafee.com”).  This e-mail read, in part, “ATTENTION: The credit card linked to your account for McAfee-based security products has expired. Please update your account now to keep your PC protected without interruption.”  After checking to make sure that we were no longer using McAfee software on any of our computers, we ignored the message.  After all, the credit card had expired.

It turns out that McAfee’s software had a bug in it that cause it to submit the charge anyway, because my October 13 American Express bill has a charge for $43.29 from McAfee.com.  No receipt was mailed for the charge, unlike in previous years, though I’m not sure this has anything to do with the expired card. We’re going to contest and reverse this charge, of course; it is a relatively minor hassle.

The testing implications of this situation are significant, though.  Remember, antivirus software, like other security software, is software that we rely on to keep systems safe.  Given the kind of damage a widespread interruption to computer systems due to virus attacks can cause, you’d expect the entire suite of software–including the online systems that provide updates and handle customer financial matters–to be well tested, following known best practices and techniques.

So, what’s the explanation for this situation?  If a credit card has expired, the system should not silently put through a charge on the card, especially when the system has sent an e-mail to the customer giving the impression that, unless the customer takes a specific and deliberate action to update the card, no charge will occur and the subscription will expire.   We have an obvious equivalence partition. Equivalence partitioning as a test technique is over 25 years old.  We clearly have a block of code that recognizes the equivalence partition and triggers an e-mail to the customer prior to the charge occuring.  Statement coverage, the lowest of the white box coverage criteria, is also a test technique that is over 25 years old.

In risk based testing–a topic I’ve covered often in this blog–it’s almost certain that two financial-related risks would be noted during quality risk identification:

  • Processing a credit card transaction that should be rejected for processing
  • Not processing a credit card transaction that should be accepted for processing

Best practices of risk based testing would generally lead to such risks given a high impact rating, even if the likelihood rating were low.  That would require more than cursory testing against such risks.  Now, analytical risk based testing as a strategy is relatively leading edge, having been perfected only in the last ten years.

Maybe someone at McAfee would care to post a comment about which of the following statements is true:

  1. McAfee software systems receive only cursory testing, with financial processing code tested to less than 100% equivalence partition coverage and less than 100% statement coverage.  McAfee’s test team was not aware that the system attempts charges against credit cards it knows to be no longer valid.
  2. McAfee’s test team was aware that the system attempts charges against credit cards it knows to be no longer valid.  A bug was reported against this behavior by the testers, and deliberately deferred or cancelled by the product or project manager because they decided that they wanted the additional revenue from former customers.

Of course, it’s quite possible that McAfee’s financial processing code is tested to less than 100% equivalence partition coverage and less than 100% statement coverage, but even so their testers found this bug.  After all, testing to force all possible messages to occur–a simple experience based technique recommended by James Whittaker in How to Break Software–would have revealed this bug as well.

In all, any of three well-established test design techniques–one black box (equivalence partitioning), one white box (statement coverage), and one experience based (the force-all-messages attack)–would have found this bug.  Risk based testing would have lead to more thorough coverage of the underlying quality risk. If financial related quality risks are not being tested using well-established best practices, then what else isn’t being tested in McAfee’s systems?

A Holiday Gift for Yourself: Improving Your Testing by Christmas

For those of us on the Western calendar, we have some holiday time coming soon, including the December break.  Many of us will spend this time relaxing, which is always good. However, why not invest a little of your holiday time in improving your testing operation? After all, if you’re like most testers, you are time constrained and need to make improvements quickly that show fast results. So here are three practical ideas which you can put into action before January arrives, which will make a noticeable difference when you start to take on the projects that await in 2011.

Get Hip to Risk-Based Testing

I’ve gone on quite a bit in this blog about risk based testing, but let’s keep it short and sweet here.  I have a simple rule of thumb for test execution: Find the scary stuff first. How do we do this? Make smart guesses about where high-impact bugs are likely. How do we do that? Risk-based testing.

In a nutshell, risk-based testing consists of the following:

1. Identify specific risks to system quality.

2. Assess and assign the level of risk for each risk, based on likelihood (technical considerations) and impact (business considerations).

3. Allocate test effort and prioritize (sequence) test execution based on risk.

4. Revise the risk analysis at regular intervals in the project, including after testing the first build.

You can make this process as formal or as informal as necessary. We have helped clients get started doing risk-based testing in as little as one day, though one week is more typical. You can mine this blog for more ideas, check out a few articles on the RBCS web site (such as this one and this one), the year-long series of videos on our Digital Library, , or my books Managing the Testing Process (for the test management perspective) or Pragmatic Software Testing (for the test analyst perspective).

Whip Those Bug Reports into Shape

One of the major deliverables for us as testers is the bug report. But, like Rodney Dangerfield, the bug report gets “no respect” in too many organizations. Just because we write them all the time doesn’t mean they aren’t critical—quite the contrary—and it doesn’t mean we know how to write them well. Most test groups have opportunities to improve their bug reporting process.

When RBCS does test assessments for clients, we always look at the quality of the bug reports. We focus on three questions:

1. What is the percentage of rejected bug reports?

2. What is the percentage of duplicate bug reports?

3. Do all project stakeholder groups feel they are getting the information they need from the bug reports? If

the answer to questions one or two is, “More than 5%,” we do further analysis as to why. (Hint: This isn’t always a matter of tester competence, so don’t assume it is.) If the answer to question three is, “No,” then we spend time figuring out which project stakeholders are being overlooked or underserved. Recommendations in our assessment reports will include ways to gets these measures where they ought to be. Asking the stakeholders what they need from the bug reports is a great way to start—and to improve your relationships with your coworkers, too.

Read a Book on Testing

Most practicing testers have never read a book on testing. This is regrettable. We have a lot we can learn from each other in this field, but we have to reach out to gain that knowledge.

(Lest you consider this suggestion self-serving, let me point out that writing technical books yields meager book royalties. In fact, on an hourly basis it’s more lucrative to work bagging groceries at a supermarket. Other benefits, including the opportunity to improve our field, are what motivate most of us.)

There are many good books on testing out there now. Here’s a small selection, any one of which you could work your way through during a winter vacation:

  • General tips and techniques for test engineers: Pragmatic Software Testing, Rex Black; A Practitioner’s Guide to Software Test Design, Lee Copeland.
  • Object-oriented testing: Testing Object-Oriented Systems, Robert Binder.
  • Web testing: The Web Testing Handbook, Steve Splaine
  • Security testing: Testing Web Security, Steve Splaine; How to Break Software Security, James Whittaker
  • Dynamic test strategies and techniques: How to Break Software, James Whittaker; Advanced Software Testing: Volume 1, Rex Black.
  • Test management: Managing the Testing Process, Rex Black; Advanced Software Testing: Volume 2, Rex Black
  • Test process assessment and improvement: Critical Testing Processes, Rex Black; Test Process Improvement, Martin Pol et al
  • ISTQB tester certification: Foundations of Software Testing, Rex Black et al; The Testing Practitioner, ed. Erik van Veenendaal; Advanced Software Testing: Volumes 1, 2, and 3, Rex Black et al. 

I have read each of these books (some of which I also wrote or co-wrote). I can promise you that, if you need to learn about the topic given, reading one of the books for that topic will repay you in hours and hours saved over the years, as well as teaching you at least one or two good ideas you can put in place immediately.

The Future of Test Management

The smart test manager plans for the future.  These plans should cover not only the current project, but also the current decade.  How will you succeed as a test manager in the 2010s decade? Here are ten things you must learn to do:

  1. Connect testing to business value, including measuring effectiveness and efficiency against strategy goals;
  2. Manage testing on outsourced projects, including outsourcing of testing and outsourcing on Agile projects;
  3. Perform system integration testing on systems-of-systems projects effectively;
  4. Test systems that include open source software, and use open source tools;
  5. Test integration of new systems with legacy systems, and test the maintenance of legacy systems;
  6. Test effectively and efficientlywhen there’s too much testing work, too little time, and too few resources;
  7. Deal with the tester “skills gluts” that are created by outsourcing and crowd-sourcing, with millions of entry-level testers;
  8. Deal with the tester “skills shortages” that are created at the upper end of the skills triangle by these entry-level testers, especially in developing regions;
  9. Choose the right certifications, including security, tools, ISTQB, technology, and more;
  10. Manage testing on iterative and Agile projects.

The smart test manager who can do these ten things will be in a strong position to succeed as this decade unfolds.  Hear more about the future of test management here.

Picking Certifications: Software Testing and Beyond

One key to quality software is the quality of the people involved in creating and maintaining it.  One of the tools for increasing the quality of your team is through training of existing employees, which I’ll address in a later blog post. For this post, I want to focus on something that is often confused with training, but actually is (or at least should be) something entirely different: certification.

All IT managers–whether software test managers or other software managers–want to hire qualified people.  Certainly, IT certification can be part of the qualification puzzle in many IT fields.  IT professionals often use certification in key skill areas to demonstrate their qualifications.  However, with all the certification programs floating around out there, how do managers and professionals distinguish useful certifications from pointless wallpaper?  In this post, I’ll examine how you can pick the right IT certifications for yourself (as an individual) or for your team and the people you hire (as a manager).

Any certification worth considering will have, at its basis, a body of knowledge or syllabus.  This document should describe the skills and abilities that the certification measures.  Those people who have mastered most of these skills and abilities (sometimes called “learning objectives” in the syllabus) will be able to earn the certification, usually through some kind of exam. 

So, the first and most important step is to determine whether the skills and abilities listed in the syllabus are useful.  Does the syllabus relate to your day-to-day work?  Will the benefits of achieving the certification—increased effectiveness and efficiency, credibility of the team, etc.—justify the cost?

Of course, it’s possible that your day-to-day work should more closely resemble what is described in the syllabus.  This can happen when your organization is not following industry best practices.  So, you should also evaluate the source of the syllabus.  If the syllabus was written by a broad, international team of recognized, published industry experts, perhaps you should consider moving your practices towards those required for certification.  Adopting the certification as a guideline for your practices—and hiring people with the certification—can be a good way to move in this direction.

Selecting a certification developed by a broad team of recognized, published industry experts is important because, in general, such certifications enjoy increased acceptance over certifications developed by a small clique of like-minded people.  People in the industry will recognize the names of the authors and developers of the syllabus.  To some extent, the credibility and thus value of all certifications rests upon the reputation and credibility of the people who stand behind those certifications. 

I also mentioned that the team of experts should be international, because so often now we are engaged in globally distributed work.  If you are not working in a globally distributed fashion today, you probably will be soon.  So, you need certifications that have a global reach.  If you want to hire (or be part of) a global team of certified professionals, a single common certification is key.  This way, the whole team speaks the same language and knows the same concepts.

Of course, if you plan to hire people who hold a certification because you believe the syllabus has value, you want to be confident that those people have indeed mastered the topics in the syllabus.  This brings us back to the matter of the exam. 

Certification exams are a complicated issue, and some ill-informed polemics about exams occur on a few internet web sites.  Proper creation of exams is the province of a profession called psychometrics.  Psychometrics applies the fields of psychology, education, and statistics to the process of qualifying people through exams.  Any legitimate certification body (i.e., the organization developing and administering an exam against a syllabus) will employ professional psychometricians to ensure proper exams.

In evaluating whether an exam properly assesses someone’s qualifications, you need answers to four questions. First, is the exam statistically valid, and can the certification body prove validity?  Second, is the exam a quality instrument, free from grammatical and spelling errors, formatting problems, and other glitches that might distract exam takers, and what process does the certification body use to ensure quality?  Third, is the exam of uniform difficulty and quality whenever and wherever it is administered, and how does the certification body accomplish uniformity?  And, fourth and finally, since exam questions are developed by people, what steps does the certification body use to ensure the integrity of the exams; i.e., that the questions are not leaked to candidates, onto the internet, or to accredited training providers?

This last point—that of accredited training providers—brings us to an important consideration.  It is certainly valuable to have training available to support certification programs.  Accrediting training, whereby the certification body checks the content of the training to ensure compliance with and coverage of the syllabus, can help busy managers and professions narrow their search for quality training.  However, when the accreditation process is opaque, when only members of the certification body offer accredited training, or, worse yet, when accredited training is required to take an exam, you are not looking at a real certification: you are looking at a marketing vehicle for some company’s or cartel’s training programs.  You should pick certification programs that have open, transparent processes for accreditation, with a diverse, competitive field of training providers, and which do not require any training at all to take the exams.

Certifications can help IT managers and professionals grow their teams and their skills, if chosen carefully. If you select the right bodies of knowledge, developed by the right people and delivering the right skills for your work, certification can lead to improvements in effectiveness, efficiency, and communication within teams. It’s also essential that the certification body follow best practices in the creation and delivery of the exams. And, if you decide to use training to help achieve certification, make sure to pick a program where the training supports the certification, not vice versa.  If you follow these basic concepts, you can obtain good value from IT certification programs, both as a professional and as a hiring manager.



 
`