software testing services
 Software testing training, consulting, and outsourcing from the experts: Rex Black Consulting Services (RBCS)
(866) 438-4830
ISTQB certification testingISTQB certification testing ISTQB certification testing

Archive for the ‘software test skills’ Category

Some Questions, and Answers, on Advanced Software Testing Topics

I received some questions from a reader and webinar attendee, which I’ll answer below:

Hi Rex,

I am Jason, and few days ago was in the webinar for Reviews. As promised, I have collated the lists of questions to ask based on the Study Preparation Guide for CTAL-TM that I purchased from RBCS recently. Please do not hesitate to contact me should you require further information from me. Below are the questions and my comments:-

 71. An organization follows a requirements-based test strategy for most of its projects. Which of the following is the best example of modifying the test approach for a project based on an understanding of risks?

A. Past performance issues lead to an increased effort on performance testing

B. Test estimation is based on the number of pages in the requirement specification.

C. Test execution is outsourced to a testing company based on a low-cost bid.

D. Unit test effort is limited to ensure early commencement of system test execution.

I did not really understand this question very well. How does performance issues in the past are related to risk?

Jason, the reason that A is the right answer is because we are using past defect information (in this case, performance defects) to assess the likelihood of particular types of problems. 

84. You are managing a test effort that uses entirely reactive techniques, including a list of past bugs found in the field, a checklist of typical bugs for products using this technology, and exploratory testing based on tester experience. No written tests are developed prior to test execution, other than the list of bugs. Consider following statements

i Immune to the pesticide paradox

ii Repeatable for regression and confirmation testing

iii Useful in preventing bugs during system design

iv Cheap to maintain

v Makes no assumptions about skill tester skills

Which of the following is true about this particular testing strategy?

A. I and III are benefits of this strategy.

B. II and V are benefits of this strategy.

C. I and V are benefits of this strategy.

D. I and IV are benefits of this strategy.

Argument: I should not be part of the answer. “Exploratory testing based on tester experience” allows different test cases combination to be tested. If that is the case, why is that immune to the pesticide paradox? To my understanding, immune to the pesticide paradox refers to doesn’t have any effect to the pesticide paradox.

Since it is based on list of past bugs found in the field and checklist, I would consider II as tester can go through the regression based on these checklists.

The Foundation and Advanced syllabi are clear on the fact that detailed test cases–which do not exist here–are useful if repeating tests precisely for regression and confirmation testing purposes is needed.  So, II cannot be a benefit. 

The Foundation syllabus, in the section about general testing principles, says that the only way to overcome the pesticide paradox is to run different tests rather than repeating the exact same tests. Because all tests would be subtly (or even dramatically) different if repeated, especially if testers with different experience are used for subsequent executions.

125. You manage a test team for a bank. Your test team uses two primary test strategies, checklist-based and dynamic. As its checklist, your team has a list of main areas in which the test team, in-house users, or bank customers have reported defects on past releases. For the dynamic testing, it employs members of the test team with experience in the bank branches and back-office to do exploratory testing.

Based on this information alone, which of the following is an improvement that you would expect from a STEP assessment?

A. Get involved earlier in the lifecycle

B. Analyse requirements specifications

C. Run only scripted tests.

D. Improve the office environment.

The answer given in the guide is B. Analyse requirements specification. Could you please elaborate the reason to this answer?

Yes.  The reason is that, as mentioned in the Advanced syllabus, STEP is based on an assumption that requirements analysis and requirements-based testing will occur.

Looking forward hearing from you soon. Thanks.

I hope this is helpful.

Useful Pairwise Testing Links

As people familar with my books, webinars, and training on test design techniques know, there are a few situations where pairwise and other combinatorial testing can make a lot of sense, especially for higher-risk systems.  Following a webinar, listener Terry Croskrey sent the following useful links:

I first got introduced to you on your ITUNES Podcasts and then your books. You are an excellent writer and presenter and I appreciated your enormous contribution to the Profession of Software Testing!

Thanks, Terry.

I wanted to pass on some recent software I discovered  for ALL Pairs and Orthogonal Array creation…that is easy to use.



NIST free software:

ACTS GUI software:

I’d add to this the link

Where to Find Quick Tips on Software Testing Techniques?

I received a request from Martin for help on how to apply orthogonal arrays.  He wrote:


I am working on putting together testing at work using an orthogonal array. This is the first time I’ve used this method. I read about it in detail in Rex’s books. But I’m having difficulty remembering all the steps. I need a little help to jog my memory. Could you please send me the high-level, bulleted, steps for creating the orthogonal array with my test case data?Thank you.



Sure, Martin, you can take a quick listen to the key parts of my webinar on pairwise testing techniques. This webinar addresses how to use orthogonal arrays and other pairwise tools.  After you listen to that, I’d suggest you go to to find a free downloadable tool. Last I heard, the US Department of Commerce had funded a project to create a pairwise tool (your tax dollars at work!) that will actually do classification trees as well, and I think you might find it there.

In general, Martin, if you are looking for a refresher on test design techniques, a good first stop is the RBCS Digital Library.

RBCS Webinar Audio Adjustment

Just in time for tomorrow’s and next week’s webinar–and this week’s webinar-style virtual Foundation bootcamp–I can share some advice on audio adjustments from long-time webinar attendee Avner:

…I experienced a personal issue with the gotowebinar sound. The problem manifests as untenable echo on the presenters voice. I don’t know why by default the audio started with this effect ON. After a few minutes of experimenting I was able to find the solution. It’s an easy fix but if you don’t know where to go, it’s a deal breaker. I.e. I think I’ll go to lunch. Included is a screen print of how to fix it [see embedded picture below]. Please share this with Mr Black or whoever needs to know. This has happened to me before on other seminars where I was not able to fix the problem & had a good lunch instead.

Regards & thank you again for an informative webinar,

Avner Uzan

Adjusting Webinar Audio to Avoid Echo

Avner, thanks for sharing this tip.  Audio for these webinars can indeed be tricky, both for the presenter and for the listener.  I’m sure you’ve helped a few people catch webinars they would other miss…for lunch.

Measuring Software Test Processes (and Software Testers)

On the heels of the webinar last week, listeners have had lots of comments (all good) and some questions.  Here’s an interesting set of questions from listener Stephen Ho. I’ve interspersed my answers in his e-mail, with “RB:” in front to make it easier to follow:


Thanks for such Webinar. This webinar did not talk about how to organize and build up a good testing Metrics.

RB:  There was a general discussion early on about how to go from an objective to a specific metric and specific targets for that metric.  Perhaps you were missing the part about implementing the metrics with specific tools? 

However, it provided some interesting points to measure the successful of a testing project, such as: BFE. Just for more realistic, how can we know that a testing metrics is good?

RB: One attribute of a good metric is that it is traceable back to some specific objective. That objective should relate to a process (e.g., finding defects is an objective for the test process), to a project (e.g., reaching 100% completion of all schedule tests is an objective for many projects), or to a product (e.g., reaching 100% coverage of requirements with passing tests is an objective for some products).  Another important attribute is that the metric supports smart decision-making and, if necessary, guides corrective action.  Yet another important attribute is that the metric have a realistic target.

At here, I have another topic for your interest.

“What is effective & efficiency testing?”

RB:  We have to be more specific than this.  What are the objectives for testing, as you mean it here?  Once you have defined those objectives, you can then discuss effectively and efficiently meeting them.  For example, if you define finding defects as an objective, then you can use the DDP metric (discussed in the presentation) as a metric of effectiveness.  Cost of quality (which is discussed in various articles in the RBCS web site, such as this one) can serve as a metric of efficiency.

-how to narrow it down to know that our existing testing job is in effective & efficiency ways.

RB:  You might want to read the chapter I wrote (Chapter 2) on this topic in the book Beautiful Software.  That’s a book worth reading, anyway, because there a number of other good chapters in it.

-What are the right way to measure the performance of a QA?

RB:  I assume you’re talking about an individual tester here. If we can define specific objectives for the tester, then we can use the same method to define metrics.  Keep in mind the rule about objectives needing to be SMART.

-How can we know that a QA is in competence level?

RB: Check out my book, Managing the Testing Process, 3e, for a discussion about how to use skills inventories to manage the skills of your test team.

-How to increase the productivity of a QA?

RB: This question is too general, I’m afraid.  Productive at what?  I suggest that you define specific objectives for the test team, and then measure the current efficiency with which those objectives are achieved.  At that point, you can make realistic (and measurable) goals for improvement of productivity.

You may have existing webinar or article regarding this topic. If yes, I am thirsty to study your material. Would you direct me how to access to this information. I would definitely provide my feedback to you.

RB:  Follow this link for another article on metrics that you might find useful.



RB: You’re welcome.

Boundary Value Analysis

Long-time reader Din asked a good question about the ISTQB Advanced syllabus and the test design techniques covered in Chapter 4.  He wrote:

Hi Rex,

As far as the ISTQB syllabus is concerned, topic on boundary value analysis & testing is discussed in Foundation and Advanced level. Just want to check whether deeper discussion on extension of BVA/BVT such as robustness testing and robust worst- case testing (and other related approach) of BVA/BVT are covered in our ISTQB syllabus, especially in Advanced level. I raise this as I came across several academic presentations discussing on BVA/BVT of more than one parameter/variables which I didn’t realize so much before this. Hope to get your feedback.



Well, Din, I can’t speak for other training providers–and they probably wouldn’t want me to even if I could! :-)   However, I can say that the RBCS Advanced Test Analyst and Advanced Technical Test Analyst courses–and the corresponding books–go into the topic of boundary value testing (and the related concept of equivalence partition testing) in great depth.  This includes multi-value boundary value testing, along with the use of these techniques in combination with other test design techniques such as all-pairs testing, state-based testing, decision-table testing, and so forth.

More ISTQB Certified Tester Advanced Level Exam Advice from the Trenches

Following on my most recent post, another one of the Advanced Test Analyst course attendees took the exam this week.  Kelly Rasmussen passed on the following advice.  While a bit more targetted to the CTAL-TA exam than the other CTAL exams, there are some good general guidelines, especially regarding Foundation-level questions, the weighting of questions, and the typical e-exam experience.

Just took my exam today and I passed also! Don’t know if you all took the test yet or not, but wanted to share my experience with you. Hopefully it’ll be of some help.

Firstly, I would like to say that there was no lockers [at the Kriterion exam center] to lock up my stuff. I didn’t know this going in so I had my purse and cell phone with me. The receptionist took my stuff and put it under her desk, which I didn’t feel all that comfortable doing but didn’t want to go back to my car and put my sutff up. The exam didn’t take the whole 3 hours. I was able to answer all the questions within 2 hours. Then I used next 30 minutes or so going back to the questions that I marked for review. I wasn’t sure if I was quite ready for my results yet, so I sat there for a minute or two deciding whether to click the submit button or not. lol… Yes, I was very nervous to find out if I passed or not, but I was also tired and wanted to go home. So clicked the button, and luckily, I passed.

 As far as the type of questions I got, I was surprised to see so many K1 questions from the Foundation Level. I think my first 7 questions or so were from the Foundation Level, which worked to my advantage. There were plenty of scenario questions, which were tricky in my opinion. [People with extensive testing experience will find that helpful] in deciding on the correct answer. Another thing that I thought was tricky is that you had to choose which test technique would be the best one to use depending on the scenario. So make sure you know when it is better to use one technique over another and of course how to figure out how many test cases one would need to test. You want to get as many of the K3 questions correct as possible since these are worth 3 points. I basically ditto what Jennifer said. Go over everything!!! Pay close attention to chapters 2 and 4. There were a lot of questions from these two chapters!!!

I’ll add that, implicit in Kelly’s comments, and Jennifer’s, is the lesson that extensive pre-exam preparation is required.  In addition to attending a class or going through e-learning, be sure to spend lots of additional time studying for the exam.

ISTQB Certified Tester Advanced Level Exam Advice from the Trenches

As followers of the RBCS Facebook and Twitter micro-blogs will know, last week we offered an ISTQB Advanced Test Analyst course.  The corresponding exam has the mouthful-name “ISTQB Certified Tester Advanced Level-Test Analyst,” typically abbreviated “CTAL-TA.” CTAL-TA is also the authorized resume acronym for those who pass the exam, by the way. 

Jennifer Parran of Booz Allen Hamilton took an electronic version of the exam at a Kriterion exam center just this week, and she passed on the following details.  This is really good advice for anyone taking an ISTQB Advanced exam, especially the Test Analyst and Technical Test Analyst exams.

Hello –

I just wanted to let you all know I took the exam yesterday and PASSED! To give you an idea of what it was like… I took the electronic exam at the Career Technology Center in Falls Church [Virginia]. You have 3 hours once you click the “Start” button and there is a timer counting down on the screen (try not to let that distract you). (Also you may be in the room with other people taking other exams, some of which are allowed to bring books in and they might be making noise flipping back and forth in the book.)

You can mark questions for “review” if you want to review the answer or if you want to skip a question and come back to it later. You can also navigate backwards and forwards so if you answer a question and want to go right back to it, you are able to do so. Once you’re all done and click the “Submit” button (at which point I felt like I wanted to throw up) it takes a couple seconds and then displays your score and that you “passed”. It also sends you an email with this information which I saw later.

Oh yeah…don’t touch the keyboard while taking the exam! I accidentally hit the spacebar and it backed me out of the exam and I had to get the proctor to log back in and bring the exam back up. THANKFULLY it saved all my answers as I only had an hour left but to play it safe – only use the mouse!

The questions were very similar in style to the sample questions [provided in the RBCS live and e-learning courses and exam prep guides] so definitely do all of those. My questions covered every chapter in the advanced syllabus so you definitely want to read the entire syllabus [as discussed in the class]. Some of the questions seemed more foundational but it was hard to distinguish between K1 Foundation and K1 Advanced. I also had questions pertaining to several of the referenced standards so like Rex said in class – you don’t want to be “sad” when you’re reading one those questions and you can’t remember what it is.

Make sure you are very comfortable with the black-box techniques like we practiced in class because there are quite a few questions that are very long and involve a lot of reading [that involved] these techniques so you don’t want to have to spend a lot of time reading the question and remembering how to do the technique. These are of course the K3 questions.

As far as preparation goes, I read through all of the foundation and advanced syllabus’ once and then skimmed over my “highlights” several more times (once right before I took the exam). I did all the sample questions and mock exams. I skimmed over the referenced standards a couple times. I studied a little Monday night, 3 hours Tuesday night, all day Wednesday, and then read over my highlights right before the exam.

Good luck to all of you and Thanks Rex for all the great explanations and preparation!


By the way, for those of you considering taking an ISTQB exam and still mulling over your exam preparation options, RBCS will have an exciting announcement in the next couple weeks on this topic.  Watch our Facebook page, Twitter posts, and our newsletter for more information.  If you’ve already decided to take an RBCS course, you can find the course schedule on the home page.

Advanced Software Testing: Preconditions for Test Execution

Here’s another good question from a reader.  Tingting Ren asks:

2.10. Sample Exam Questions

3. Which of the following is not always a pre condition for test execution?

A. A properly configured test environment

B. A thoroughly specified test procedure

C. A process for managing identified defects

D. A test oracle

I think the answer is C but in the book it gives B.

This is a good question.  The answer is “B” because oftentimes testers use exploratory testing and logical (high-level) test cases when running tests, rather than concrete (low-level) test cases that spell out every action to be taken and result to be observed.   For example, see Whittaker’s book How to Break Software.

The answer cannot be “A” because the test environment must be configured correctly in order for the results to be reliable.  The answer cannot be “C” because we must have an agreed-upon way–a bug tracking system, ideally–for handling failures that we observe when running tests.  The answer cannot be “D” because we must have a way of distinguishing expect results from unexpected results.

Are Software Testers Made or Born?

I received an interesting e-mail from long-time reader, John Singleton:

I’m so proud of my 6-year old, Josh.  This weekend, we were playing Wii. Those of you likewise hooked on Wii will know that pushing the Home button on the controller pauses the current game and allows the user to access some configurations and such. It also displays the battery level for all connected Wii controllers. This time, the battery icon had only one bar, with the red color to get your attention. Josh said, “Hey Dad, have you ever noticed that when you push Home, it shows the battery is full for just a second before it shows that it’s empty?”

I had him show me, and sure enough. There on the Home screen, it shows the battery meter as blue and full for just an instant before it refreshes with the red, almost-empty icon. I think I spouted off something geeky about how it probably shows a default value for just a moment while it queries for the actual battery value.

I don’t know if I mentioned, but my son also has some special needs, including Sensory Processing Disorder, which tends to make one much more acutely aware of any kind of sensory stimulus. I wonder if this episode is similar to the kinds of dynamics that have led people to hire individuals with Aspergers’ or Autism Spectrum Disorder for software testing. Or maybe he’s just quirky, like his dad…

Regardless, my heart swells to about ten times its normal size when I hear my six-year-old finding obscure software defects in robust commercial products!

Test on!


An interesting set of questions raised here, John.  Certainly, my children are very adept at digital devices, but they don’t seem to show that level of awareness for problems.  Is testing an innate skill?  Do certain traits which are thought of as “disorders” in some contexts actually provide skills in other contexts?

Comments, anyone?