software testing services
 Software testing training, consulting, and outsourcing from the experts: Rex Black Consulting Services (RBCS)
(866) 438-4830
ISTQB certification testingISTQB certification testing ISTQB certification testing

Archive for April, 2012

Kind Words on our Web Resources

A new reader of the RBCS website sent some kind words that we want to share here:

The RBCS web site is outstanding. Only a few days ago, I was introduced to RBCS’ work via a CAI webinar. I’m delighted to learn from the online articles and publications about the many facets of testing. Enlightening concepts include: establishing relationships and involving other members of the project team into the testing process, aligning testing goals with the business client’s goals via risk-based testing, and examining the topic of measuring to ensure that we fully understand what we’re measuring and why.

Bill Minckler, MBA, PMP, CLSSBB
Senior Project Manager State of Ohio

Thanks, Bill.  I’m glad you find the resources useful.  I hope you’ll subscribe to the RBCS monthly webinars, which are also free resources, focused on content not advertising.  You can catch then on our YouTube channel, or better yet, attend live to earn PMI PDUs and participate directly in the 30 minute Q&A at the end.

Useful Pairwise Testing Links

As people familar with my books, webinars, and training on test design techniques know, there are a few situations where pairwise and other combinatorial testing can make a lot of sense, especially for higher-risk systems.  Following a webinar, listener Terry Croskrey sent the following useful links:

I first got introduced to you on your ITUNES Podcasts and then your books. You are an excellent writer and presenter and I appreciated your enormous contribution to the Profession of Software Testing!

Thanks, Terry.

I wanted to pass on some recent software I discovered  for ALL Pairs and Orthogonal Array creation…that is easy to use.



NIST free software:

ACTS GUI software:

I’d add to this the link

Test Strategies for Agile Situations

Following today’s webinar on test strategies, listener Cees Jan had some follow-up questions.  I’ll address them below, with answer interspersed in the text.


 I really enjoyed your webinar.

Thanks, I’m glad it was useful for you.

Here is a question you did not get to: Hi I am what would qualify as a “falsifier”: Find the worst-case scenario and see how product behaves in terms of robustness and graceful termination. Quite the opposite of regression testing which I consider “trivial pursuit” (good for build stability and sanity, but never uncovering new defects). Would that be a wise strategy? I get comments that it is far-fetched, but I am stretching the limits of a system and see where it fails. It provides a lot of eye-openers. CeesJan

It sounds like, based on your earlier question the webinar (which I did answer), that you are basically following a reactive test strategy.  You are apparently (based on this comment) focused on finding a lot of bugs, especially by checking out unusual circumstances and extreme conditions, which isa typical characteristic of reactive testing. 

I’m a bit concerned about omitting regression testing, though.  Far from being “trivial pursuit” (though I realize it’s not fun when done manually), in Agile lifecycles the high rate of change makes regression risk very significant.  If you are no addressing regression risk, who is?  Are the developers doing good automation unit (e.g., with J-unit) and feature verification tests (e.g., with Fitnesse)?  If not, then the confidence-building and risk-mitigating objectives of testing are probably not being met.

The above sounds like reverse engineering: lack of requirements and undocumented legacy code force me to this “out of the box” methodology/averse strategy. Reading books like black box testing (Beizer) and black swan (Taleb) inspired me too much, I guess.

 Certainly, in testing we have to play the hand we’re dealt.  If you have no requirements and no useable documentation, then this will often force you into a purely reactive test strategy.  However, you should be aware of how limited such a strategy is.  I would think that adding analytical risk based testing, along with addressing the regression risks, would make sense.

Regards, Cees Jan den Haan (IQSTB Practitioner, CAT)

I have to correct you here, Cees Jan.  The ISEB Practitioner certification is not part of the ISTQB program, and neither is CAT.  The ISTQB program has the Foundation, Advanced, and Expert certifications.  I know that some European training providers have created confusion on this point (perhaps for marketing purposes), but it’s wrong to associate those two certifications with the ISTQB.

How Long to Become Effective?

I was recently in Moscow to give a presentation on testing and quality at the Microsoft QA Day event there.  After my talk, I had a conversation with a software tester, who followed up with an e-mail.  I’ll respond here to Nataly’s questions.

Dear, Rex.

First of all I want to thank you for your speech on the Microsoft QA Days in Moscow. Do you like the snow in late March? :)

Well, it was a bit cold while I was there.  However, having my expectations shaped by books like War and Peace, Crime and Punishment, Ten Days that Shook the World, and Doctor Zhivago, the wintry landscape was exactly what I imagined.

You may remember, I came up to you at the end of the day with the question about participation of the test team in reviews and risk analysis. But because I have not had sufficient practice in conversational English, I could not properly ask the question :(

Certainly, Nataly, your English–written or conversational–is vastly superior to my Russian! :-)

I will try to formulate the question more correctly. I’ll be very grateful if you could respond to my letter. Imagine that there is a test team with the appropriate skills, which should participate in review or risk analysis. Also imagine that the team was trained before the partisipation: they read the corresponding books about the review process or risk analysis process.

Well, Nataly, I’m not sure I would consider just reading a book on risk analysis or reviews sufficent training.  It will be for some people, but many of our clients find that they need a little more help than that before they can become truly effective.

Despite the preliminary training, we can expect that the result of the first participation of the team will be low or zero, because of the lack of experience. However, management expects that the money invested in training will pay off immediately. Therefore, it would be wise to prepare them in advance to that the benefits of the participation of the test team will not be visible immediately.

It seems that, if all management did was pay for two books, one describing risk analysis and another describing software reviews, they can hardly have high expectations about how much behavioral and capability change is going to result from that extremely small investment.  What they are going to pay for, in that situation, is the low efficiency associated with the series of practical review or risk analysis sessions that is necessary to learn-by-doing. This is exactly what you are describing.  It’s often more effecient to pay for a training session, since, if the training is carefully chosen and the participants apply themselves to it, the participants will leave the training ready to be effective in their first review.

Could you please tell based on your experience, how much time (in average) usually pass before the moment when the participation of testers will bring tangible benefit? 1 review? 2 reviews? 3 reviews…?

So, we have a one-day training for risk analysis, and a two-day training for reviews.  When we train people in how to do reviews or risk based testing, we find that they are immediately effective upon leaving that training. That’s not to say that they don’t continue to get better over time, but they are immediately capable of participating in an active and contributing fashion.  However, if all the people do is read a book, it’s hard to say what level of effectiveness they would have. 

The reason training is so much more effective than just reading a book is that training–at least good training–includes hands-on, practical, realistic exercises.  This is true for any subject.  Our training on risk based testing and reviews involves actually doing a risk analysis or reviewing a requirements specification.  The instructor is involved during the exercise, to guide the participants to success.  That way, the participants leave the training having actually, effectively and efficiently, carried out the process. 

With a book, even a book that includes exercises, there is no instructor there to help guide the reader through the process.  So, if the reader gets confused, or gets stuck, or thinks they know what they are doing but is actually wrong, the capability gained may be low.

I fear that managers might say something like: “testers spent so much time studying the theory, they spend time and money to the participation in review, so when we can see the benefit of their participation?”

Yes, this is a significant risk.  If testers get involved in activities like reviews and risk analysis, and they are not capable of carrying those activities out, that can cause significant damage to the test team’s credibility and lead to managers deciding not to include them in the future. 

Thanks in advance!

Sincerely, Nataly.

Thanks for getting in touch with me, Nataly.  I hope my response has been helpful.  If you do decide to pursue the avenue of reading books, please be sure that you arrange some exercises, just with the testers, before you try to participate in a real review or to initiate risk based testing.  However, I would recommend that you seek out good training, training that includes hands-on exercises and instructor support, to improve the odds of success. Alternatively, if you have a person in the organization who can mentor the testers, someone with experience in reviews or risk analysis, cross-training can also work.