Archive for the ‘risk based testing’ Category
Sunday, April 1st, 2012 by Rex Black
I was recently in Moscow to give a presentation on testing and quality at the Microsoft QA Day event there. After my talk, I had a conversation with a software tester, who followed up with an e-mail. I’ll respond here to Nataly’s questions.
Dear, Rex.
First of all I want to thank you for your speech on the Microsoft QA Days in Moscow. Do you like the snow in late March? 
Well, it was a bit cold while I was there. However, having my expectations shaped by books like War and Peace, Crime and Punishment, Ten Days that Shook the World, and Doctor Zhivago, the wintry landscape was exactly what I imagined.
You may remember, I came up to you at the end of the day with the question about participation of the test team in reviews and risk analysis. But because I have not had sufficient practice in conversational English, I could not properly ask the question 
Certainly, Nataly, your English–written or conversational–is vastly superior to my Russian!
I will try to formulate the question more correctly. I’ll be very grateful if you could respond to my letter. Imagine that there is a test team with the appropriate skills, which should participate in review or risk analysis. Also imagine that the team was trained before the partisipation: they read the corresponding books about the review process or risk analysis process.
Well, Nataly, I’m not sure I would consider just reading a book on risk analysis or reviews sufficent training. It will be for some people, but many of our clients find that they need a little more help than that before they can become truly effective.
Despite the preliminary training, we can expect that the result of the first participation of the team will be low or zero, because of the lack of experience. However, management expects that the money invested in training will pay off immediately. Therefore, it would be wise to prepare them in advance to that the benefits of the participation of the test team will not be visible immediately.
It seems that, if all management did was pay for two books, one describing risk analysis and another describing software reviews, they can hardly have high expectations about how much behavioral and capability change is going to result from that extremely small investment. What they are going to pay for, in that situation, is the low efficiency associated with the series of practical review or risk analysis sessions that is necessary to learn-by-doing. This is exactly what you are describing. It’s often more effecient to pay for a training session, since, if the training is carefully chosen and the participants apply themselves to it, the participants will leave the training ready to be effective in their first review.
Could you please tell based on your experience, how much time (in average) usually pass before the moment when the participation of testers will bring tangible benefit? 1 review? 2 reviews? 3 reviews…?
So, we have a one-day training for risk analysis, and a two-day training for reviews. When we train people in how to do reviews or risk based testing, we find that they are immediately effective upon leaving that training. That’s not to say that they don’t continue to get better over time, but they are immediately capable of participating in an active and contributing fashion. However, if all the people do is read a book, it’s hard to say what level of effectiveness they would have.
The reason training is so much more effective than just reading a book is that training–at least good training–includes hands-on, practical, realistic exercises. This is true for any subject. Our training on risk based testing and reviews involves actually doing a risk analysis or reviewing a requirements specification. The instructor is involved during the exercise, to guide the participants to success. That way, the participants leave the training having actually, effectively and efficiently, carried out the process.
With a book, even a book that includes exercises, there is no instructor there to help guide the reader through the process. So, if the reader gets confused, or gets stuck, or thinks they know what they are doing but is actually wrong, the capability gained may be low.
I fear that managers might say something like: “testers spent so much time studying the theory, they spend time and money to the participation in review, so when we can see the benefit of their participation?”
Yes, this is a significant risk. If testers get involved in activities like reviews and risk analysis, and they are not capable of carrying those activities out, that can cause significant damage to the test team’s credibility and lead to managers deciding not to include them in the future.
Thanks in advance!
Sincerely, Nataly.
Thanks for getting in touch with me, Nataly. I hope my response has been helpful. If you do decide to pursue the avenue of reading books, please be sure that you arrange some exercises, just with the testers, before you try to participate in a real review or to initiate risk based testing. However, I would recommend that you seek out good training, training that includes hands-on exercises and instructor support, to improve the odds of success. Alternatively, if you have a person in the organization who can mentor the testers, someone with experience in reviews or risk analysis, cross-training can also work.
Tags: quality risk analysis, reviews
Posted in reviews, risk based testing, software testing | 8 Comments »
Wednesday, March 21st, 2012 by Rex Black
Vaibhavee K attended my webinar on Agile Testing Opportunities this evening, and asked an interesting follow up question:
Hello,
I have a query on Test Estimations techniques for Agile testing.
Which is the best method – Estimation should be done based on Sprints OR overall User story-Goal? What kind of Estimation techniques can be used?
Regards,
Vaibhavee
Certainly some of the estimation on Agile projects need to be based on user stories in an Agile world. However, there are systemic testing concerns associated with any application. We recommend that our clients–both in Agile and traditional lifecycles–use risk based testing to identify these systemic quality risks. These should be considered as part of the estimation for each sprint.
Tags: agile testing, risk based testing
Posted in risk based testing, software testing | No Comments »
Thursday, July 21st, 2011 by Rex Black
Regular reader Gianni Pucciani wrote me recently to discuss risk based testing in an Agile world:
Dear Rex,
Risk based testing has been discussed in several places, and from different perspective, including in your books and online resources. There are almost no doubts about the benefits it can bring. What is still missing in my opinion is a good discussion about adopting risk based testing in an agile environment.
In this case risk identification and assessment should be performed at the beginning of each sprint, analyzing the risks connected to the features that will be developed in the coming sprint. Another point worth mentioning is the importance of test automation for regression testing, but what about a situation where most of the tests are manual? I would like to hear from you and the readers of your blogs if you have any experience/suggestions to share.
Thank you.
Best regards,
Gianni Pucciani
Yes, Gianni, this is exactly how risk based testing works in an Agile environment. Here’s summary of a risk based testing process document we created for a client who uses the Scrum methodology:
1. At the beginning of the planning period for a release, identify project risk based analysis team participants.
2. Schedule risk meetings (90-120 minutes each).
3. Prepare interview documents.
4. Hold interviews with all risk identification participants.
5. Analyze risk items.
6. Normalize risks.
7. Review project Quality Risk Analysis with stakeholders.
8. Apply RPN values to test planning and test case development.
9. When release backlogs are being determined and when sprint backlogs are being revised, At major project milestones, review and revise the risk analysis.
In terms of manual regression testing, given all the great tool support for test automation in Agile environments, I’m not sure why an organization would choose to do this. It’s the best way to manage regression risk in an Agile environment.
Tags: Agile, risk based testing
Posted in best practices, risk based testing, software testing, test automation | 2 Comments »
Monday, July 4th, 2011 by Rex Black
As long-time listeners–or even brand new listeners, for that matter–of the RBCS webinars know, we use Citrix’s GoToWebinar service for our free monthly webinars. Now, I’ve been fairly satisfied with GoToWebinar. I’ve used one or two of the competing services, and been less happy with those. Of course, webinar listeners (and readers of this blog) might remember I chided Citrix back in May for the ungraceful way the system handles audio drop-outs by the presenter.
So, during the June webinar, webinar attendee Keith Stobie reported an inability to see the presentation using Internet Explorer 9. He said that Chrome (not sure which version) worked just fine. I reported the problem to Citrix on Wednesday of last week. Five days later, I receive the following reply, quoted in its entirety (minus the links provided at the end):
Thank you for contacting Citrix Online Global Customer Support,
Dear Rex Black,
IE 9 has not been tested with any of our products as of yet. we will try to help fix any issues the best we can, but cannot guarantee anything. Hopefully we should get this done as soon as possible.
If you have any additional questions or need further clarification regarding this matter, please feel free to reply directly to this email. For any other product inquiries or technical assistance, please visit us at our Support Centers listed at the bottom of this email. Our Support Centers include Self Help files and our Global Customer Support Contact Information.
Thank you,
Richard Carrel | Global Customer Support
So, I appreciate the reply, though I have to say that five days isn’t quick turnaround for a customer complaint about a browser-based service that’s incompatible with a major vendor’s browser.
More surprising to me is the admission that Citrix hadn’t tested IE9. I don’t keep up with the browser wars, so I’m not sure what share of the browsing action IE9 has, but I’m pretty sure that Microsoft’s IE family of browsers remains at least one of the 800 pound gorillas in the room.
Putting myself in the position of the Director of Quality or VP of Testing or whatever the head-testing-honcho’s title is at Citrix, I understand that there are constraints on compatibility testing. I wouldn’t bother to test four-year-old versions of Opera, for example. But come on, not testing IE9? If I were in charge of testing for any SaaS provider, compatibility would be one of my top quality risks, and testing browser/OS/malware configuration combinations would receive a fair amount of time, money, and attention. Of course, functionality, reliability, performance, and security would also be high on the list of risk categories, too.
Here’s some free consulting advice to my fellow test professionals who work at Citrix: Spend a little time getting ramped up on how to do quality risk analysis and risk based testing. You can find lots of free resources on our web site, especially in the articles and the Digital Library. You’ll notice that compatibility is one of the quality risk categories included in our free quality risk checklist. If you need more help, let me know, as we can provide a one-week risk based testing bootstrapping service that will get you headed in the right direction.
Morale of the story: If you are in charge of testing at any SaaS vendor, and you’re not testing for compatibility, it’s only a matter of time before someone writes a blog post like this one about your product and the degree to which you aren’t testing it.
Tags: software compatibility, software testing
Posted in best practices, risk based testing, software quality, software testing, test coverage | No Comments »
Monday, April 4th, 2011 by Rex Black
I received an interesting question from a colleague in Malaysia, Dhiauddin Suffian. He wrote:
Hi Rex, I have one simple question with regard to Fundamental Test Process. As we aware, the process involves Planning & Control, Analysis & Design, Implementation & Execution, Evaluating Exit Criteria & Reporting and Test Closure. My concern is on the Test Planning and Control, since it goes along the way of the whole process. I have no issue on the “Planning” portion. My question is directed to the “Control” part. What are “Control” activities involved in subsequent phases, i.e. “Control” activities that happen in Analysis & Design, Implementation & Execution, Evaluating Exit Criteria & Reporting and Test Closure, respectively. Thanks. Regards, -Din (CTFL, CTAL-TM)-
Test control can be thought of as the test management tasks required throughout the test process in order to keep the testing aligned with the software development process, the needs of the project, and the needs of the organization. These tasks occur as needed, based on the judgement of the test manager or other members of the project team, and can also occur on a planned basis.
For example, we might plan to regularly check our risk analysis to see if we have discovered new risks, or uncovered information that tells us we should revise the risk levels for the existing risks. As another example, if we find that a key piece of testing hardware will be available earlier than we expected, we might re-work our test execution schedule to accelerate the tests that use that hardware.
Yet another example could be if we discovered, during test execution, that a key test staff member will be leaving the team. In this case, if we did a thorough job during test planning, we might have identified a contingency plan for loss of a key staff member. This is a classic project risk, after all, and a good manager should consider all such risks. If we do have a contingency plan, triggering that contingency plan would be an act of test control.
Here’s an analogy: Think of the test plan as a roadmap, with the starting location and the final destination clearly indicated. This roadmap will help you drive to your chosen destination. However, throughout your drive, you should plan to stop at traffic lights, mind your lane and speed, adapt to unexpected events (such as pedestrians stepping into a crosswalk), and even adaptively overcome errors in the roadmap (such as discovering a planned route is closed due to roadwork). While a good test plan makes test control easier–just as a good roadmap makes driving easier–the smart manager remains ever alert to the possible need for test control.
Tags: software testing, test control, test management
Posted in risk based testing, software test management, software testing | 2 Comments »
Tuesday, March 1st, 2011 by Rex Black
A quick follow-up related to my earlier post on evidence. As some readers may know, avionics software that controls flight on airplanes (e.g., cockpit software) is subject to a test coverage standard, FAA DO-178B. That standard applies lower standards of test coverage to software that is not safety critical.
So far, so good.
Here’s an example of why such standards are useful. During my flight from the US to China today, I managed to crash the entertainment software running at my seat not once by three times. I did this by pausing, rewinding, and resuming play when the flight attendants were taking my dinner orders (i.e., not by unusual actions). I was ultimately able to get it working again, thanks to a series of hard reboots by a flight attendant. One of my fellow passengers wasn’t so lucky, as his system never recovered.
Okay, that’s just entertainment, and anyone who travels regularly knows they should bring a book or plan to winnow down their sleep deprivation balance on long flights.
However, what if the flight control software were as easy to crash? Who would want to hear a cockpit announcement along the lines of the following: “Our entire flight control system just crashed. This enormous airliner is now essentially an unpowered and uncontrolled glider. We’ll reboot the system until we get it working again, or until we have an uncontrolled encounter with terrain”?
Personally, I want people testing the more safety-critical aspects of avionics software to adhere to higher standards of coverage, and to be able to provide evidence of the same.
Posted in risk based testing, software test management, software testing, test coverage | 3 Comments »
Sunday, February 27th, 2011 by Rex Black
I received another interesting e-mail from a colleague a few weeks ago. Sorry about the delay in response, Simon, but here are my thoughts. First, Simon’s e-mail:
Rex
I have been reading the Advanced Test Manager book & have been discussing the possibility of adopting an informal risk based approach in my test team, but I am encountering some resistance, which has also got me thinking. You have covered (in several places) the topic of gaps in risk analysis from a breadth point of view, but how about the issue of disparity in ‘depth’ for identified risk items? For example in your ‘Basic-Sumatra’ Spreadsheet there is a huge variation in depth
between, for example the risk item ‘Can’t cancel incomplete actions using cancel or back.’ (A functional item that has a risk score) and ‘Regression of existing Speedy Writer features.’ (This is also a functional item, but may constitute several hundred test cases).
In my case an experienced tester is against the idea of informal risk analysis due to the effort involved. The scenario is one where a regression ‘plan’ (set of test cases) is already in place for an enterprise scale solution with 10 main components deployable in both a
Web & Windows client manner. So the usual regression test execution ‘plan’ requires executing a complex test procedure 10×2 times. In total there is several hundred test cases to execute (some components have approx 100 test cases).
When I suggested an informal (PRAM) style risk identification to each new project the response was:-
The effort of establishing such a ‘test plan’ seems to be enormous considering that the whole thing has to be performed per application component for each Win and Web client (i.e. 10 x 2 times). I estimate that the number of items requiring risk scoring will be approx 100 for each of the bigger components let alone the whole of the application.
In response to this I pointed out that we could have a ‘coarse grained’ risk item identification & score – perhaps 20 lines on the risk assessment spreadsheet- 1 for each component\deployment combination.
The response to that was:-
If each of these 20 lines has got an RPN and all the test cases assigned to it just inherited this RPN, this would mean that we would perform an 8 hour test on ‘Securities Win client’ before even beginning with the test of another component, which has got a lower
RPN. Further, this could mean that low-priority components might not be tested at all in a tight time schedule. This cannot be the desired test procedure. It must be ensured that each component is at least tested basically on Win and Web … which would again lead us to scoring risk items at the test case level within each component for Windows and Web & that has the problem of the effort involved.
Do you have any suggestions for handling this depth of risk identification issue?
Regards Simon
This is an important question, Simon, that brings up three important points.
First, the amount of effort invested must be considered. We usually find that the risk analysis can be completed within a week. The time involved depends on the approach used. If you use the group brainstorm approach, then each participant must invest an entire day, with the leader of the risk analysis typically investing a couple days in addition on preparation, creating the analysis, doing follow-up, etc. If you use the sequential interview approach, then each participant invests about three hours, with 90 minutes in the initial interview and 90 minutes in the review/approval process for the document, with the leader of the risk analysis again investing about three days of effort.
Second, the question of granularity of the risk analysis is also important. The granularity must be fine-grained enough to allow unambiguous assignment of likelihood and impact scores. However, if you get too fine-grained then the effort goes up to an unacceptable level. A proper balance must be struck.
Third, the question of whether we might not test certain important areas at all because they are seen as low risk is indeed a problem. What we typically suggest is what’s called a “breadth-first” approach, which means that to some extent the risk-order execution of tests is modified to ensure that all major areas of the software are tested. These areas are tested in a risk-based fashion, but every area gets at least some amount of testing.
Many of these topics are addressed in the sequence of videos on risk based testing that you can find on our digital library. I’d encourage interested readers to take a look at those brief videos for more ideas on these topics.
Tags: risk based testing
Posted in risk based testing, software testing | No Comments »
Sunday, February 27th, 2011 by Rex Black
I recently received an interesting e-mail from a colleague:
To Whom It May Concern-
Do you have any articles on the value of collecting/capturing detailed test evidence (e.g., screenshots attached to test cases)?
In my opinion, for mature systems with experienced, veteran testers, the need for an abundance of test evidence in the form of screenshots attached to test runs in QC is overkill and unecessary that adds more time to release cycles. The justification for this is awlays “For Audit” as opposed to “Improves Quality”. I looked in several articles on this fantastic site, and couldn’t find anything pertaining to test evidence. Do you have any articles that provide evidence that an abundance of test evidence improves quality (even if it’s just a correlation and not necessarily causation)?
Thanks
Erik Tuininga
We have clients that do need to retain such detailed software testing evidence; e.g., clients working in safety critical systems (such as medical systems) who must satisfy outside regulators that all necessary tests have been run and have passed. For them, retaining such evidence is a best practice, as not doing so can result in otherwise-worthy systems being barred from the market due to the lack of adequate paperwork.
As someone who relies on such systems to work–indeed, as we all do–I appreciate these regulations and would not want to see software held to a lesser standard. However, Erik makes a very valid point in terms of the trade-off. As time is spent on these audit-trail activities, that is time not spent doing other tasks that would perhaps result in a higher level of quality. Of course, these audit-trail activities are designed to ensure that all critical quality risks are addressed. So, the key question is how should organizations balance the risk of failing to test certain critical quality attributes against the reduction in breadth of quality risk coverage?
I’d be interested in hearing from other readers of this blog on their thoughts. Erik, if you have further comments on this matter, I’m sure the readers of this blog would benefit from those ideas, as this is clearly an important area to consider. I certainly agree it’s an interesting topic for an article, and this blog discussion may well inspire me to collaborate with you and other respondents to write one.
Tags: software testing evidence
Posted in best practices, risk based testing, software test management, software test metrics, software testing | 6 Comments »
Friday, February 4th, 2011 by Rex Black
The Financial Times today featured an article on how a software bug–abysmally handled–in a financial application cost the company US$ 242,000,000:
http://www.ft.com/cms/s/0/5e1ba340-2feb-11e0-a7c6-00144feabdc0.html#axzz1D2IDiwLs
Because I don’t know how long that link will live, here’s the summary.
Axa Rosenberg Group had some quantitative analysis software that it used to service its clients accounts. Axa Rosenberg Group manages money for other people, and the software is an internal application, albeit one they touted as a key differentiator, apparently–and indeed it did turn out to be, though not in a happy way.
The software had a bug that disabled a key risk-management component of the software, which was released to production in 2007. Apparently management found out about the bug in November 2009. However, rather than fix the problem, they tried to cover up the reasons for the poor performance of their funds.
Over one third of their customers were affected by the bug.
A wee bit of analysis from yours truly: I have clients in the financial world, and I know how hard it can be to test these kinds of applications. When a calculation is wrong, it can be wrong in a way that is beyond the ability of a human tester to detect. However, Axa Rosenberg Group’s handling of the bug after they found out about it is truly a textbook illustration of how not to handle a software quality problem.
Tags: cost of bugs, cost of poor quality, cost of quality
Posted in risk based testing, software quality, software test management, software test metrics, software testing | No Comments »
Tuesday, February 1st, 2011 by Rex Black
While I typically restrict myself to discussions and posts related purely to how to do and manage software testing better, I feel I must make a brief side expedition to the land of commentary. This should not be a controversial commentary, but I’m afraid it will be with some. I’d like to make a brief call for more civility in the way software testing professionals address each other, both in print and in person.
The following are real quotes from published articles this year (not an old year). They are phrases used to describe software testing professionals. They are used by people who style themselves as experts and coaches in the software testing profession. See how professional and encouraging these words sound to you: “profiteer and bully,” “risk-based testing cargo cult,” “moral and intellectual bankrupt,” “shadowy pseudo-experts,” “power mad,” and “embarrassingly stupid.”
I could go on, but you get the picture.
I have a simple rule for public discourse, both on-line and in-person: if people want to participate in a debate or discussion with me, they can expect me to be civil and respectful towards them and towards other software testing professionals, and I expect the same from them. It’ll be a better software testing world, and we’ll make a lot more progress together, when this simple rule–one we all learned as children, if we paid attention in school–wins out over the sort of self-promotion-through-name-calling that dominates so much of our debate.
Back to your regularly scheduled fact-focused software testing blogging…
Posted in risk based testing, software testing | 3 Comments »
|
Rex Black is President of RBCS (
