I received another interesting e-mail from a colleague a few weeks ago.  Sorry about the delay in response, Simon, but here are my thoughts.  First, Simon’s e-mail:

Rex

I have been reading the Advanced Test Manager book & have been discussing the possibility of adopting an informal risk based approach in my test team, but I am encountering some resistance, which has also got me thinking. You have covered (in several places) the topic of gaps in risk analysis from a breadth point of view, but how about the issue of disparity in ‘depth’ for identified risk items? For example in your ‘Basic-Sumatra’ Spreadsheet there is a huge variation in depth

between, for example the risk item ‘Can’t cancel incomplete actions using cancel or back.’ (A functional item that has a risk score) and ‘Regression of existing Speedy Writer features.’ (This is also a functional item, but may constitute several hundred test cases).

In my case an experienced tester is against the idea of informal risk analysis due to the effort involved. The scenario is one where a regression ‘plan’ (set of test cases) is already in place for an enterprise scale solution with 10 main components deployable in both a

Web & Windows client manner. So the usual regression test execution ‘plan’ requires executing a complex test procedure 10×2 times. In total there is several hundred test cases to execute (some components have approx 100 test cases).

When I suggested an informal (PRAM) style risk identification to each new project the response was:-

The effort of establishing such a ‘test plan’ seems to be enormous considering that the whole thing has to be performed per application component for each Win and Web client (i.e. 10 x 2 times). I estimate that the number of items requiring risk scoring will be approx 100 for each of the bigger components let alone the whole of the application.

In response to this I pointed out that we could have a ‘coarse grained’ risk item identification & score – perhaps 20 lines on the risk assessment spreadsheet- 1 for each component\deployment combination.

The response to that was:-

If each of these 20 lines has got an RPN and all the test cases assigned to it just inherited this RPN, this would mean that we would perform an 8 hour test on ‘Securities Win client’ before even beginning with the test of another component, which has got a lower

RPN. Further, this could mean that low-priority components might not be tested at all in a tight time schedule. This cannot be the desired test procedure. It must be ensured that each component is at least tested basically on Win and Web … which would again lead us to scoring risk items at the test case level within each component for Windows and Web & that has the problem of the effort involved.

Do you have any suggestions for handling this depth of risk identification issue?

Regards Simon

This is an important question, Simon, that brings up three important points.

First, the amount of effort invested must be considered.  We usually find that the risk analysis can be completed within a week.  The time involved depends on the approach used.  If you use the group brainstorm approach, then each participant must invest an entire day, with the leader of the risk analysis typically investing a couple days in addition on preparation, creating the analysis, doing follow-up, etc.  If you use the sequential interview approach, then each participant invests about three hours, with 90 minutes in the initial interview and 90 minutes in the review/approval process for the document, with the leader of the risk analysis again investing about three days of effort.

Second, the question of granularity of the risk analysis is also important. The granularity must be fine-grained enough to allow unambiguous assignment of likelihood and impact scores.  However, if you get too fine-grained then the effort goes up to an unacceptable level.  A proper balance must be struck.

Third, the question of whether we might not test certain important areas at all because they are seen as low risk is indeed a problem.  What we typically suggest is what’s called a “breadth-first” approach, which means that to some extent the risk-order execution of tests is modified to ensure that all major areas of the software are tested.  These areas are tested in a risk-based fashion, but every area gets at least some amount of testing. 

Many of these topics are addressed in the sequence of videos on risk based testing that you can find on our digital library.  I’d encourage interested readers to take a look at those brief videos for more ideas on these topics.