In this blog, I have talked a lot about RBCS’ approach to risk based testing, whichwe call the Pragmatic Risk Analysis and Management process. As you know if you’ve followed our videos on risk based testing (e.g., this one), PRAM defines the following extents of testing, in decreasing order of thoroughness:

• Extensive
• Broad
• Cursory
• Opportunity
• Report bugs only
• None

Risk based testing does not prescribe specific test design techniques to mitigate quality risks based on the level of risk, as the selection of test design technique for a given risk item is subject to many factors. These factors include the suspected defects (what Beizer called the “bug hypothesis”), the technology of the system under test, and so forth. However, risk based testing does give guidance in terms of the level of test design (e.g., see here), implementation, and execution effort to expend, and that does influence the selection of test design techniques. The following subsections will provide heuristic guides to help test engineers select appropriate test techniques based on the extent of testing indicated for a risk item by the quality risk analysis process. These guides apply to testing during system and system integration testing by independent test teams.

Extensive

According to the quality risk analysis process template, for risks rated to receive this extent of testing, the tester should “run a large number of tests that are both broad and deep, exercising combinations and variations of interesting conditions.” Because combinational testing is specified, testers should select test design techniques that generate test values to cover combinations. These techniques are either: (a) domain analysis or decision tables; or, b) classification trees, pairwise testing, or orthogonal arrays. The techniques in option (a) are appropriate where the mode of interaction between factors is understood (e.g., rules determining output values). The techniques in option (b) are appropriate where the mode of interaction between factors is not understood or indeed interaction should not occur (e.g., configuration compatibility). For each technique selected, the strongest coverage criteria should be applied; e.g., all columns in a decision table, including the application of boundary value analysis and equivalence partitioning on the conditions in the decision table. The use of these combinational techniques guarantees deep coverage.

In addition, testers should ensure that, for all relevant inputs or factors, tests cover all equivalence partitions and, if applicable, boundary values. This contributes to broad coverage.

Testers should plan to augment the test values with values selected using experience-based and defect-based techniques. This augmentation can occur during the design and implementation of tests or alternatively during test execution. This augmentation can be used to broaden test coverage, to deepen test coverage, or both.

If available, use cases should be tested, and the tester should cover all normal and exception paths.

If available, the tester should use state transition diagrams. Complete state/transition coverage is required, 1-switch (or higher) coverage is recommended, and, in the case of a safety-related risk items, state transition table coverage is also recommended.

In some cases—e.g., safety critical risks, risks related to key features, etc.—the tester may elect to use code coverage measurements for risks assigned this extent of coverage, and to apply white box test design techniques to fill any code coverage gaps detected by such measures.

As a general rule of thumb, around 50% of the total test design, implementation, and execution effort should be spent addressing the risk items assigned this extent of testing.

Broad

According to the quality risk analysis process template, for risks rated to receive this extent of testing, the tester should “run a medium number of tests that exercise many different interesting conditions.” Testers should create tests that cover all equivalence partitions and, if applicable, boundary values. Testers should plan to augment the test values with values selected using experience-based and defect-based techniques. This augmentation can occur during the design and implementation of tests or alternatively during test execution. This augmentation should be used to broaden test coverage.

If available, use cases should be tested, and the tester should cover all normal and exception paths.

If available, the tester should use state transition diagrams. Complete state/transition coverage is required, but higher levels of coverage should only be used if possible without greatly expanding the number of test cases.

If available, the tester should use decision tables, but strive to have only one test per column.

Other than the possible use of decision tables, combinational testing typically should not be used unless it can be done without generating a large number of test cases.

As a general rule of thumb, between 25 and 35% of the total test design, implementation, and execution effort should be spent addressing the risk items assigned this extent of testing.

Cursory

According to the quality risk analysis process templates, for risks rated to receive this extent of testing, the tester should “run a small number of tests that sample the most interesting conditions.” Testers should use equivalence partitioning or boundary value analysis on the appropriate areas of the system to identify particularly interesting test values, though they should not try to cover all partitions or boundary values.

Testers should plan to augment these test values with values selected using experience-based and defect-based techniques. This augmentation can occur during the design and implementation of tests or alternatively during test execution.

If available, use cases should be used. The tester should cover normal paths, though the tester need not cover all exception paths.

The tester may use decision tables, but should not try to cover columns that represent unusual situations.

The tester may use state transition diagrams, but need not visit unusual states or force unusual events to occur.

Other than the possible use of decision tables, combinational testing should not be used.

As a general rule of thumb, between 5 and 15% of the total test design, implementation, and execution effort should be spent addressing the risk items assigned this extent of testing.

Opportunity

According to the quality risk analysis process templates, for risks rated to receive this extent of testing, the tester should “leverage other tests or activities to run a test or two of an interesting condition, but invest very little time and effort.” Experience-based and defect-based techniques are particularly useful for opportunity testing, as the tester can augment other tests with additional test values that fit into the logical flow of the tests. This can occur during the design and implementation of tests or alternatively during test execution.

In addition, testers can use equivalence partitioning or boundary value analysis on the appropriate areas of the system to identify particularly interesting test values, though they should not try to cover all partitions or boundary values.

As a general rule of thumb, less than 5% of the total test design, implementation, and execution effort should be spent addressing all of the risk items assigned this extent of testing. In addition, no more than 20% of the effort allocated to design, implement, and execute any given test case should be devoted to addressing any risk item assigned this extent of testing.

Report Bugs Only

According to the quality risk analysis process templates, for risks rated to receive this extent of testing, the tester should “not test at all, but, if bugs related to this risk arise during other tests, report those bugs.” Therefore no test design, implementation, or execution effort should occur, and it is a misallocation of testing effort if it does.

None

According to the quality risk analysis process templates, for risks rated to receive this extent of testing, the tester should “neither test for these risks nor report related bugs.” Therefore no test design, implementation, or execution effort should occur, and it is a misallocation of testing effort if it does.