Software Testing Articles

This page includes synopses of some of the most recent published software testing articles and commentary from RBCS experts regarding the software testing industry and other topics. To view the full article, click on the provided link.

A Few Thoughts on Test Data

by Rex Black

This article is excerpted from Chapter 3 of Rex Black's popular book Managing the Testing Process, 3e.

A number of RBCS clients find that obtaining good test data poses many challenges. For any large-scale system, testers usually cannot create sufficient and sufficiently diverse test data by hand; i.e., one record at a time. While data-generation tools exist and can create almost unlimited amounts of data, the data so generated often do not exhibit the same diversity and distribution of values as production data. For these reasons, many of our clients consider production data ideal for testing, particularly for systems where large sets of records have accumulated over years of use with various revisions of the systems currently in use, and systems previously in use.

However, to use production data, we must preserve privacy. Production data often contains personal data about individuals which must be handled securely. However, requiring secure data handling during testing activities imposes undesirable inefficiencies and constraints. Therefore, many organizations want to anonymize (scramble) the production data prior to using it for testing.

This anonymization process leads to the next set of challenges, though. The anonymization process must occur securely, in the sense that it is not reversible should the data fall into the wrong hands. For example, simply substituting the next digit or the next letter in sequence would be obvious to anyone it doesn’t take long to deduce that “Kpio Cspxo” is actually “John Brown”-which makes the de-anonymization process trivial.

Using Domain Analysis for Testing

by Rex Black

Many of you are probably familiar with basic test techniques like equivalence partitioning and boundary value analysis. In this article, Rex presents an advanced technique for black-box testing called domain analysis. Domain analysis is an analytical way to deal with the interaction of factors or variables within the business logic layer of a program. It is appropriate when you have some number of factors to deal with. These factors might be input fields, output fields, database fields, events, or conditions. They should interact to create two or more situations in which the system will process data differently. Those situations are the domains. In each domain, the value of one or more factors influences the values of other factors, the system's outputs, or the processing performed.

In some cases, the number of possible test cases becomes very large due to the number of variables or factors and the potentially interesting test values or options for each variable or factor. For example, suppose you have 10 integer input fields that accept a number from 0 to 99. There are 10 billion billion valid input combinations.

Equivalence class partitioning and boundary value analysis on each field will reduce but not resolve the problem. You have four boundary values for each field. The illegal values are easy, because you have only 20 tests for those. However, to test each legal combination of fields, you have 1,024 test cases. But do you need to do so? And would testing combinations of boundary values necessarily make for good tests? Are there smarter options for dealing with such combinatorial explosions?

This article was originally published in Quality Matters.

Advanced Software Test Design Techniques:  Use Cases

by Rex Black

The following is an excerpt from my recently-published book, Advanced Software Testing: Volume 1. This is a book for test analysts and test engineers. It is especially useful for ISTQB Advanced Test Analyst certificate candidates, but contains detailed discussions of test design techniques that any tester can—and should—use. In this third article in a series of excerpts, I discuss the application of use cases to testing workflows.

At the start of this series, I said we would cover three techniques that would prove useful for testing business logic, often more useful than equivalence partitioning and boundary value analysis. First, we covered decision tables, which are best in transactional testing situations. Next, we looked at state-based testing, which is ideal when we have sequences of events that occur and conditions that apply to those events, and the proper handling of a particular event/condition situation depends on the events and conditions that have occurred in the past. In this article, we’ll cover use cases, where preconditions and postconditions help to insulate one workflow from the previous workflow and the next workflow. With these three techniques in hand, you have a set of powerful techniques for testing the business logic of a system.

This article was originally published in Testing Experience Magazine.  Subscribe today!

Advanced Software Test Design Techniques, State Diagrams, State Tables and Switch Coverage

by Rex Black

In this article, we look at state-based testing.  State-based testing is ideal when we have sequences of events that occur and conditions that apply to those events, and the proper handling of a particular event/condition situation depends on the events and conditions that have occurred in the past.  In some cases, the sequences of events can be potentially infinite, which of course exceeds our testing capabilities, but we want to have a test design technique that allows us to handle arbitrarily-long sequences of events. Read this article to learn more about state-based testing.

This article was originally published in Testing Experience Magazine.  Subscribe today!

Advanced Software Test Design Techniques, Decision Tables and Cause-Effect Graphs

by Rex Black

This article is an excerpt from Rex Black's recently-published book, Advanced Software Testing: Volume 1.  This is a book for test analysts and test engineers.  It is especially useful for ISTQB Advanced Test Analyst certificate candidates, but contains detailed discussions of test design techniques that any tester can-­and should­-use.  In this first article in a series of excerpts, Black starts by discussing the related concepts of decision tables and cause-effect graphs.

Equivalence partitioning and boundary value analysis are very useful techniques.  They are especially useful when testing input field validation at the user interface.  However, lots of testing that we do as test analysts involves testing the business logic that sits underneath the user interface.  We can use boundary values and equivalence partitioning on business logic, too, but three additional techniques, decision tables, use cases, and state-based testing, will often prove handier and more effective.  Read this article to learn more about these powerful techniques. 

This software testing article was originally published in the June 2009 edition of Testing Experience Magazine. 

Risk Based Testing:  What It Is and How You Can Benefit

by Rex Black

Rex Black’s pioneering Managing the Testing Process was both the first test management book and the first to discuss risk-based testing.  In this software testing article, Rex explains:
• The benefits of risk-based testing. 
• Why adding risk analysis to the test team’s responsibilities actually reduces their workload.
• The importance of stakeholder participation.
• Common mistakes that can occur in risk-based testing.

Rex illustrates these points, not through hypothetical discussion, but by examining a case study where RBCS helped a client launch risk-based testing.  Read this article to learn how to analyze risks to quality, and use that analysis to be a smarter test professional.

Quality Goes Bananas

by Rex Black, Daniel Derr and Michael Tyszkiewicz

You're familiar with test automation, but what is a dumb monkey? How can it help you automate your testing and explore very large screen flows and input sets? Is it true that you can build dumb monkeys from freeware with no tool budget? What kind of quality risks can dumb monkeys address? Read this article to learn the answers to these and other test automation questions.

How Outsourcing Affects Testing

by Rex Black

This software testing article is excerpted from Chapter 10 of Rex Black's upcoming book Managing the Testing Process, 3e.  Over the last twenty years, outsource development of one or more key components in the system has come to dominate software and hardware systems engineering. The trend started in hardware in the 1990s. RBCS clients like Dell, Hitachi, Hewlett Packard, and other computer systems vendors took advantage of cheap yet educated labor overseas to compete effectively in an increasingly commoditized market. By the end of 2002, three years into a spectacular IT downturn that saw computer science enrollments in the United States fall to less than half of their 1999 levels, price had become the primary determinant in most IT project decisions.  Mass outsourcing of software projects took hold, and it continues unabated to this day... Read this article to get a picture of the effects of outsourcing software testing.

Intelligent Use of Testing Service Providers
by Rex Black

In this software testing article, Rex Black analyzes the use of outsourcing in testing, based on some twenty years of experience with outsourcing of testing in one form or another. First, Mr. Black enumerates the key differences between in-house and outsourced test teams. Next, driven by these key differences, he analyzes which tasks fit better with outsourced testing service providers, followed by a similar analysis for in-house test teams. Then, Mr Black lists some of the technical, managerial, and political challenges that confront a company trying to make effective use of outsourced testing. Finally, he addresses some of the processes needed to use testing service providers effectively and with the least amount of trouble. Read this article to significantly improve your use of outsourced testing.

A Case Study in Successful Risk-Based Testing at CA
by Rex Black, Peter Nash and Ken Young

This article presents a case study of a risk-based testing pilot project at CA, the world's leading independent IT management software company. The development team chosen for this pilot is responsible for a widely-used mainframe software product called CA SYSVIEWR Performance Management, an intuitive tool for proactive management and real-time monitoring of z/OS environments.  By analyzing a vast array of performance metrics, CA SYSVIEW can help organizations identify and resolve problems quickly.
CA piloted risk-based testing as part of our larger effort to ensure the quality of the solutions we deliver.  The pilot consisted of six main activities:
.         Training key stakeholders on risk-based testing
.         Holding a quality risk analysis session
.         Analyzing and refining the quality risk analysis
.         Aligning the testing with the quality risks
.         Guiding the testing based on risks
.         Assessing benefits and lessons
This article addresses each of these areas - as well as some of the broader issues associated with risk-based testing.  Click here to read the version of this software testing article as published in Better Software Testing.

Four Ideas for Improving Software Test Efficiency

"Do more with less. Work smarter not harder. Same coverage, fewer testers." If you're like a lot of testers and test managers, you'll be hearing statements like those a lot in 2009, since we appear headed for another tight economic period. If you need a way to demonstrate quick, measurable efficiency gains in your test operation, read this  short article to learn four great ideas that will help you improve your software test efficiency.

A Simplified Automation Solution Using WATIJ
by Steven Troy, Jamie Mitchell and Rex Black

One of our clients, CA, has continued to impress us with innovative ways to go about their testing. An upcoming software testing article will discuss how we are helping them institute risk-based testing. Read this article to learn how one of their teams is using a leading-edge open source testing tool, WATIJ, to help contain regression risk.

A Story about User Stories and Test-Driven Development
by Gertrud Bjørnvig, James O. Coplien, and Neil Harrison

Test-Driven Development, or TDD, is a term used for a popular collection of development techniques in wide use in the Agile community. While testing is part of its name, and though it includes tests, and though it fits in that part of the life cycle usually ascribed to unit testing activity, TDD pundits universally insist that it is not a testing technique, but rather a technique that helps one focus one’s design thinking. The idea is that you write your tests first, and your code second.

Read this article to explore some subtle pitfalls of TDD that have come out of our experience, our consultancy on real projects (all of the conjectured problems are things we have actually seen in practice), and a bit of that rare commodity called common sense.

The original two-part version of this article  about Test Driven Development was published in Better Software Testing. Click here to read the first part and click here for the second part.

The ISTQB Advanced Syllabus: Guiding the Way to Better Software Testing
by Rex Black

The International Software Testing Qualification Board (ISTQB) has already effected profound change in the software testing field, with almost 100,000 people having attained Foundation certification. But a Foundation certification is just that: only a Foundation. With the release of the new Advanced syllabus in October 2007, the ISTQB has expanded and improved the next rung on the ladder of test professionalism. In the slides from this tutorial, Rex Black, President of the ISTQB, shows how the ISTQB Advanced syllabus can guide you, your testing colleagues, and your organization toward better testing, reduced risk, and higher quality.

The IT Professional on the Outsourced Project
by Rex Black

More and more IT professionals work on projects where some or all of the development is done by third-parties, often overseas. While cost savings make such arrangements attractive to executives, individual contributors and managers on such projects face some significant challenges. What does outsource mean for IT professionals? In this talk, Rex Black offers insights from his extensive involvement in outsource projects ­both successful and not-so-successful. Rex will illustrate his points with case studies, and share humorous and scary anecdotes along the way.

The Right Stuff: Four Small Steps for Testers, one Giant leap for Risk Mitigation
By Rex Black and Barton Layne

Recently, we worked on a high-risk, high-visibility system where performance testing ("Let's just make sure it handles the load") was the last item on the agenda. As luck would have it, the system didn't handle the load, and very long days and nights ensued. Why it does have to be this way... Read this article about Risk Mitigation to ensure this doesn’t happen to you.

Empirix's QAZone with Rex Black
By Marina Gil Santamaria

From certification to automation, expert thoughts on where the testing industry is and where it's headed.

Quality Risk Analysis: Which Quality Risks Should We Worry About?
By Rex Black

Since it is not possible to test everything, it is necessary to pick a subset of the overall set of tests to be run. Read this article to discover how quality risks analysis can help one focus the test effort.

Component Outsourcing, Quality Risks, and Testing: Factors and Strategies for Project Managers
By Rex Black

More and more projects involve more integration of custom developed or commercial-off-the-shelf (COTS) components, rather than in-house development or enhancement of software. In effect, these two approaches constitute direct or indirect outsourcing of some or all of the development work for a system, respectively. While some project managers see such outsourcing of development as reducing the overall risk, each integrated component can bring with it significantly increased risks to system quality. Read this software testing article to learn about the factors that lead to these risks ,and strategies you can use to manage them.