In order to design, develop, and run tests, you need what’s often referred to as a test oracle, something that tells you what the expected, correct result of a specific test should be. Specifications, requirements, business rules, marketing road maps, and other such documents frequently play this role. However, what if you receive no formal information that explains what the system under test should do?

In some organizations with mature development processes, the test department will not proceed without specifications. Because everyone expects to provide a specification to the test team as part of the development process, you are seen as reasonable and within the bounds of the company’s culture when you insist on written specs.

Trouble arises, however, if you stiffen your neck this way in a company that operates in a less mature fashion. Depending on your company’s readiness to embrace formal processes (and also on your personal popularity, tenure, and political clout), any one of a spectrum of outcomes could occur:

• Your management, recognizing the need to formalize processes, backs you up 100 percent and institutes formal requirements- and design-specification processes throughout the organization as part of the planning phase of every new development project. Industry-standard templates for internal product documentation become the norm, and consultants are brought in to train people.
• Your management, not knowing quite how to handle this odd demand, assumes that you must know what you’re talking about. The dictate goes out to all the organization’s groups to support you, but since no one has any training in formal development processes, the effort produces poor-quality documents that don’t help. Furthermore, because the effort is (rightly) seen as a waste of time, people are upset with you for bringing it up.
• Your management listens to your demand but then explains that the company just isn’t ready for such cultural and process shifts. Perhaps things will change after the next few products go out, they speculate, but right now, process just isn’t on the to-do list. Besides, this product is really critical to the success of the company, and taking big chances on unproven ways of doing things would be too risky. You are told to get back to work.
• You are fired.

The moral of this story is that you should carefully consider whether your company is ready for formal processes before you insist on requirements or design specifications and other accoutrements of mature development projects.

This situation is exacerbated by the increasing popularity of Agile methodologies. Some of the people behind Agile approaches have endorsed a set of principles that include the following, “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” As you can imagine—if you are not already dealing with this situation—such a principle discounts the role of written specifications, encouraging instead an on-going dialog about the correct behavior of a system. In other words, the outcomes of discussions between stakeholders represent correct behavior. These discussions can happen in a meeting, but for some RBCS clients following Agile methods these discussions happen as one-on-ones between developers and another stakeholder.

I’m all for ongoing dialog between project stakeholders. However, unless there are written minutes from these discussions, agreed upon by all project stakeholders, including the test team, this approach has the risk that different people come to different conclusions about what the outcome of the discussion was. Of course, this is a big testing challenge if you weren’t part of the discussion.

Even more challenging is the need to deal with the possibility that at any point the definition of correct behavior can change. The Agile principles include this, saying, “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.” While I can see the business value in being able to shape software like modeling clay up until the last minute, it’s hard to create and maintain test cases if the definition of correctness is continually re-defined.

When properly applied, Agile models can bring some advantages to the test team. However, they also bring challenges, not the least of which is the tendency to devalue solid, stable, documented definitions of correct behavior.

If you have to deal with a situation where the project team cannot or will not deliver written specifications in advance, during test development, you might consider the following options for testing without specifications:

If you are testing a commercial product, remember that you have the benefit of competitors. Because your customers will expect your product to behave substantially like the products of your competitors, these competitive products are, in a sense, your test oracle. In compatibility test labs, for example, most projects have a reference platform—a competitor’s system, against which the system under test is being positioned, in the hope of demolishing it in the marketplace.

If your technical colleagues won’t tell you what the product should do, perhaps your friends in sales and marketing will. In my experience, sales and marketing people live to create glitzy presentations showing where the product line is going. Although they can be general and imprecise, these documents might tell you which features and capabilities the product should support. If you’re testing a product for which questions about supported features are harder to answer than questions regarding correct behavior, these documents might suffice for a somewhat vague but useful oracle.

Ask your customer-support colleagues. Your colleagues in customer support might not have much information about what the product should do, but they probably know what they don’t want the product to do. Since your testing stands between them and the hellish scenario outlined in the previous section, they are usually happy to tell you.

Unless the product is truly unique, you can use inductive reasoning to figure out what constitutes reasonable expectations and correct behavior in many cases. The generic categories into which products fit tell you a lot about what the products are supposed to do: a word processor, a Web browser, a PC, a laptop, a server, an operating system. Some esoteric questions might arise, but a core dump, a system crash, a burning CPU, garbage on the screen, an error message in the wrong language, and abysmal performance are indisputably bugs.

If in doubt, you should consider any suspect behavior buggy. Because you don’t have a crisp way of determining pass and fail conditions, you will make mistakes in result interpretation. Remember that calling correct behavior a bug and working through the bug life cycle is less detrimental to product quality than failing to report questionable behavior that does turn out to be a bug. Be sure to file bug reports when questions arise.

One thing to keep in mind about this situation is that you are definitely not alone. Many people are struggling with the right amount of documentation to gather, and errors are made on both sides. I try to maintain an open mind, even though the 20-questions approach to defining expected results is somewhat frustrating. It is a good idea, if you’re working in a poorly specified situation, to make sure management understands that your test development will be less efficient due to the need to pull information from other groups. My usual rule of thumb is that the lack of clear requirements and design specifications imposes a 20- to 30-percent inefficiency on test development, and I estimate accordingly.