Some of  the readers of this blog have perhaps read or heard me say that the state of the common practice of software testing lags about 25 years behind the state of the art.  Here’s a situation which either is a case study in why I say that, or it’s something worse.

We (RBCS) used McAfee AntiVirus software for years on many of the PCs in our office as well as on company laptops.  This was mostly due to laziness, not a high level of satisfaction.  McAfee often came pre-installed on computers we bought, so we would simply renew the update subscriptions.  However, repeated problems with McAfee’s virus protection–especially its aggressive interference with e-mail programs–led us to switch to Trend Micro, which now runs on all our PCs.

On September 30, we received an e-mail from McAfee (specifically, an account called “subscriptions” at “mcafee.com”).  This e-mail read, in part, “ATTENTION: The credit card linked to your account for McAfee-based security products has expired. Please update your account now to keep your PC protected without interruption.”  After checking to make sure that we were no longer using McAfee software on any of our computers, we ignored the message.  After all, the credit card had expired.

It turns out that McAfee’s software had a bug in it that cause it to submit the charge anyway, because my October 13 American Express bill has a charge for $43.29 from McAfee.com.  No receipt was mailed for the charge, unlike in previous years, though I’m not sure this has anything to do with the expired card. We’re going to contest and reverse this charge, of course; it is a relatively minor hassle.

The testing implications of this situation are significant, though.  Remember, antivirus software, like other security software, is software that we rely on to keep systems safe.  Given the kind of damage a widespread interruption to computer systems due to virus attacks can cause, you’d expect the entire suite of software–including the online systems that provide updates and handle customer financial matters–to be well tested, following known best practices and techniques.

So, what’s the explanation for this situation?  If a credit card has expired, the system should not silently put through a charge on the card, especially when the system has sent an e-mail to the customer giving the impression that, unless the customer takes a specific and deliberate action to update the card, no charge will occur and the subscription will expire.   We have an obvious equivalence partition. Equivalence partitioning as a test technique is over 25 years old.  We clearly have a block of code that recognizes the equivalence partition and triggers an e-mail to the customer prior to the charge occuring.  Statement coverage, the lowest of the white box coverage criteria, is also a test technique that is over 25 years old.

In risk based testing–a topic I’ve covered often in this blog–it’s almost certain that two financial-related risks would be noted during quality risk identification:

• Processing a credit card transaction that should be rejected for processing
• Not processing a credit card transaction that should be accepted for processing

Best practices of risk based testing would generally lead to such risks given a high impact rating, even if the likelihood rating were low.  That would require more than cursory testing against such risks.  Now, analytical risk based testing as a strategy is relatively leading edge, having been perfected only in the last ten years.

Maybe someone at McAfee would care to post a comment about which of the following statements is true:

1. McAfee software systems receive only cursory testing, with financial processing code tested to less than 100% equivalence partition coverage and less than 100% statement coverage.  McAfee’s test team was not aware that the system attempts charges against credit cards it knows to be no longer valid.
2. McAfee’s test team was aware that the system attempts charges against credit cards it knows to be no longer valid.  A bug was reported against this behavior by the testers, and deliberately deferred or cancelled by the product or project manager because they decided that they wanted the additional revenue from former customers.

Of course, it’s quite possible that McAfee’s financial processing code is tested to less than 100% equivalence partition coverage and less than 100% statement coverage, but even so their testers found this bug.  After all, testing to force all possible messages to occur–a simple experience based technique recommended by James Whittaker in How to Break Software–would have revealed this bug as well.

In all, any of three well-established test design techniques–one black box (equivalence partitioning), one white box (statement coverage), and one experience based (the force-all-messages attack)–would have found this bug.  Risk based testing would have lead to more thorough coverage of the underlying quality risk. If financial related quality risks are not being tested using well-established best practices, then what else isn’t being tested in McAfee’s systems?