software testing services
Rex Black Consulting Services | software testing experts providing consulting, outsourcing, and software training
CALL US TODAY
(866) 438-4830
ISTQB certification testingISTQB certification testing ISTQB certification testing
PMI

Archive for the ‘software test management’ Category

Successful Software Test Outsourcing

In the last decade, outsourcing became a powerful force in the software industry.  Motivations behind outsourcing vary, but the reason our clients mention most is that of cost savings.  Unfortunately, all too often our clients also mention that previous attempts at outsourcing failed to deliver the desired efficiencies, or perhaps failed to deliver anything at all.

So, is outsourcing some siren on rocky project shores, luring to doom the captains of IT who dare to listen to the siren’s song?  Not at all, but outsourcing is not without its risks.  Over the last twenty years, I’ve worked on both sides of the outsourced IT relationship, and have seen it work.  Let’s examine what successful outsourced efforts have in common.

Successful outsourcing involves planning and handling the unique logistical details of outsourcing.  For example, e-mail and intranet communication, synchronized software lifecycles, procedures for file transfer, effective configuration management, support for development, test, and staging environments, sufficient test data, common tool usage, and compliance to applicable standards are necessary for success on many software projects.  Project teams must understand the tactical details of how the work will get done, day-by-day, person-by-person, and resolve any logistical obstacles that could occur in advance.  Good project logistics are like air and water: You don’t notice them until they’re bad or, worse yet, completely missing.  However, outsourcing logistics are complex and often span organizational areas of responsibility (or even falls into gaps in areas of responsibility), problems happen often, and cause many outsourcing difficulties and failures.

Successful outsourcing also involves good working relationships with mutual trust and open communication. Studies show that simply locating people on separate floors in the same building can dramatically reduce communication and relationship building.  Having people located thousands of miles and half a dozen or more time zones away is even harder on relationship building and maintenance.  However, successful outsourcing requires that people actively nurture good working relationships across the organizational and geographical boundaries.  If relationships are weak, trust is missing, and communication is infrequent, every project challenge becomes harder to deal with.  In the long term, relationships sour and morale suffers.  In addition to creating an emotionally-unpleasant working situation for everyone, quality and efficiency both go decrease.

Successful outsourcing requires understanding what CMMI does—and doesn’t—tell you about an outsource vendor’s capabilities.  Properly applied, CMMI will lead to more orderly, consistent practices, which can increase quality and efficiency.  We have clients who use CMMI to improve their processes, reduce costs, and deliver better software.  That said, the jury is still out on whether there is a statistically valid and reliable correlation between CMMI levels and: 1) the cost per delivered KLOC (or Function Point); or, 2) the reliability or defect density of the delivered software. If that seems to contradict what I said about some of our clients, my point is that it is a logical fallacy to say that, since some companies have success with CMMI, therefore every company that achieves a high level of CMMI maturity will produce better, cheaper software than another company with a lower level of maturity.  In addition, even Bill Curtis of the Software Engineering Institute, one of the fathers of CMMI, admitted (at the ASM/SM 2002 conference) that, when used purely as a marketing device, CMMI does not significantly improve quality or efficiency.  So, if an organization says they are CMMI accredited (at whatever level), dig further to see exactly what that means in terms of their daily practices, and look at solid metrics for efficiency and quality.

This brings us to the final factor for successful outsourcing: selecting the right outsource service provider.  As I mentioned above, just looking at CMMI levels won’t suffice, but even if you satisfy yourself that a vendor is mature, efficient, and delivering quality, remember, as an investment prospectus would say, past results are not necessarily an indicator of future performance.  In other words, just because a vendor has had good results on past projects doesn’t mean they will succeed on your projects.  Here are some other questions to consider:

  • Who are the specific people who will work on your projects, how long have they been with the vendor, and what measures are in place to prevent turnover?
  • Does the vendor have experience with technologies, business domains, and projects similar to yours?
  • Does the company have sufficient infrastructure to handle your projects?
  • If you anticipate growth in your use of the vendor, do they have the ability to scale up their staff without compromising the quality of their work?
  • Are the technical and managerial leaders of the company well-established in the field, with a good reputation and perhaps even a record of thought-leadership?
  • Are the vendor’s corporate culture, work habits, and ethical standards consistent and compatible with your own?
  • Does the vendor have competencies in specialized areas such as testing or requirements engineering or should you consider using specialist vendors for those elements of the project?

If all this seems difficult and complex, keep two things in mind.  First, even relatively small projects can have significant costs, especially opportunities costs, if they fail, so outsourcing is always a decision to be made with care.  Second, in most cases the real efficiencies of outsourcing will only kick in after a few projects, so organizing for outsourcing success is worth doing well, because, if you do it right, you only have to do it once.  Once you have established a successful working relationship with an outsourcing vendor, you will find yourself reaping the benefits, project after project.

Software Testing Adequacy

A quick follow-up related to my earlier post on evidence.  As some readers may know, avionics software that controls flight on airplanes (e.g., cockpit software) is subject to a test coverage standard, FAA DO-178B. That standard applies lower standards of test coverage to software that is not safety critical.

So far, so good.

Here’s an example of why such standards are useful.  During my flight from the US to China today, I managed to crash the entertainment software running at my seat not once by three times.  I did this by pausing, rewinding, and resuming play when the flight attendants were taking my dinner orders (i.e., not by unusual actions).  I was ultimately able to get it working again, thanks to a series of hard reboots by a flight attendant.  One of my fellow passengers wasn’t so lucky, as his system never recovered.

Okay, that’s just entertainment, and anyone who travels regularly knows they should bring a book or plan to winnow down their sleep deprivation balance on long flights. 

However, what if the flight control software were as easy to crash?  Who would want to hear a cockpit announcement along the lines of the following:  “Our entire flight control system just crashed.  This enormous airliner is now essentially an unpowered and uncontrolled glider.  We’ll reboot the system until we get it working again, or until we have an uncontrolled encounter with terrain”?

Personally, I want people testing the more safety-critical aspects of avionics software to adhere to higher standards of coverage, and to be able to provide evidence of the same.

Software Testing Evidence

I recently received an interesting e-mail from a colleague:

To Whom It May Concern-

Do you have any articles on the value of collecting/capturing detailed test evidence (e.g., screenshots attached to test cases)?

In my opinion, for mature systems with experienced, veteran testers, the need for an abundance of test evidence in the form of screenshots attached to test runs in QC is overkill and unecessary that adds more time to release cycles. The justification for this is awlays “For Audit” as opposed to “Improves Quality”. I looked in several articles on this fantastic site, and couldn’t find anything pertaining to test evidence. Do you have any articles that provide evidence that an abundance of test evidence improves quality (even if it’s just a correlation and not necessarily causation)?

Thanks

Erik Tuininga

We have clients that do need to retain such detailed software testing evidence; e.g., clients working in safety critical systems (such as medical systems) who must satisfy outside regulators that all necessary tests have been run and have passed.  For them, retaining such evidence is a best practice, as not doing so can result in otherwise-worthy systems being barred from the market due to the lack of adequate paperwork. 

As someone who relies on such systems to work–indeed, as we all do–I appreciate these regulations and would not want to see software held to a lesser standard. However, Erik makes a very valid point in terms of the trade-off.  As time is spent on these audit-trail activities, that is time not spent doing other tasks that would perhaps result in a higher level of quality.  Of course, these audit-trail activities are designed to ensure that all critical quality risks are addressed.  So, the key question is how should organizations balance the risk of failing to test certain critical quality attributes against the reduction in breadth of quality risk coverage?

I’d be interested in hearing from other readers of this blog on their thoughts.  Erik, if you have further comments on this matter, I’m sure the readers of this blog would benefit from those ideas, as this is clearly an important area to consider.  I certainly agree it’s an interesting topic for an article, and this blog discussion may well inspire me to collaborate with you and other respondents to write one.

Moving Software Testing Up in Price and Value

I had an interesting set of questions from a reader arrive in my inbox today.  I’ve interleaved my answers with his questions, with “RB:” in front. 

Dear Mr. Black,

 Would you please comment on the following three questions, or perhaps direct me to where I might gain some meaningful information that addresses them?

What is today’s trend in pricing for the software testing industry i.e. is it increasing, decreasing, stable, etc.?

RB: There certainly are what marketers refer to as “value customers” who make service purchase decisions solely on price, and these customers continue to drive down pricing on average.  However, at the top end, especially for clients that need and value senior consultants, we have managed to resist that. 

Is the service looked at as value added or a commodity, with pricing accordingly?

RB:  For the “value customer” mentioned above, it’s a commodity.  For other customers, it’s really a matter of doing a good job of connecting what is happening in testing with strategic business objectives.  I talked about this in my chapter in the book, Beautiful Testing.  To the extent that testing is very tactical and inward focused–especially when the focus is almost entirely on finding a large number of potentially unimportant bugs–it will be seen as a commodity.

Given that much of the labor is offshore in India and China, and subject to increase as these countries develop, will market be receptive to required price increases to allow a reasonable margin?

RB:  The “value customer” will not be receptive to such price increases, because price is all that matters.  The value customer will try to have their cake and eat it, too, by raising the minimum bar of qualifications while not allowing price to rise. Because there are billions of under-utilized human brains in the world, and because technology has almost eliminated barriers to entry for using those brains as commodity software testers, the value customer will get to have their cake and eat it, too. 

Best Regards,

Randy Francisco

SGS Consumer Testing Services

Randy, thanks for the questions. I talked about some matters relevant to these questions in my webinar on the Future of Test Management, which you can view here.

I’d be interested in other people’s comments.  What do you think about these questions?

Metrics and Bonuses

One of the topics I find very interesting and useful for our clients is the proper use of metrics.  We do a lot of metrics-related engagements, and in fact just this morning I’ll be talking with a client about some US$ 100 million in defect-related waste that we’ve found in their software development process.  I’ve written a lot on the topic, including in my books and in various articles

Regular blog reader Gianni Pucciani asks an interesting metrics question in an e-mail:

The question is: how can you give a bonus to your test team, to motivate it, based on 90% of bugs found before the release to production date? How can you know that you found 90% of the bugs at the time you release the software?

Gianni is referring to a metric called defect detection effectiveness or defect detection percentage.  This is a metric I’ve discussed quite a bit in my books, especially Managing the Testing Process.

Defect detection effectiveness is a very useful metric for measuring the effectiveness of a test process at defect detection.  Most testing processes have defect detection as a primary objective, and we certainly should have effectiveness and efficiency metrics for objectives.

That said, it is a retrospective metric that can only be calculated some time after a release, if you intend to calculate it on a release-by-release basis.  (Some of our clients calculate it on an annual basis, aggregating all their projects together, which also works.)  It’s typical to wait 90 days after a release to calculate defect detection effectiveness, though you really should verify what time period is required to have say 80% or more of the field-reported defects.

I could go on for days about this metric, but, since it’s a blog and since Gianni asked a specific question, I’ll address the other point he brought up, which is the use of this metric for bonuses.  Defect detection effectiveness is a process metric, which is not the same as a metric of individual or collective performance.  Many things are required to enable good defect detection effectiveness, including good testers, and many things can reduce defect detection effectiveness, some of which are beyond the control of testers.  I’d encourage a web search on the string “Deming red bead black bead experiment” for a discussion on the risks of rewarding or punishing based on metrics that might not be entirely in the individuals’ control.

In addition, while defect detection is typically a primary objective of testing, it’s not the only objective, and defect detection effectiveness is only an effectivness metric.  It doesn’t measure the efficiency or the elegance with which the test team detects defects.  A test process should have a fully articulated set of objectives, with effectiveness, efficiency, and (ideally) elegance metrics for each objective, rather than a single unidimensional metric by which it is measured.

For further information on defect detection effectiveness, I’d refer people to my book Managing the Testing Process, 3e.  My colleague Capers Jones also contributed an article to our web site on a couple related defect metrics that readers might find interesting.

Cost of Poor Software Quality: $242,000,000

The Financial Times today featured an article on how a software bug–abysmally handled–in a financial application cost the company US$ 242,000,000:

http://www.ft.com/cms/s/0/5e1ba340-2feb-11e0-a7c6-00144feabdc0.html#axzz1D2IDiwLs

Because I don’t know how long that link will live, here’s the summary.

Axa Rosenberg Group had some quantitative analysis software that it used to service its clients accounts.  Axa Rosenberg Group manages money for other people, and the software is an internal application, albeit one they touted as a key differentiator, apparently–and indeed it did turn out to be, though not in a happy way.

The software had a bug that disabled a key risk-management component of the software, which was released to production in 2007.  Apparently management found out about the bug in November 2009.  However, rather than fix the problem, they tried to cover up the reasons for the poor performance of their funds.

Over one third of their customers were affected by the bug.

A wee bit of analysis from yours truly:  I have clients in the financial world, and I know how hard it can be to test these kinds of applications.  When a calculation is wrong, it can be wrong in a way that is beyond the ability of a human tester to detect.  However, Axa Rosenberg Group’s handling of the bug after they found out about it is truly a textbook illustration of how not to handle a software quality problem.

Test Environments

My colleague Gianni Pucciani wrote recently to suggest a discussion:

I would like to propose a discussion on your blog, about how to manage the testing environment when multiple testers are running tests concurrently, basically sharing the test environment. In my organization we rely heavily on virtualization, therefore each tester has it’s own installation of the system under test on a separate virtual machine, and there are no concurrency issues. I was wondering whether this is a standard practice and how this issue was managed when virtualization software was not used as much as now.

This is a great topic for discussion.  Certainly, many of our clients are using virtualization to try to insulate testers from each other, and also to insulate manual and automated environments.  Probably the worst train wrecks that I’ve seen, from a test environment perspective, related to unvirtualized environments shared across manual and automated tests.

Of course, in some cases the systems under test only read data from shared repositories, which prevents the concurrency problem Gianni mentioned.  In other cases each instance of the system under test (one instance per tester) has its own data for reading and writing, which also avoids the problem.

So, how about other readers of the blog?  What have you done to deal with the problems that can arise with parallel testers, testing in the same hardware environments at the same time, or with concurrent manual and automated testing in the same hardware environments?  How has the much-ballyhooed cloud affected this issue, if at all?

Advanced Test Manager: A Bug Found

It was bound to happen: Sharp-eyed reader Gianni Pucciani caught a bug in the Advanced Software Testing: Volume 2 book he is using to prepare for the ISTQB exam.

Question 15: You are a test manager in charge of system testing on a project to update a cruise-control module for a new model of a car. The goal of the cruise-control software update is to make the car more fuel efficient.

You have written a first release of the system test plan based on the final requirements specification. You receive an early draft of the design specification. Identify all of the following statements that are true.

A. Do not update the system test plan until the final version of the design specification is available.

B. Produce a draft update of the system test plan based on this version of the design specification.

C. Check this version of the design specification for inconsistencies with the requirements specification.

D. Participate in the final review of the design specification but not any preliminary reviews of the design specification.

E. Review the quality risk analysis to see if the design specification has identified additional risk items.

The answer key in the book says that A, C, and E are correct answers, but, as Gianni pointed out to me, the right answer is B, C, and E.  As he explained, “My reasoning was following the ‘test early’ principle, so even if the design is not complete, the information in there could help preparing the testing activities, especially if your are short of time and trust the design team.”  That is, of course, correct.  Nice catch, Gianni.

Risk Based Software Testing: Better Ways to Report Results

 As readers of this blog will know, I’ve spent a lot of time in this blog (and in articles, books, and consulting work) on the topic of risk based testing.  I recently spent some time in Japan, working with various teams in Sony to implement better risk based testing.  Part of that time was spent working with Atsushi Nagata, who is helping teams in Sony put risk based testing  into action.  Together, he and I have written an article describing some ground-breaking work that they’ve done on risk based test results reporting.  That article was just published in ST&QA magazine.  You can read it (and comment on it) by clicking here.

Processes (Not Just Software Testing Processes), Enabled by Tools

Often, software engineering processes–including but not limited to software testing processes–are made more efficient by tools, or in some cases are only enabled by the use of a tool.  When the tool is missing, the process breaks down.  The dependency–and thus the breakdown–might not be as obvious as shown in the picture below; sometimes you have to think harder about the problem.

What Is Missing?

What Is Missing?



 
`