Archive for the ‘software test outsourcing’ Category
Friday, March 11th, 2011 by Rex Black
In the last decade, outsourcing became a powerful force in the software industry. Motivations behind outsourcing vary, but the reason our clients mention most is that of cost savings. Unfortunately, all too often our clients also mention that previous attempts at outsourcing failed to deliver the desired efficiencies, or perhaps failed to deliver anything at all.
So, is outsourcing some siren on rocky project shores, luring to doom the captains of IT who dare to listen to the siren’s song? Not at all, but outsourcing is not without its risks. Over the last twenty years, I’ve worked on both sides of the outsourced IT relationship, and have seen it work. Let’s examine what successful outsourced efforts have in common.
Successful outsourcing involves planning and handling the unique logistical details of outsourcing. For example, e-mail and intranet communication, synchronized software lifecycles, procedures for file transfer, effective configuration management, support for development, test, and staging environments, sufficient test data, common tool usage, and compliance to applicable standards are necessary for success on many software projects. Project teams must understand the tactical details of how the work will get done, day-by-day, person-by-person, and resolve any logistical obstacles that could occur in advance. Good project logistics are like air and water: You don’t notice them until they’re bad or, worse yet, completely missing. However, outsourcing logistics are complex and often span organizational areas of responsibility (or even falls into gaps in areas of responsibility), problems happen often, and cause many outsourcing difficulties and failures.
Successful outsourcing also involves good working relationships with mutual trust and open communication. Studies show that simply locating people on separate floors in the same building can dramatically reduce communication and relationship building. Having people located thousands of miles and half a dozen or more time zones away is even harder on relationship building and maintenance. However, successful outsourcing requires that people actively nurture good working relationships across the organizational and geographical boundaries. If relationships are weak, trust is missing, and communication is infrequent, every project challenge becomes harder to deal with. In the long term, relationships sour and morale suffers. In addition to creating an emotionally-unpleasant working situation for everyone, quality and efficiency both go decrease.
Successful outsourcing requires understanding what CMMI does—and doesn’t—tell you about an outsource vendor’s capabilities. Properly applied, CMMI will lead to more orderly, consistent practices, which can increase quality and efficiency. We have clients who use CMMI to improve their processes, reduce costs, and deliver better software. That said, the jury is still out on whether there is a statistically valid and reliable correlation between CMMI levels and: 1) the cost per delivered KLOC (or Function Point); or, 2) the reliability or defect density of the delivered software. If that seems to contradict what I said about some of our clients, my point is that it is a logical fallacy to say that, since some companies have success with CMMI, therefore every company that achieves a high level of CMMI maturity will produce better, cheaper software than another company with a lower level of maturity. In addition, even Bill Curtis of the Software Engineering Institute, one of the fathers of CMMI, admitted (at the ASM/SM 2002 conference) that, when used purely as a marketing device, CMMI does not significantly improve quality or efficiency. So, if an organization says they are CMMI accredited (at whatever level), dig further to see exactly what that means in terms of their daily practices, and look at solid metrics for efficiency and quality.
This brings us to the final factor for successful outsourcing: selecting the right outsource service provider. As I mentioned above, just looking at CMMI levels won’t suffice, but even if you satisfy yourself that a vendor is mature, efficient, and delivering quality, remember, as an investment prospectus would say, past results are not necessarily an indicator of future performance. In other words, just because a vendor has had good results on past projects doesn’t mean they will succeed on your projects. Here are some other questions to consider:
- Who are the specific people who will work on your projects, how long have they been with the vendor, and what measures are in place to prevent turnover?
- Does the vendor have experience with technologies, business domains, and projects similar to yours?
- Does the company have sufficient infrastructure to handle your projects?
- If you anticipate growth in your use of the vendor, do they have the ability to scale up their staff without compromising the quality of their work?
- Are the technical and managerial leaders of the company well-established in the field, with a good reputation and perhaps even a record of thought-leadership?
- Are the vendor’s corporate culture, work habits, and ethical standards consistent and compatible with your own?
- Does the vendor have competencies in specialized areas such as testing or requirements engineering or should you consider using specialist vendors for those elements of the project?
If all this seems difficult and complex, keep two things in mind. First, even relatively small projects can have significant costs, especially opportunities costs, if they fail, so outsourcing is always a decision to be made with care. Second, in most cases the real efficiencies of outsourcing will only kick in after a few projects, so organizing for outsourcing success is worth doing well, because, if you do it right, you only have to do it once. Once you have established a successful working relationship with an outsourcing vendor, you will find yourself reaping the benefits, project after project.
Tags: offshoring, outsourcing, software test offshoring, software test outsourcing Posted in best practices, software test management, software test outsourcing, software testing | 1 Comment »
Monday, February 21st, 2011 by Rex Black
I had an interesting set of questions from a reader arrive in my inbox today. I’ve interleaved my answers with his questions, with “RB:” in front.
Dear Mr. Black,
Would you please comment on the following three questions, or perhaps direct me to where I might gain some meaningful information that addresses them?
What is today’s trend in pricing for the software testing industry i.e. is it increasing, decreasing, stable, etc.?
RB: There certainly are what marketers refer to as “value customers” who make service purchase decisions solely on price, and these customers continue to drive down pricing on average. However, at the top end, especially for clients that need and value senior consultants, we have managed to resist that.
Is the service looked at as value added or a commodity, with pricing accordingly?
RB: For the “value customer” mentioned above, it’s a commodity. For other customers, it’s really a matter of doing a good job of connecting what is happening in testing with strategic business objectives. I talked about this in my chapter in the book, Beautiful Testing. To the extent that testing is very tactical and inward focused–especially when the focus is almost entirely on finding a large number of potentially unimportant bugs–it will be seen as a commodity.
Given that much of the labor is offshore in India and China, and subject to increase as these countries develop, will market be receptive to required price increases to allow a reasonable margin?
RB: The “value customer” will not be receptive to such price increases, because price is all that matters. The value customer will try to have their cake and eat it, too, by raising the minimum bar of qualifications while not allowing price to rise. Because there are billions of under-utilized human brains in the world, and because technology has almost eliminated barriers to entry for using those brains as commodity software testers, the value customer will get to have their cake and eat it, too.
Best Regards,
Randy Francisco
SGS Consumer Testing Services
Randy, thanks for the questions. I talked about some matters relevant to these questions in my webinar on the Future of Test Management, which you can view here.
I’d be interested in other people’s comments. What do you think about these questions?
Tags: commodity testing, future of test management, outsource testing, test management Posted in software test management, software test outsourcing, software testing | 2 Comments »
Thursday, October 28th, 2010 by Rex Black
More and more projects involve more integration of custom or commercial off the shelf packages, rather than in-house development or enhancement of software. In effect, this is direct (under contract) or indirect (market purchase) outsourcing of some of the development work.
While some project managers see such outsourcing of development as reducing the overall risk, each integrated component can bring with it significantly increased risks to system quality. Let’s take a look at each factor that can increase risk to system quality, and then talk about strategies for mitigating such risks.
One factor that increases risks is coupling, which creates a strong interaction with the system—or consequence to the system—when the component fails. Another factor that increases risks is irreplaceability, when there are few similar components available. To the extent that the component creates quality problems, you are stuck with them. Yet another factor that increases risks is essentiality, where some key feature or features of the system will be unavailable if the component does not work properly. The final factor that increases risks is vendor quality problems, especially if accompanied by slow turnaround on bug fixes. If there is a high likelihood of the vendor sending you a bad component, the level of risk to the quality of the entire system is higher.
How can you mitigate these risks? I have seen and used various options.
One is to integrate, track, and manage the vendor testing of their component as part of an overall, distributed test effort for the system. This involves up-front planning, along with having sufficient clout with the vendor or vendors to insist that they consider their test teams and test efforts subordinate to and contained within yours. When I have used this approach, it has worked well.
Another option is simply to trust the vendor component testing to deliver a working component to you. This approach may sound silly and naive, expressed in such words. However, project teams do this all the time. My suggestion is, if you choose to do so, do so with your eyes open, understanding the risks you are accepting and allocating schedule time to deal with issues.
Another option is to decide to fix the component vendor testing or quality problems. On one project, my client hired me to do exactly that for a vendor. It worked out nicely. Again, though, your organization must have the clout to insist that you be allowed to go in and straighten out what’s broken in their testing process and that they have time allocated to fix what you find. And don’t you have your own problems to attend to? As such, this is an ideal job for a test consultant.
A final option, especially if you find yourself confronted by proof of incompetent testing by the vendor, is to disregard their testing, assume the component is coming to you untested, and retest the component. I’ve had to do this, notably on one project when the vendor sold my client an IMAP mail server package that was seriously buggy.
Both of the last two options have serious political implications. The vendor is unlikely to accept your assertion that their testing is incompetent, and will likely attack your credibility. Since someone made the choice to use that vendor—and it may have been an expensive choice—that person will likely also side with the vendor against your assertion. You’ll need to bring data to the discussion. Better yet, see if you can influence the contract negotiations up front to include proof of testing along with acceptance testing by your team prior to payment. It’s amazing how motivational that can be for vendors!
With the risks to system quality managed at the component level, it’s still possible to make a serious mistake in the area of testing. Remember that even the best-tested and highest-quality components might not work well in the particular environment you intend to use them in. So, plan on integration testing and system testing the integrated package yourself.
Tags: risk based testing, system integration, systems of systems Posted in risk based testing, software quality, software test outsourcing, software testing | No Comments »
Thursday, October 21st, 2010 by Rex Black
The smart test manager plans for the future. These plans should cover not only the current project, but also the current decade. How will you succeed as a test manager in the 2010s decade? Here are ten things you must learn to do:
- Connect testing to business value, including measuring effectiveness and efficiency against strategy goals;
- Manage testing on outsourced projects, including outsourcing of testing and outsourcing on Agile projects;
- Perform system integration testing on systems-of-systems projects effectively;
- Test systems that include open source software, and use open source tools;
- Test integration of new systems with legacy systems, and test the maintenance of legacy systems;
- Test effectively and efficientlywhen there’s too much testing work, too little time, and too few resources;
- Deal with the tester “skills gluts” that are created by outsourcing and crowd-sourcing, with millions of entry-level testers;
- Deal with the tester “skills shortages” that are created at the upper end of the skills triangle by these entry-level testers, especially in developing regions;
- Choose the right certifications, including security, tools, ISTQB, technology, and more;
- Manage testing on iterative and Agile projects.
The smart test manager who can do these ten things will be in a strong position to succeed as this decade unfolds. Hear more about the future of test management here.
Tags: agile testing, test management Posted in software test management, software test metrics, software test outsourcing, software test skills, software testing | 6 Comments »
Friday, August 20th, 2010 by Rex Black
I took a few moments today to record another video blog entry, which you can find at Regional Software Testing Immaturity: Fact or Fallacy?
Here’s the synopsis: I’m in Los Angeles, on my way to the STANZ conference in Australia and New Zealand. That and other recent international trips have gotten me thinking about something that I often hear at such international conferences, which are comments along the lines of, “You know, software testing as a profession and a practice is really immature in region X,” where region X might or might not be where I’m at. Based on my experience with clients around the world, though, the gap isn’t as big as people often think it is. Is software testing actually significantly less mature in some regions than others? What has your experience been? I’d be interested in opinions, case studies, and stories from you, especially the many international readers of this blog but also people in North America.
Tags: software test management, software testing, testing immaturity Posted in best practices, software test management, software test metrics, software test outsourcing, software testing | 5 Comments »
Sunday, July 25th, 2010 by Rex Black
It’s not just hype; it’s government policy. That phrase could describe Malaysia’s announced ambitions to become a software testing “hub.” Hub seems to mean a preferred software test outsourcing destination, but it might also have other connotations that are less obvious.
In his opening remarks at the SOFTEC 2010 conference, government minister Mohamed Yakcop made the point again and again. Come to Malaysia (or stay in Malaysia) and help us build the country into a worldwide, premier venue for software test outsourcing.
It’s a nice ambition for a country to have. As another fellow foreign testing dignitary who was sitting next to me as Yakcop spoke (and who shall remain nameless) asked after Yakcop’s speech, “Can they pull it off?” The answer to that question depends on a number of factors. With 20+ years of experience with outsource testing, I can identify some key enablers to outsourcing success that help a country establish itself as a powerhouse:
- People: Certainly the history of high tech outsourcing is about what is politely referred to as “labor rate arbitrage,” which, more franky expressed, means, lots of people who will work comparatively cheaply. Malaysia has about 30 million people, slightly smaller than the population of California.
- Education: Since software testing, like all software engineering, is brain work, said cheap workforce must also be an educated workforce. The Malaysian Software Testing Board has announced a goal of taking 10,000 people (including recent college grads) through the ISTQB certification program (using materials licensed from RBCS).
- Location: As I’ve said for years, outsourcing requires jet fuel. Key people will need to visit, in both directions, for outsourcing to work. Malaysia is a long trip from just about any North American or European city, but certainly there are plenty of flights to Kuala Lumpur. Malaysia’s not quite as clean as Singapore, but it’s as clean as Taiwan or China. And it is a charming place to visit, with friendly people and a culinary capacity that almost surpasses Taipei. Don’t come expecting to lose weight, unless you wire your mouth shut before arriving!
- Infrastructure: It’s really helpful to outsource to a country that works. Malaysia works, infrastructure-wise. The roads work. The internet works. The power works. The planes run on time. The airports don’t look like a cross between a landfill and the waiting area of the visitors lounge in a prison. You can safely drink the water in the parts of Malaysia that I’ve visited (you’ll never underestimate the value of potable water after you’ve had a bout of water-borne illness in a foreign country). Infrastructure is an enabler that is often missing from other countries, which can make doing business in those countries a real hassle. Malaysia’s infrastructure makes it about as easy to get work done as in any western country.
- Political stability: Outsourcing doesn’t require laissez-faire capitalism to work (cf., China for a case in point), but it does require a set of rules, enforced with reasonable transparency, that don’t change all the time. Anxiety about revolutions, full-scale civil war, or major hostilities with neighbors can create obstacles, or at least hesitancy, though Taiwan’s, Korea’s, and India’s success proves that the potential for war is not as scary to outside businesses as you might think it would be. Malaysia, a constitutional, parliamentary democracy, has been governed by the ruling party, UMNO, since it’s founding, and has maintained a business-friendly environment.
- Time: According to Malcolm Gladwell’s book, Outliers, it takes about 10,000 hours of practice to master any intellectually difficult field. Software testing certainly is intellectually difficult. 10,000 hours comes to about five years. This factor is probably one of the bigger challenges for Malaysia, given the relative paucity of experts currently available to mentor and lead the software testing hubfolk (hubizens? hubinots?), though there’s plenty of evidence that the market will overlook issues of experience if the price per hour is low enough. An attempt to overcome this challenge by importing foreign mentors was an subtext of Yakcop’s remarks, at least as I heard them.
- Anchors: As someone who has run an international consultancy for over 15 years now, I know that it really helps to have a few strong, anchor clients. The same is going to be true for Malaysia’s testing hub. Getting a few multinationals to establish testing centers of excellence in Malaysia and getting a few large, successful Malaysia testing service providers off the ground will be key.
- Buzz: Let’s face it. When it comes to following fads, the tech industry is second only to the fashion industry. (For example, check out Gartner’s venerable hype cycle.) If Malaysia obtains the industry buzz that India had in 2001 or that China has today, at least in terms of software testing, then they can probably expect to get all the software testing hubbiness that 30 million people can handle–and then some. Which brings us back to people and education, and the need to be ready to scale the software testing workforce quickly.
I’ve been to Malaysia about a dozen times now over the last few years. From what I’ve seen, from the hundreds of people I’ve talked to and trained there, and from the leaders behind this hub concept, it certainly looks like most of the enablers are there. Malaysia is a place to watch in the software testing industry, and it would be unwise to bet against people like Mohamed Yakcop and Mastura abu Samah, President of the Malaysian Software Testing Board. To paraphrase Lenin, the Malaysians involved in this Malaysian Software Testing Hub initiative are software testers in a hurry.
Tags: malaysia, software testing Posted in software test outsourcing | No Comments »
|