Archive for the ‘best practices’ Category
Tuesday, August 2nd, 2011 by Rex Black
Long-time reader Patricia Osorio asks about some material from our Advanced Test Manager e-learning course (and also from Advanced Software Testing: Volume 2):
1. In the section 6.4.1 Defect Removal Effectiveness of Reviews, I founded the following paragraph, Could you please explain me how to get the numbers [of remaining defects]?
First, imagine that you started with 1, 000 defects. You follow worst practices in reviews, but at least you review all types of items. In this case, you would enter testing with about 166 defects. Now, imagine that you started with 1, 000 defects again. However, this time you follow best practices (and again you review all types of items). This time, you go into testing with 3 defects.
The question refers to the following table:
| |
Least
|
Average
|
Most
|
| Requirements review |
20%
|
30%
|
50%
|
| High-level design review |
30%
|
40%
|
60%
|
| Functional design review |
30%
|
45%
|
65%
|
| Detailed design review |
35%
|
55%
|
75%
|
| Code review |
35%
|
60%
|
85%
|
So, if you follow the “least” column, after the requirements review, there are 800 defects remaining (20% removed), after the high level design review there are 560 defects remaining (30% removed), and so forth, with 166 the final figure for defects remaining. If you follow the “most” column, after the requirements review, there are only 500 defects remaining (50% removed), after the high level design review there are 200 defects remaining (60% removed), and so forth, with 3 the final figure for defects remaining.
Now, admittedly the real situation is a bit more complex than this. Defects are not introduced only in the requirements, but indeed in every work product. However, the basic point remains: the more defects are removed earlier in the lifecycle, the easier the job of testing.
Tags: software reviews, software testing Posted in ISTQB certification, best practices, software testing | 1 Comment »
Friday, July 22nd, 2011 by Rex Black
I received the following question from a reader of this blog, Kathleen Marrs. She wrote:
Rex,
I have a colleague who swears up and down about a standard test case and process document style guide that is supposedly endorsed by you and the ISTQB and ASTQB. The style that he has introduced makes extensive use of quotation marks to denote controls, fields, and form names.
E.g. Click on the “Orders” form, then click on the “Current Orders” tab, enter “Bob Smith” in the “Customer Name” field and click on the “Find” button.
To me this is not grammatically correct as quotation marks should be used for actual quotes, user entered text, single character entries or familiar words that are used in an unfamiliar context. Have you seen this writing style and do you endorse its use or has my colleague been misled?
I am a proponent of the standard that is used by MIT and Microsoft and others which looks like:
Go to Orders -> Current Orders tab -> Customer Name and enter “Bob Smith” and click OK
Your input on this matter would be greatly appreciated.
In my book, Managing the Testing Process, I discuss various styles and templates for documenting test cases or procedures. The ISTQB, in the Foundation and current Advanced syllabus, discusses the use of the IEEE 829 templates. However, I don’t recall addressing the stylistic point raised above in any of these documents. To me, it’s really a matter of choice. The test organization should standardize on a single approach and everyone should follow it. Otherwise, you get into the “Tower of Babel” problem.
What I would comment on is that the example given above is what is called a concrete test case, in the sense that all of the inputs and outputs are explicitly specified. This is certainly necessary for automated tests. For manual tests, you can consider what degree of detail must be captured in the test cases. In some situations, using logical test cases–where the type of input and output are described generally and the tester is allowed to use discretion, white-box and black-box test design techniques, and exploratory testing to select the specific inputs and to evaluate the results–is often more lightweight, effective, and maintainable. This issue of the degree of specificity in test cases is something covered in Managing the Testing Process.
Posted in best practices, software test templates, software testing | No Comments »
Thursday, July 21st, 2011 by Rex Black
Regular reader Gianni Pucciani wrote me recently to discuss risk based testing in an Agile world:
Dear Rex,
Risk based testing has been discussed in several places, and from different perspective, including in your books and online resources. There are almost no doubts about the benefits it can bring. What is still missing in my opinion is a good discussion about adopting risk based testing in an agile environment.
In this case risk identification and assessment should be performed at the beginning of each sprint, analyzing the risks connected to the features that will be developed in the coming sprint. Another point worth mentioning is the importance of test automation for regression testing, but what about a situation where most of the tests are manual? I would like to hear from you and the readers of your blogs if you have any experience/suggestions to share.
Thank you.
Best regards,
Gianni Pucciani
Yes, Gianni, this is exactly how risk based testing works in an Agile environment. Here’s summary of a risk based testing process document we created for a client who uses the Scrum methodology:
1. At the beginning of the planning period for a release, identify project risk based analysis team participants.
2. Schedule risk meetings (90-120 minutes each).
3. Prepare interview documents.
4. Hold interviews with all risk identification participants.
5. Analyze risk items.
6. Normalize risks.
7. Review project Quality Risk Analysis with stakeholders.
8. Apply RPN values to test planning and test case development.
9. When release backlogs are being determined and when sprint backlogs are being revised, At major project milestones, review and revise the risk analysis.
In terms of manual regression testing, given all the great tool support for test automation in Agile environments, I’m not sure why an organization would choose to do this. It’s the best way to manage regression risk in an Agile environment.
Tags: Agile, risk based testing Posted in best practices, risk based testing, software testing, test automation | 2 Comments »
Saturday, July 16th, 2011 by Rex Black
I recently had an experience with a couple important lessons for software quality and software testing. I ordered a pair of identical products from an e-commerce retailer. I’ll not distract the discussion by mentioning the specific products, or the problems with them, as it would entail a long, technical digression that detracts rather than enforces my point.
Anyway, the order fulfillment process went reasonably well. The retailer did an acceptable job of keeping me posted on the status of the order, which apparently involved working directly with the vendor making the products.
The two products finally arrived, and I immediately tested them out. I was not impressed with the quality of the products. Both exhibited an annoying failure each time they were used, and about 2% of the time they exhibited a failure that made the items not fit for purpose under certain circumstances. However, the products hadn’t cost much and I could imagine using them in non-critical situations, so I decided not to bother with returning them.
A week later, an e-mail popped into my inbox, from the retailer, asking me to review the product. Okay, it’s worth five minutes warning other potential customers about these problems. I write a review, click to submit it, and immediately see an error message. I go back, copy the review, and try it with another browser. Exact same problem.
So, now I’m a bit irritated. I send an e-mail to the retailer, advising them of the problem with their site and with the product. I get back a terse response from them, via their online support web page, saying, “It sounds to me like you got a defective [product]. What I would suggest is contacting the manufacturer of the item and most likely they will replace it for you.”
I then asked the retailer whether they were able to post my review on my behalf, as their system didn’t work. Here’s another terse response: “I don’t have a way to do that on my end, you can attempt that again.” I did try again–and this is now three days after the initial failure of their site–and it still doesn’t work.
Okay, I concluded, enough nonsense. I’ve now flipped the bozo bit on the retailer. A textbook case of how not to handle customer complaints about your website and the products you sell.
Okay, let’s compare this to best-of-breed processes for handling customer quality complaints. When I recently had a problem with a Blue Snowball microphone, the vendor worked with me to repair the microphone. When their repairs failed, Blue then sent me a new one for free, and apologized for my inconvenience.
As another example–not to blow our own horn too loudly–we (RBCS) have a policy that, whenever a customer has a problem using our website, we give them a discount code, even when the problem is not our fault (e.g., the recent failures due to a stealth software update by Pair, our site host). In fact, when people attending our free webinars have had problems with attending the webinar, we have given them discount codes.
So, what lessons can we learn as IT people, software professionals, and software testers specifically?
1. If you provide a means for customers to provide online feedback on your product or service, make sure it works! Testing should cover that feedback mechanism, and compatibility testing should be included in that, due to browser diversity.
2. If a customer reports a problem via an online feedback mechanism, make sure that the workflow includes tracking that problem to resolution, and test that workflow. Just telling the customer, “Tough crackers, take it up with the vendor,” is actually worse than not having an online feedback mechanism at all.
All in all, for me this experience has gone from the “that’s too bad, but not big deal” category to “I’ll never do business with that retailer again.” I’m sure that’s not the customer experience the retailer had in mind when they were designing their website, and, had they bothered to test it sufficiently, it probably wouldn’t have been the experience I had.
Tags: ecommerce testing, software testing Posted in best practices, software testing, test coverage | No Comments »
Monday, July 4th, 2011 by Rex Black
As long-time listeners–or even brand new listeners, for that matter–of the RBCS webinars know, we use Citrix’s GoToWebinar service for our free monthly webinars. Now, I’ve been fairly satisfied with GoToWebinar. I’ve used one or two of the competing services, and been less happy with those. Of course, webinar listeners (and readers of this blog) might remember I chided Citrix back in May for the ungraceful way the system handles audio drop-outs by the presenter.
So, during the June webinar, webinar attendee Keith Stobie reported an inability to see the presentation using Internet Explorer 9. He said that Chrome (not sure which version) worked just fine. I reported the problem to Citrix on Wednesday of last week. Five days later, I receive the following reply, quoted in its entirety (minus the links provided at the end):
Thank you for contacting Citrix Online Global Customer Support,
Dear Rex Black,
IE 9 has not been tested with any of our products as of yet. we will try to help fix any issues the best we can, but cannot guarantee anything. Hopefully we should get this done as soon as possible.
If you have any additional questions or need further clarification regarding this matter, please feel free to reply directly to this email. For any other product inquiries or technical assistance, please visit us at our Support Centers listed at the bottom of this email. Our Support Centers include Self Help files and our Global Customer Support Contact Information.
Thank you,
Richard Carrel | Global Customer Support
So, I appreciate the reply, though I have to say that five days isn’t quick turnaround for a customer complaint about a browser-based service that’s incompatible with a major vendor’s browser.
More surprising to me is the admission that Citrix hadn’t tested IE9. I don’t keep up with the browser wars, so I’m not sure what share of the browsing action IE9 has, but I’m pretty sure that Microsoft’s IE family of browsers remains at least one of the 800 pound gorillas in the room.
Putting myself in the position of the Director of Quality or VP of Testing or whatever the head-testing-honcho’s title is at Citrix, I understand that there are constraints on compatibility testing. I wouldn’t bother to test four-year-old versions of Opera, for example. But come on, not testing IE9? If I were in charge of testing for any SaaS provider, compatibility would be one of my top quality risks, and testing browser/OS/malware configuration combinations would receive a fair amount of time, money, and attention. Of course, functionality, reliability, performance, and security would also be high on the list of risk categories, too.
Here’s some free consulting advice to my fellow test professionals who work at Citrix: Spend a little time getting ramped up on how to do quality risk analysis and risk based testing. You can find lots of free resources on our web site, especially in the articles and the Digital Library. You’ll notice that compatibility is one of the quality risk categories included in our free quality risk checklist. If you need more help, let me know, as we can provide a one-week risk based testing bootstrapping service that will get you headed in the right direction.
Morale of the story: If you are in charge of testing at any SaaS vendor, and you’re not testing for compatibility, it’s only a matter of time before someone writes a blog post like this one about your product and the degree to which you aren’t testing it.
Tags: software compatibility, software testing Posted in best practices, risk based testing, software quality, software testing, test coverage | No Comments »
Sunday, June 26th, 2011 by Rex Black
Long-time reader Patricia Osorio Aristizabal sent the following question via e-mail (info@rbcs-us.com):
Hi Mr Black
I have a dilemma and I would like to know your thoughts about it.
It is clear for me the difference between error, defect, and failure (IEEE 610). Besides, the convenience of keeping in mind and use these words orally in order to help the project team – may be – to find out where or when an issue (incident) occurs. However, it is not easy to change all the team to use these words without cause resistance and incredulity to the importance of that difference. In your experience, how I could get to change first, my team and then all the organization to standardized the use of that concept? Could you please give some tips to do this change?
For those readers not familiar with the distinction to which Patricia alludes, here are the ISTQB Glossary definitions for each of those terms:
error (or mistake): A human action that produces an incorrect result.
defect (or bug): A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
failure: Deviation of the component or system from its expected delivery, service or result.
incident: Any event occurring that requires investigation. [Note: During testing, any failure that occurs would be an incident, though some incidents ultimately turn out to be false positives caused by things like bad test cases, bad test data, bad test environments, etc.]
So, based on these definitions, the programmer makes an error (or mistake) in their thinking while writing a program, leading to a defect in the code. The code is executed during testing and the defect produces a failure. The tester observes the failure, investigates the incident, and presumably files a bug report.
Certainly, having clear thinking in one’s mind–reflected by understanding the distinctions drawn–can help testers understand what’s going on. However, it can be difficult to convince people to adopt the definitions. As a consultant, I’m familiar with the challenge of trying to motivate people to make changes of any sort.
The general rule for change, in my opinion, derives from a basic rule of sales: People move away from pain much more quickly and reliably than they move toward a desirable situation. In other words, if you want to motivate change, make people aware of the organizational pain–waste, delay, frustration, etc.–associated with the behavior you want to change, and then suggest how to make that pain go away based on your proposed change.
So, Patricia, if you can find a way in which a lack of clarity on these definitions is causing waste, delay, frustration, or other sorts of organizational problems, then you might succeed at motivating changes here.
Tags: software testing, software testing best practices Posted in ISTQB certification, best practices, software testing | No Comments »
Monday, May 30th, 2011 by Rex Black
On the heels of the webinar last week, listeners have had lots of comments (all good) and some questions. Here’s an interesting set of questions from listener Stephen Ho. I’ve interspersed my answers in his e-mail, with “RB:” in front to make it easier to follow:
Rex,
Thanks for such Webinar. This webinar did not talk about how to organize and build up a good testing Metrics.
RB: There was a general discussion early on about how to go from an objective to a specific metric and specific targets for that metric. Perhaps you were missing the part about implementing the metrics with specific tools?
However, it provided some interesting points to measure the successful of a testing project, such as: BFE. Just for more realistic, how can we know that a testing metrics is good?
RB: One attribute of a good metric is that it is traceable back to some specific objective. That objective should relate to a process (e.g., finding defects is an objective for the test process), to a project (e.g., reaching 100% completion of all schedule tests is an objective for many projects), or to a product (e.g., reaching 100% coverage of requirements with passing tests is an objective for some products). Another important attribute is that the metric supports smart decision-making and, if necessary, guides corrective action. Yet another important attribute is that the metric have a realistic target.
At here, I have another topic for your interest.
“What is effective & efficiency testing?”
RB: We have to be more specific than this. What are the objectives for testing, as you mean it here? Once you have defined those objectives, you can then discuss effectively and efficiently meeting them. For example, if you define finding defects as an objective, then you can use the DDP metric (discussed in the presentation) as a metric of effectiveness. Cost of quality (which is discussed in various articles in the RBCS web site, such as this one) can serve as a metric of efficiency.
-how to narrow it down to know that our existing testing job is in effective & efficiency ways.
RB: You might want to read the chapter I wrote (Chapter 2) on this topic in the book Beautiful Software. That’s a book worth reading, anyway, because there a number of other good chapters in it.
-What are the right way to measure the performance of a QA?
RB: I assume you’re talking about an individual tester here. If we can define specific objectives for the tester, then we can use the same method to define metrics. Keep in mind the rule about objectives needing to be SMART.
-How can we know that a QA is in competence level?
RB: Check out my book, Managing the Testing Process, 3e, for a discussion about how to use skills inventories to manage the skills of your test team.
-How to increase the productivity of a QA?
RB: This question is too general, I’m afraid. Productive at what? I suggest that you define specific objectives for the test team, and then measure the current efficiency with which those objectives are achieved. At that point, you can make realistic (and measurable) goals for improvement of productivity.
You may have existing webinar or article regarding this topic. If yes, I am thirsty to study your material. Would you direct me how to access to this information. I would definitely provide my feedback to you.
RB: Follow this link for another article on metrics that you might find useful.
Thanks,
Stephen
RB: You’re welcome.
Tags: metrics, software testing effectiveness, software testing efficiency Posted in best practices, software test management, software test metrics, software test skills, software testing | No Comments »
Wednesday, April 13th, 2011 by Rex Black
Avid blog reader Din asks an interesting question via e-mail:
Hi Rex,
It’s me again. As I am reading a lot on gaining deeper understanding on different software development process or life cycle (or whatever others may call it) and match it back to standard testing process institutionalized by ISTQB, I need your feedback on the V-Model explained in ISTQB syllabus as well as the W-Model introduced by Paul Herzlich in 1993. As we learned on the static & dynamic testing in the syllabus where we involve in activities such as review of requirements and also actual execution of the test for system, I observe those practices represents total picture of applying W-Model. Looking forward for your feedback.
Regards,
Din
I think that the W-model and the V-model are essentially presenting the same set of activities, it’s just that the W-model shows them as additional nodes on the “inside” of the left and right sides of the V. In other words, on the V-model, in the “cross-arms” connecting the development activities on the left side with the test activities on the right side, labelled “develop tests” in the figure below, that would include the review of the various test basis documents which is shown explicitly in the W-model.

I think an important thing to remember about these lifecycle models–whether V-model, W-model, Agile model, iterative model, etc.–is encapsulated in a witticism attributed to W.E. Deming: “All models are wrong; some are useful.” What he meant by that, as I understand it, is that every model is a simplification of reality that omits some elements of reality. To the extent that the model helps you think more clearly about something you need to do or understand in the real world, that’s great; it’s a helpful model. To the extent that a model becomes dogma and interferes with real-world progress, then it becomes a problem. We see this happen with some of our clients from time to time, where following a model becomes more important than doing the right thing in terms of achieving the goals of the project.
So, Din, if the W-model helps you and your colleagues think more clearly and act more correctly on your projects–especially in terms of integrating testing into the lifecycle from beginning to end–then that’s great. Personally, I find a prefer the simply V-model diagram, as I understand that implicit in that diagram is testing involvement from the start of the project and also the principle that every work product should be subjected to both static and dynamic tests, starting as early as possible.
Posted in best practices, software testing | 1 Comment »
Saturday, March 19th, 2011 by Rex Black
Reader Gianni Pucciani has another good question in his review of the Advanced Software Testing: Volume 2 book. He asks:
My doubt is on question 4 of chapter 10 (People Skills and Team Composition).
The correct answer is B, and I had chosen B by excluding all the others which were for sure wrong.
However, my question is: how do you know that your team found 90% of defects by the time you need to give bonuses?
You know for sure the number of defects found prior to release, but how do you know the total number of defects if not after an agreed period (1 year?) of production use?
How would you implement this approach in a real life situation?
Here’s the question from the book:
You are a test manager in charge of system testing on a project to update a cruise-control module for a new model of a car. The goal of the cruise-control software update is to make the car more fuel efficient. Assume that management has granted you the time, people, and resources required for your test effort, based on your estimate. Which of the following is an example of a motivational technique for testers that will work properly and is based on the concept of adequate rewards as discussed in the Advanced syllabus?
A. Bonuses for the test team based on improving fuel efficiency by 20% or more
B. Bonuses for the test team based on detecting 90% of defects prior to release
C. Bonuses for individual testers based on finding the largest number of defects
D. Criticism of individual testers at team meetings when someone makes a mistake
Gianni is of course right, the answer is B. He is also right that there is some lag time after release required to calculate the defect detection effectiveness. Defect detection effectiveness is calculated as
DDE = (defects detected)/(defects present).
In the case of the final stage of testing, you can calculate this as
DDE = (defects detected in testing)/(defects detected in testing + defects detected in production).
The bottom side of that equation (the denominator) is a reasonably good approximation for “defects present” is you wait long enough.
So, how long is “long enough”? Most of our clients find that they can determine the typical period of time in which 90% of the defects will be reported on a given release, usually through analysis of the field failure information. In some organizations, this is as short as 30 days, though 90 days seems a more typical number.
Tags: defect detection effectiveness, test team bonuses Posted in best practices, software test management, software test metrics, software testing | No Comments »
Friday, March 11th, 2011 by Rex Black
In the last decade, outsourcing became a powerful force in the software industry. Motivations behind outsourcing vary, but the reason our clients mention most is that of cost savings. Unfortunately, all too often our clients also mention that previous attempts at outsourcing failed to deliver the desired efficiencies, or perhaps failed to deliver anything at all.
So, is outsourcing some siren on rocky project shores, luring to doom the captains of IT who dare to listen to the siren’s song? Not at all, but outsourcing is not without its risks. Over the last twenty years, I’ve worked on both sides of the outsourced IT relationship, and have seen it work. Let’s examine what successful outsourced efforts have in common.
Successful outsourcing involves planning and handling the unique logistical details of outsourcing. For example, e-mail and intranet communication, synchronized software lifecycles, procedures for file transfer, effective configuration management, support for development, test, and staging environments, sufficient test data, common tool usage, and compliance to applicable standards are necessary for success on many software projects. Project teams must understand the tactical details of how the work will get done, day-by-day, person-by-person, and resolve any logistical obstacles that could occur in advance. Good project logistics are like air and water: You don’t notice them until they’re bad or, worse yet, completely missing. However, outsourcing logistics are complex and often span organizational areas of responsibility (or even falls into gaps in areas of responsibility), problems happen often, and cause many outsourcing difficulties and failures.
Successful outsourcing also involves good working relationships with mutual trust and open communication. Studies show that simply locating people on separate floors in the same building can dramatically reduce communication and relationship building. Having people located thousands of miles and half a dozen or more time zones away is even harder on relationship building and maintenance. However, successful outsourcing requires that people actively nurture good working relationships across the organizational and geographical boundaries. If relationships are weak, trust is missing, and communication is infrequent, every project challenge becomes harder to deal with. In the long term, relationships sour and morale suffers. In addition to creating an emotionally-unpleasant working situation for everyone, quality and efficiency both go decrease.
Successful outsourcing requires understanding what CMMI does—and doesn’t—tell you about an outsource vendor’s capabilities. Properly applied, CMMI will lead to more orderly, consistent practices, which can increase quality and efficiency. We have clients who use CMMI to improve their processes, reduce costs, and deliver better software. That said, the jury is still out on whether there is a statistically valid and reliable correlation between CMMI levels and: 1) the cost per delivered KLOC (or Function Point); or, 2) the reliability or defect density of the delivered software. If that seems to contradict what I said about some of our clients, my point is that it is a logical fallacy to say that, since some companies have success with CMMI, therefore every company that achieves a high level of CMMI maturity will produce better, cheaper software than another company with a lower level of maturity. In addition, even Bill Curtis of the Software Engineering Institute, one of the fathers of CMMI, admitted (at the ASM/SM 2002 conference) that, when used purely as a marketing device, CMMI does not significantly improve quality or efficiency. So, if an organization says they are CMMI accredited (at whatever level), dig further to see exactly what that means in terms of their daily practices, and look at solid metrics for efficiency and quality.
This brings us to the final factor for successful outsourcing: selecting the right outsource service provider. As I mentioned above, just looking at CMMI levels won’t suffice, but even if you satisfy yourself that a vendor is mature, efficient, and delivering quality, remember, as an investment prospectus would say, past results are not necessarily an indicator of future performance. In other words, just because a vendor has had good results on past projects doesn’t mean they will succeed on your projects. Here are some other questions to consider:
- Who are the specific people who will work on your projects, how long have they been with the vendor, and what measures are in place to prevent turnover?
- Does the vendor have experience with technologies, business domains, and projects similar to yours?
- Does the company have sufficient infrastructure to handle your projects?
- If you anticipate growth in your use of the vendor, do they have the ability to scale up their staff without compromising the quality of their work?
- Are the technical and managerial leaders of the company well-established in the field, with a good reputation and perhaps even a record of thought-leadership?
- Are the vendor’s corporate culture, work habits, and ethical standards consistent and compatible with your own?
- Does the vendor have competencies in specialized areas such as testing or requirements engineering or should you consider using specialist vendors for those elements of the project?
If all this seems difficult and complex, keep two things in mind. First, even relatively small projects can have significant costs, especially opportunities costs, if they fail, so outsourcing is always a decision to be made with care. Second, in most cases the real efficiencies of outsourcing will only kick in after a few projects, so organizing for outsourcing success is worth doing well, because, if you do it right, you only have to do it once. Once you have established a successful working relationship with an outsourcing vendor, you will find yourself reaping the benefits, project after project.
Tags: offshoring, outsourcing, software test offshoring, software test outsourcing Posted in best practices, software test management, software test outsourcing, software testing | 1 Comment »
|