Posts Tagged ‘software testing’
Friday, October 14th, 2011 by Rex Black
Long time reader ML Gregory wrote me with the following concern:
Subject: To QA or not to QA, that is the question
I was looking for an article about QA in an agile world. I’m uneasy about a change our company is doing – eliminating QA. Testers who choose to stay must work in development as a developer writing tests. No test cases will be written, no bugs will be written. I’ve seen it before and was hoping to find some authoritative material on this type of change.
I’ve written about various ways Agile methodologies can challenge testing (most recently in the blog, but also in an article and a webinar). What you’re describing here, though, is a variant of the problem which is actually independent on Agile per se.
What you are describing is a problem I refer to as the Christmas Tree Ornament problem. I would be willing to bet that these three changes–no independent testers, no bug reports, no written tests cases–have been changes that people have wanted for some time. When the change to an Agile methodology came along, people hung these wishes onto Agile like a Christmas tree ornament.
It’s not unusual for these particular problems to occur with Agile, especially the “no bug reports” problem. Some Agile advocates do push for no tracking of bugs found during the iteration, which of course has the effect of making it impossible to gather metrics about the software process’s quality capability. However, I’ve seen other organizations that are definitely not using Agile have this same problem.
There’s not necessarily a lot you can do to resist these changes. They usually come with a lot of momentum. Standing in the way of an approaching train is not a good career move. Perhaps passing on the links I provided above, along with this blog post, can help.
I’d be interested in other thoughts from you and from other readers.
Tags: agile testing, software testing
Posted in best practices, software testing | 2 Comments »
Wednesday, October 12th, 2011 by Rex Black
Long time reader Patricia Osorio wrote to ask about the sample exam questions at the end of Chapter 7 in Advanced Software Testing: Volume 2. She’d like me to explain the answers.
Here are the questions:
Assume you are a test manager working on a project to create a programmable thermostat for home use to control central heating, ventilation, and air conditioning (HVAC) systems. In addition to the normal HVAC control functions, the thermostat also has the ability to download data to a browser-based application that runs on PCs for further analysis. During quality risk analysis, you identify compatibility problems between the browser-based application and the different PC configurations that can host that application as a quality risk item with a high level of likelihood. Your test team is currently executing compatibility tests.
Consider the following excerpt from the failure description of a compatibility bug report:
1. Connect the thermostat to a Windows Vista PC.
2. Start the thermostat analysis application on the PC. Application starts normally and recognizes connected thermostat.
3. Attempt to download the data from the thermostat.
4. Data does not download.
5. Attempt to download the data three times. Data will not download.
Based on this information alone, which of the following is a problem that exists with this bug report?
A. Lack of structured testing
B. Inadequate classification information
C. Insufficient isolation
D. Poorly documented steps to reproduce
The answer is C. We would expect, given the bug that’s being reported, that the tester would try other operating sytems.
The questions continue:
Continue with the previous scenario. Your test team is still executing compatibility tests. Consider the following excerpt from the failure description of a compatibility bug report:
1. Install the thermostat analysis application on a Windows XP PC.
2. Attempt to start the thermostat analysis application.
3. Thermostat analysis application does not start.
4. Re-install the thermostat analysis application three times. Thermostat analysis application does not start after any re-installation.
5. This test passed on the previous test release.
Based on this information alone, which of the following is the most reasonable hypothesis about this bug?
A. The bug might be a regression.
B. The bug might be intermittent.
C. The application didn’t install on Windows XP PCs before.
D. The bug might be a duplicate.
The answer is A. The bug is very likely a regression, because the test worked on the previous version of the software installed in the test environment.
Tags: software bugs, software defects, software testing
Posted in ISTQB certification, software testing | No Comments »
Friday, October 7th, 2011 by Rex Black
I received a request from Martin for help on how to apply orthogonal arrays. He wrote:
Hello,
I am working on putting together testing at work using an orthogonal array. This is the first time I’ve used this method. I read about it in detail in Rex’s books. But I’m having difficulty remembering all the steps. I need a little help to jog my memory. Could you please send me the high-level, bulleted, steps for creating the orthogonal array with my test case data?Thank you.
Sincerely,
Martin
Sure, Martin, you can take a quick listen to the key parts of my webinar on pairwise testing techniques. This webinar addresses how to use orthogonal arrays and other pairwise tools. After you listen to that, I’d suggest you go to www.pairwise.org to find a free downloadable tool. Last I heard, the US Department of Commerce had funded a project to create a pairwise tool (your tax dollars at work!) that will actually do classification trees as well, and I think you might find it there.
In general, Martin, if you are looking for a refresher on test design techniques, a good first stop is the RBCS Digital Library.
Tags: software test design, software testing
Posted in best practices, software test skills, software testing | No Comments »
Friday, September 30th, 2011 by Rex Black
I recently received an e-mail from a software tester, which raises some interesting issues about testing in an Agile environment. I’ve interspersed my comments with the original e-mail, using “RB:” to indicate my text.
Hi Rex, I am a ISTQB Foundation Level certified software tester and your book on Foundation level helped me clear the exam. Thanks for that. I never knew you had a website and visited it recently, infact today only.
RB: You’re welcome. I’m glad you found the book and the website useful. I hope you like the blog, too.
There is somethign that I want to know and I thought you might help. I am into manual testing and currently into a project which follows Agile methodology.
RB: Many people are in this situation these days, though you seem to be suffering from a bad implementation of Agile.
The problem that I face is, we are given new features to write test cases and test them once or twice every week but some of them do not have any specification at all. And when we raise our concern that we need some requirement or specification to identify our test cases etc , the functionality is explained in one liners over chat windows and the excuse that we are following Scrum methodology and hence we do not have any written specification or requirement for this feature.
RB: That’s not following Agile best practices, but we have run into situations with a signficant number of clients where not all of the Agile best practices are in place. Ideally, there are user stories (lightweight requirements) which are documented and managed. Techniques and tools for doing so differ, but what you describe is not sufficient for testing, and probably not for development either.
This is a classic mistake of interpreting a situation as a Manichean choice (i.e., one hand good, other hand evil), rather than understanding that the question is one of degree. In this case, the Manichean choice is being made with respect to the Agile principle that says to “value working software over complete specifications.” Notice that this does not say, “Don’t bother to write any specifications whatsoever,” but plenty of companies following Agile have interpreted it that way.
Another concern is, we do not have any test reporting template. We are just given features to test over email and we test and report them over email as well. But I would like to maintain at least some bare minimum documentation so that I have at least some record of what I have done which should also help me in keeping everybody on the same page.
RB: Yes, this is another challenge that happens in testing, and not only in Agile. The failure to record what was tested (both in terms of tests themselves as well as traceability back to the test basis) is common. As you point out, this leads to test results that are completely subjective and undocumented. Not only is this inadequate for release decisions, it’s completely useless in terms of process assessment.
What do you think should be my approach in such a scenario? Please advice.
RB: You are facing a lot of the challenges I first discussed over three years ago in a presentation on Agile testing challenges, which later became a webinar (http://www.rbcs-us.com/software-testing-resources/92) and also an article (http://www.rbcs-us.com/images/documents/agile-testing-challenges-english.pdf). I’d recommend you take a listen/look at those.
Thanks and Regards Neha Srivastava
RB: You’re welcome. Good luck. You might need to find a place where you can practice more effective and enlightened testing, because you’re unlikely to learn much other than bad habits from working in an environment like this one.
Tags: bad Agile implementation, effective software testing, software testing
Posted in best practices, software testing | 10 Comments »
Wednesday, August 24th, 2011 by Rex Black
We have lots of terms in software testing, but we’re not always clear what they mean. A common confusion is about the terms bug, defect, fault, failure, anomaly, incident, false positive, error, and mistake. Reader Rodrigo Cursino asks:
I read [Foundations of Software Testing]… You defined bug as a synonymous with fault or defect. If we exercise the part of software that contains that bug we’ll have a failure.
Now I’m reading your article “The Bug Reporting Processes” and I’m a little confused. Should the correct [title] be “The Failure Reporting Processes”. It’s because we report failures not bugs. The developers, based on the steps to reproduce the failure will be able to debug it and find the bug.
Rodrigo, you are right. This is a case of common usage prevailing of terminological rigor, I suppose. Many people talk about “bug reporting,” so I used that term in the article and in my book Critical Testing Processes.
If we want to be rigorous in our terminology, here’s the way to think about the sequence of events:
- The programmer makes a mistake (also called an error). This can be a misunderstanding of the internal state of the software, an oversight in terms of memory management, confusion about the proper way to calculate a value, etc.
- The programmer introduces a bug (also called a defect) into the code. This is the programmatical manifestation of the mistake.
- The tester executes the part of the software that contains the bug.
- If the test was properly designed to reveal the bug, the test can cause the buggy software to execute in such a way that the behavior of the software is not what the tester–who is closely observing the behavior–would expect. This difference between expected behavior and actual behavior is called an anomaly.
- The tester then investigates the anomaly to determine the exact failure. The failure may go beyond the obvious, immediately observable misbehaviors associated with the anomaly. For example, data might have been corrupted, another process improperly terminated, etc.
- The results of that investigation are a report, which is commonly referred to in the software business as a bug report, an incident report, a defect report, an issue, a problem report, or various other names.
- Whatever we call it, that report gets prioritized and (in some cases) ultimately routed to a programmer. The programmer then debugs the program in order to repair the underlying bug.
- Ideally, the bug fix (typically as part of a larger test release) comes back to the tester who reported the problem in the first place for a confirmation test. If the confirmation test passes, the report can be closed as fixed. If the confirmation test fails, the report should be re-opened.
Now, a few additional points are worth making:
- In some cases, when a tester runs a test, she observes an anomaly, but not due to a failure. This happens when the anomaly results from a bad test, bad test data, an improperly configured test environment, or simply a misunderstanding on the tester’s part. This situation is referred to as a false positive. Because some reports will inevitably be false positives–testers being human, they will also make mistakes–some people like to refer to them as incident reports. An incident is some situation that requires further investigation, and in this case the programmer will investigate whether the incident was really caused by a failure.
- Bugs (or defects) can be introduced into work products other than software. For example, a business analyst can put a bug into a requirements specification. Bugs in requirements specifications and design specifications (and code, for that matter) are ideally detected by reviews. When a bug is detected by a review (or by static analysis), notice that the bug is what is actually detected; the software is not executing, so no failure occurs.
- Some people use the word fault instead of bug or defect. I don’t like that term, and avoid it. When I talk about a fault, perhaps it sounds like I’m talking about something that is someone’s fault; i.e., implications of blame can arise. Bugs happen for various reasons, and individual carelessness is not at the top of the list. We should see bugs (and bug reports) as a way to understand the quality capability of the software process, not as a way to apportion blame.
So, back to Rodrigo’s bigger point: Does it really matter what we call the report? If you want to be 100% correct in your terminology, then incident report is probably the best name. However, I think that failure report is fine, too. I also think that, because these terms are so widely used, defect report and bug report are also acceptable. However, when using the terms defect report or bug report, it’s important that people keep in mind the sequence of events laid out above, and that the report actually describes the symptom of the bug, not the bug itself.
Tags: bug, defect, error, failure, fault, mistake, software testing
Posted in software quality, software test management, software testing | 2 Comments »
Tuesday, August 2nd, 2011 by Rex Black
Reader Patricia Osorio asks two good questions about a little-known test technique:
This is a question/suggestion. In the chapter 4 [of the Advanced syllabus], Test Techniques, I found Classification tree method technique. I know this technique…but it does not appear in the Foundation Syllabus. It appears in Advanced Syllabus. Why is this technique not included in Foundation Syllabus? Why it is classified as the same level of pair wise and orthogonal arrays? I have understood this technique as join application of BVA + Equivalence Partitioning + Error guessing + Defect based.
The answer to the first question, Patricia, is that this technique is really too sophisticated and complex to expect entry-level testers to master. Remember, the Foundation syllabus is meant to serve as an entry-level certification, such that people with no testing knowledge can take the training and exams as preparation to enter a testing career (or simply to be competent if posted temporarily in a testing role).
The answer to the second question is that pairwise testing techniques, include orthogonal arrays, are about testing combinations. That’s what classification trees allow you to do: define sophisticated types of combinations that you want to test. However, if you see affinities with other test techniques, that’s good; it probably means you’re thinking of ways to combine those techniques together and use them for specific problems, which is exactly what you want.
Tags: software test techniques, software testing
Posted in ISTQB certification, software testing | No Comments »
Tuesday, August 2nd, 2011 by Rex Black
Long-time reader Patricia Osorio asks about some material from our Advanced Test Manager e-learning course (and also from Advanced Software Testing: Volume 2):
1. In the section 6.4.1 Defect Removal Effectiveness of Reviews, I founded the following paragraph, Could you please explain me how to get the numbers [of remaining defects]?
First, imagine that you started with 1, 000 defects. You follow worst practices in reviews, but at least you review all types of items. In this case, you would enter testing with about 166 defects. Now, imagine that you started with 1, 000 defects again. However, this time you follow best practices (and again you review all types of items). This time, you go into testing with 3 defects.
The question refers to the following table:
| |
Least
|
Average
|
Most
|
| Requirements review |
20%
|
30%
|
50%
|
| High-level design review |
30%
|
40%
|
60%
|
| Functional design review |
30%
|
45%
|
65%
|
| Detailed design review |
35%
|
55%
|
75%
|
| Code review |
35%
|
60%
|
85%
|
So, if you follow the “least” column, after the requirements review, there are 800 defects remaining (20% removed), after the high level design review there are 560 defects remaining (30% removed), and so forth, with 166 the final figure for defects remaining. If you follow the “most” column, after the requirements review, there are only 500 defects remaining (50% removed), after the high level design review there are 200 defects remaining (60% removed), and so forth, with 3 the final figure for defects remaining.
Now, admittedly the real situation is a bit more complex than this. Defects are not introduced only in the requirements, but indeed in every work product. However, the basic point remains: the more defects are removed earlier in the lifecycle, the easier the job of testing.
Tags: software reviews, software testing
Posted in ISTQB certification, best practices, software testing | No Comments »
Saturday, July 16th, 2011 by Rex Black
I recently had an experience with a couple important lessons for software quality and software testing. I ordered a pair of identical products from an e-commerce retailer. I’ll not distract the discussion by mentioning the specific products, or the problems with them, as it would entail a long, technical digression that detracts rather than enforces my point.
Anyway, the order fulfillment process went reasonably well. The retailer did an acceptable job of keeping me posted on the status of the order, which apparently involved working directly with the vendor making the products.
The two products finally arrived, and I immediately tested them out. I was not impressed with the quality of the products. Both exhibited an annoying failure each time they were used, and about 2% of the time they exhibited a failure that made the items not fit for purpose under certain circumstances. However, the products hadn’t cost much and I could imagine using them in non-critical situations, so I decided not to bother with returning them.
A week later, an e-mail popped into my inbox, from the retailer, asking me to review the product. Okay, it’s worth five minutes warning other potential customers about these problems. I write a review, click to submit it, and immediately see an error message. I go back, copy the review, and try it with another browser. Exact same problem.
So, now I’m a bit irritated. I send an e-mail to the retailer, advising them of the problem with their site and with the product. I get back a terse response from them, via their online support web page, saying, “It sounds to me like you got a defective [product]. What I would suggest is contacting the manufacturer of the item and most likely they will replace it for you.”
I then asked the retailer whether they were able to post my review on my behalf, as their system didn’t work. Here’s another terse response: “I don’t have a way to do that on my end, you can attempt that again.” I did try again–and this is now three days after the initial failure of their site–and it still doesn’t work.
Okay, I concluded, enough nonsense. I’ve now flipped the bozo bit on the retailer. A textbook case of how not to handle customer complaints about your website and the products you sell.
Okay, let’s compare this to best-of-breed processes for handling customer quality complaints. When I recently had a problem with a Blue Snowball microphone, the vendor worked with me to repair the microphone. When their repairs failed, Blue then sent me a new one for free, and apologized for my inconvenience.
As another example–not to blow our own horn too loudly–we (RBCS) have a policy that, whenever a customer has a problem using our website, we give them a discount code, even when the problem is not our fault (e.g., the recent failures due to a stealth software update by Pair, our site host). In fact, when people attending our free webinars have had problems with attending the webinar, we have given them discount codes.
So, what lessons can we learn as IT people, software professionals, and software testers specifically?
1. If you provide a means for customers to provide online feedback on your product or service, make sure it works! Testing should cover that feedback mechanism, and compatibility testing should be included in that, due to browser diversity.
2. If a customer reports a problem via an online feedback mechanism, make sure that the workflow includes tracking that problem to resolution, and test that workflow. Just telling the customer, “Tough crackers, take it up with the vendor,” is actually worse than not having an online feedback mechanism at all.
All in all, for me this experience has gone from the “that’s too bad, but not big deal” category to “I’ll never do business with that retailer again.” I’m sure that’s not the customer experience the retailer had in mind when they were designing their website, and, had they bothered to test it sufficiently, it probably wouldn’t have been the experience I had.
Tags: ecommerce testing, software testing
Posted in best practices, software testing, test coverage | No Comments »
Monday, July 4th, 2011 by Rex Black
As long-time listeners–or even brand new listeners, for that matter–of the RBCS webinars know, we use Citrix’s GoToWebinar service for our free monthly webinars. Now, I’ve been fairly satisfied with GoToWebinar. I’ve used one or two of the competing services, and been less happy with those. Of course, webinar listeners (and readers of this blog) might remember I chided Citrix back in May for the ungraceful way the system handles audio drop-outs by the presenter.
So, during the June webinar, webinar attendee Keith Stobie reported an inability to see the presentation using Internet Explorer 9. He said that Chrome (not sure which version) worked just fine. I reported the problem to Citrix on Wednesday of last week. Five days later, I receive the following reply, quoted in its entirety (minus the links provided at the end):
Thank you for contacting Citrix Online Global Customer Support,
Dear Rex Black,
IE 9 has not been tested with any of our products as of yet. we will try to help fix any issues the best we can, but cannot guarantee anything. Hopefully we should get this done as soon as possible.
If you have any additional questions or need further clarification regarding this matter, please feel free to reply directly to this email. For any other product inquiries or technical assistance, please visit us at our Support Centers listed at the bottom of this email. Our Support Centers include Self Help files and our Global Customer Support Contact Information.
Thank you,
Richard Carrel | Global Customer Support
So, I appreciate the reply, though I have to say that five days isn’t quick turnaround for a customer complaint about a browser-based service that’s incompatible with a major vendor’s browser.
More surprising to me is the admission that Citrix hadn’t tested IE9. I don’t keep up with the browser wars, so I’m not sure what share of the browsing action IE9 has, but I’m pretty sure that Microsoft’s IE family of browsers remains at least one of the 800 pound gorillas in the room.
Putting myself in the position of the Director of Quality or VP of Testing or whatever the head-testing-honcho’s title is at Citrix, I understand that there are constraints on compatibility testing. I wouldn’t bother to test four-year-old versions of Opera, for example. But come on, not testing IE9? If I were in charge of testing for any SaaS provider, compatibility would be one of my top quality risks, and testing browser/OS/malware configuration combinations would receive a fair amount of time, money, and attention. Of course, functionality, reliability, performance, and security would also be high on the list of risk categories, too.
Here’s some free consulting advice to my fellow test professionals who work at Citrix: Spend a little time getting ramped up on how to do quality risk analysis and risk based testing. You can find lots of free resources on our web site, especially in the articles and the Digital Library. You’ll notice that compatibility is one of the quality risk categories included in our free quality risk checklist. If you need more help, let me know, as we can provide a one-week risk based testing bootstrapping service that will get you headed in the right direction.
Morale of the story: If you are in charge of testing at any SaaS vendor, and you’re not testing for compatibility, it’s only a matter of time before someone writes a blog post like this one about your product and the degree to which you aren’t testing it.
Tags: software compatibility, software testing
Posted in best practices, risk based testing, software quality, software testing, test coverage | No Comments »
Sunday, June 26th, 2011 by Rex Black
Long-time reader Patricia Osorio Aristizabal sent the following question via e-mail (info@rbcs-us.com):
Hi Mr Black
I have a dilemma and I would like to know your thoughts about it.
It is clear for me the difference between error, defect, and failure (IEEE 610). Besides, the convenience of keeping in mind and use these words orally in order to help the project team – may be – to find out where or when an issue (incident) occurs. However, it is not easy to change all the team to use these words without cause resistance and incredulity to the importance of that difference. In your experience, how I could get to change first, my team and then all the organization to standardized the use of that concept? Could you please give some tips to do this change?
For those readers not familiar with the distinction to which Patricia alludes, here are the ISTQB Glossary definitions for each of those terms:
error (or mistake): A human action that produces an incorrect result.
defect (or bug): A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
failure: Deviation of the component or system from its expected delivery, service or result.
incident: Any event occurring that requires investigation. [Note: During testing, any failure that occurs would be an incident, though some incidents ultimately turn out to be false positives caused by things like bad test cases, bad test data, bad test environments, etc.]
So, based on these definitions, the programmer makes an error (or mistake) in their thinking while writing a program, leading to a defect in the code. The code is executed during testing and the defect produces a failure. The tester observes the failure, investigates the incident, and presumably files a bug report.
Certainly, having clear thinking in one’s mind–reflected by understanding the distinctions drawn–can help testers understand what’s going on. However, it can be difficult to convince people to adopt the definitions. As a consultant, I’m familiar with the challenge of trying to motivate people to make changes of any sort.
The general rule for change, in my opinion, derives from a basic rule of sales: People move away from pain much more quickly and reliably than they move toward a desirable situation. In other words, if you want to motivate change, make people aware of the organizational pain–waste, delay, frustration, etc.–associated with the behavior you want to change, and then suggest how to make that pain go away based on your proposed change.
So, Patricia, if you can find a way in which a lack of clarity on these definitions is causing waste, delay, frustration, or other sorts of organizational problems, then you might succeed at motivating changes here.
Tags: software testing, software testing best practices
Posted in ISTQB certification, best practices, software testing | No Comments »
|
Rex Black is President of RBCS (
