Archive for the ‘software quality’ Category
Tuesday, April 3rd, 2012 by Rex Black
Following today’s webinar on test strategies, listener Cees Jan had some follow-up questions. I’ll address them below, with answer interspersed in the text.
Rex,
I really enjoyed your webinar.
Thanks, I’m glad it was useful for you.
Here is a question you did not get to: Hi I am what would qualify as a “falsifier”: Find the worst-case scenario and see how product behaves in terms of robustness and graceful termination. Quite the opposite of regression testing which I consider “trivial pursuit” (good for build stability and sanity, but never uncovering new defects). Would that be a wise strategy? I get comments that it is far-fetched, but I am stretching the limits of a system and see where it fails. It provides a lot of eye-openers. CeesJan
It sounds like, based on your earlier question the webinar (which I did answer), that you are basically following a reactive test strategy. You are apparently (based on this comment) focused on finding a lot of bugs, especially by checking out unusual circumstances and extreme conditions, which isa typical characteristic of reactive testing.
I’m a bit concerned about omitting regression testing, though. Far from being “trivial pursuit” (though I realize it’s not fun when done manually), in Agile lifecycles the high rate of change makes regression risk very significant. If you are no addressing regression risk, who is? Are the developers doing good automation unit (e.g., with J-unit) and feature verification tests (e.g., with Fitnesse)? If not, then the confidence-building and risk-mitigating objectives of testing are probably not being met.
The above sounds like reverse engineering: lack of requirements and undocumented legacy code force me to this “out of the box” methodology/averse strategy. Reading books like black box testing (Beizer) and black swan (Taleb) inspired me too much, I guess.
Certainly, in testing we have to play the hand we’re dealt. If you have no requirements and no useable documentation, then this will often force you into a purely reactive test strategy. However, you should be aware of how limited such a strategy is. I would think that adding analytical risk based testing, along with addressing the regression risks, would make sense.
Regards, Cees Jan den Haan (IQSTB Practitioner, CAT)
I have to correct you here, Cees Jan. The ISEB Practitioner certification is not part of the ISTQB program, and neither is CAT. The ISTQB program has the Foundation, Advanced, and Expert certifications. I know that some European training providers have created confusion on this point (perhaps for marketing purposes), but it’s wrong to associate those two certifications with the ISTQB.
Tags: agile testing, test strategies
Posted in software quality, software tester certification, software testing | No Comments »
Monday, March 26th, 2012 by Rex Black
I had a query come in about a sample exam question in our Advanced Test Manager course. Shukti asked me to confirm the answer to the following:
A given organization is using reviews for development work products like code, requirements and design specifications; test work products like test plans, quality risk analyses, and test design specifications; and, documentation and help screens. The review processes have been in place for two years and are delivering excellent financial, quality, and schedule benefits.
You are attending a management team meeting. A senior executive raises the need to update the objectives by which the individual contributors are measured on their yearly performance evaluations. He suggests using defect counts from review meetings. He circulates a draft plan. Under the plan, people will be rewarded based on the number of defects they find in reviews. Further, people will be penalized if items they have produced incur too many defects during reviews.
Which of the following is a psychological factor affecting review success and failure that is likely to cause such an initiative to undermine the current success of the reviews?
- Scrutinize the document and not the author.
- Focus all participants on delivering high-quality items. [Correct]
- Try to find as many defects as possible.
- Assemble the right team of reviewers.
Here’s why that answer is correct. Bonuses and other financial incentives/disincentives are based on the assumption that people are basically rational economic actors who will behave in a way to maximize their financial situation. (Now, we can have a whole separate discussion about whether this assumption holds perfectly in all situations, but it really doesn’t need to be perfect, as long as more often than not it is true.)
So, in this scenario, what will the reviewers be focused on? Not on delivering the highest quality items, but on finding the maximum number of bugs in each item and “claiming” those bugs for themselves (i.e., squabbling with other reviewers over who should get credit.) What will the authors be focused on? Arguing about every single bug that reviewers report, trying to insist that the document is perfect. None of these behaviors is supportive of increased quality, and the authors’ behaviors are directly contrary to that goal.
In short, a really bad idea. I wish I could say that I never saw organizations make this kind of mistake with process metrics (i.e., mistaking a software process metric for a people metric), but unfortunately it is all too common.
Tags: metrics, people metrics, process metrics, reviews
Posted in reviews, software quality, software test management, software test metrics, software testing | No Comments »
Sunday, March 25th, 2012 by Rex Black
Are you a happy software tester? According to this article, those of us in this profession are among the happiest, most satisfied employees. I certainly enjoy the intellectual challenges of this career, and it’s rewarding, too, to be involved in such important work.
What makes you happy about being a software tester? Do you consider yourself a happy tester? Share your comments with me and with other readers of this blog.
Tags: happy software testers, software testing
Posted in software quality, software testing | 1 Comment »
Thursday, October 13th, 2011 by Rex Black
Last week, Steve Jobs died, and I published an encomium to him in this blog. Today, I was saddened to read of the death of Dennis Ritchie, one of the greatest software innovators that most people have never heard of, but who had arguably as large an impact on our world as Steve Jobs did. I learned much of what I know about how software works by reading his book on C programming and by applying what I learned from that book to programming Unix systems in the 1980s and 1990s.
Shortly after I read about his death, I received an excellent post from Lauren Weinstein, relating his own thoughts and experiences about Dennis Ritchie’s inestimable contributions to the software world. I asked Lauren if I could quote his post, and he graciously agreed. Here follow Lauren’s fine remarks:
A Few Brief Thoughts on the Death of Computer Science Titan Dennis Ritchie http://lauren.vortex.com/archive/000903.html
Dennis Ritchie has died after a long illness ( http://j.mp/pQfU8J [CNET]).
He was only the relatively young age of 70. Unlike Steve Jobs, Dennis was not a household name. He should have been, but he was a quiet and private person, and would have hated the publicity.
I’ve known Dennis since the early days of the UNIX operating system at Bell Labs in the 70s, where he created the ubiquitous “C” programming language, and co-created UNIX with Ken Thompson.
It is impossible to overstate the importance of this work to today’s world. UNIX was almost entirely written in “C” — and UNIX’s direct descendants like Linux power Google, the Web at large (the vast majority of Web servers run Linux), and likely multiple devices in your home and office right now. The current Mac OS. Android. TiVo. The impact of C and Linux are everywhere. Your ability to read these words rests directly to a major degree on Dennis’ work.
But beyond the nitty-gritty of software design, the creation of UNIX was also the inception of what would ultimately become the open source community, and the vast collaborations of the ARPANET/Internet that have led to the global phenomenon we see today.
Such goals were explicit in the design of UNIX. In this 1980s Bell Labs video from my collection, featuring Ken and Dennis explaining the origins of UNIX, Dennis explains how they wanted the system to specifically foster community and fellowship.
I said this would be brief. I’ll close with one personal story. Back in the early UNIX days, on one of my visits to Bell Labs’ main facilities in Murray Hill, New Jersey, I was sitting at a terminal in the 1127 Labs where UNIX was developed, logged in over the nascent Net
back to UCLA. As usual, I was trying to get a bit of coding done even on this trip.
My rapid typing suddenly stopped as I puzzled over a particularly complex C language declaration in the program, that definitely was incorrect as it stood.
Sitting at the terminal next to me was Dennis. So I asked his advice on the declaration — after all, who better to know than the creator of the language?
He thought about it for no more than a few seconds, then immediately (and graciously) provided an elegant solution that, frankly, would not have occurred to me. I got on with my programming.
In later years, I realized that this exchange was probably the closest I’d ever come in my life to asking a question directly to, and receiving a detailed answer from, a true god.
Rest in peace, Dennis.
–Lauren–
Founder:
Member: ACM Committee on Computers and Public Policy
Tel: +1 (818) 225-2800 / Skype: vortex.com
Lauren, thanks for permission to re-post this. Dennis, thank you very much for what you taught me through your book and your work, and for what you did to advance software engineering. May we always be so fortunate as to work in an industry with such generous hearts and brilliant minds as Dennis had.
Tags: Dennis Ritchie
Posted in software quality | No Comments »
Friday, October 7th, 2011 by Rex Black
Long time reader, Patricia Osorio Aristizabal, asked me to clarify the following question:
During a walkthrough meeting for a high-level design, a tester who is participating as a reviewer asks the system architect who created the document why only physical network connections are supported in the design, rather than also supporting wireless connections. The designer replies that they did not have time to address the security concerns raised by wireless access to the system. The tester accepts that answer and the review proceeds.
Which of the following is an objective of a review that was achieved in that exchange?
A. Increased reviewer understanding
B.Early detection of a defect
C. Correction of a defect
D. Analysis of system design
The correct answer is A. The tester now has a better understanding of why the system is built the way it is. The design of the system is not defective, as there is a deliberate reason for not having WiFi support, so C is not the right answer.
Tags: software reviews
Posted in software quality, software testing | No Comments »
Friday, September 30th, 2011 by Rex Black
This week, I’m pleased to feature a bonus bug of the week, courtesy of long-time reader John Singleton. He wrote to tell us:
Rex,
I was posting some comments in response to your blog post, entitled “Bad Agile Implementation Causes Software Test Challenges”. For whatever reason, the Submit button was entirely non-responsive in Firefox. Thankfully, switching over to IE resolved it.
Not griping, just thought you’d want to know. Good, thoughtful response you gave.
Cheers,
John
I responded:
Hi John–
That’s weird. It’s usually the other way around with WordPress (the software we use for our blog): Firefox works when IE doesn’t. I guess we have a bonus bug of the week. 
Glad you liked the post. I’m looking forward to your comments.
Regards,
Rex
So, while this isn’t a very serious bug, it is part of a very serious pattern: Failure of major players in the web world to test browser compatibility. See my previous posts http://bit.ly/mdBIRY and http://bit.ly/mPkD9z. Is this a lack of understanding of basic test design techniques such as equivalence partitioning and (for particularly risky situations) pairwise testing? Or just a lack of caring?
I have to say that this pattern is not universal. Working with a major e-commerce client this week, I was pleased to see that they include testing of most of the major browsers, covering the equivalence partitions.
What do you think about this pattern? Are you in the web world, and, if so, do you test browser compatibility? If not, why?
Tags: browser incompatibility, equivalence partitioning, pairwise testing, web testing
Posted in best practices, compatibility testing, software quality, software testing, test coverage | No Comments »
Thursday, September 29th, 2011 by Rex Black
As those of you who receive our newsletter know, we had a little glitch this week. The newsletter greeted readers with “Dear (Contact First Name)”, when of course it should have given the reader’s first name rather than the parenthesized stuff.
Immediately–given the 14,000+ good testers who read our newsletter–we received many reports of the defect. A massive self-head-smack and “Doh!” action (a la Homer Simpson) ensued at the RBCS mothership. We promptly issued a 20% discount code to our readers as a way of apologizing for–well, someone forgetting their names.
So, who forgot their names? We use ConstantContact for our e-mail, so one logical explanation is a bug in ConstantContact’s mail merge code. We suspect that is the cause, because a test e-mail sent to Laurel Becker, RBCS VP, included the “Dear Laurel” string, but in the wrong place.
That said, there’s the “never attribute to malice that which can be explained by human error” angle to this. Did we accidentally mangle the newsletter template in a way that overrode the mail merge function? It’s certainly a possibility, and we can’t tell from the source newsletter whether that is the case. This is a bug in ConstantContact’s editing tool, because it’s possible to damage these mail merge keywords in such a way that you can’t see the mistake in either the source letter or the test copies.
So, the conclusion at this end is: It’s a mystery. We apologize to our readers again for appearing to have forgotten their names. We’ll keep an eye on this for future newsletters.
In the meantime, this is…uh, yeah, Rex, that’s right…Rex here reporting a possible bug…or just an inattentive copy-paste action somewhere in the process.
Tags: software defect, software quality
Posted in software quality, software testing | No Comments »
Thursday, September 8th, 2011 by Rex Black
This post is part of my all-too-regular series of posts on bugs that have vexed, frustrated, or inconvenienced me, my clients, or the world in general. It is a series that shows no sign of abating, sadly.
This week we had two bugs, one fairly nasty, one less so but still irritating.
To start with the nasty, as some readers will know, we use Articulate to author and host our e-learning. I’ve used other authoring and hosting tools in the past, and found them all relatively immature and buggy. Articulate is better than the others I’ve used, but by no means good enough, as this latest incident illustrates.
Articulate did a “scheduled update” on Saturday. They were nice enough to let us know about it a week beforehand, so we could warn our e-learning customers about downtime. In the past, we’ve had “stealth updates” where they didn’t tell us and then all of a sudden something stopped working; they would then deny that they had done an update and insist that we must have changed something.
However, Articulate apparently wasn’t nice enough to thoroughly regression test the update before they put it into production. As a result, all of our podcast attachments (MP3 audios of the e-learning courses) are corrupted. Re-attaching them doesn’t work, so we had to file a bug report with Articulate.
Did I say they didn’t regression test the update? Wait, I’m wrong, they are regression testing it, using their paying customers to do so. Why bother with equivalence partitioning the support attachments and testing that they all work when you can just promote untested code into production and make your paying customers do free work for you as testers? What’s most frustrating about this incident is that this is certainly not the first time it’s happened, and Articulate is by no means alone in employing this infamous tactic.
Now, on to the irritating. I did a webinar today. Over 400 people showed up, which was great. What wasn’t so great was that a few of them had a weird bug affect them. They reported not seeing the slides, but only the white welcome page with a greeting from me and my picture. As attendee James Heeren, who was affected by the bug, put it:
A funny thing happened when I closed the Webinar Window, a second window appeared with the picture of the graph of defects reported and defects closed. [This was a slide in the webinar presentation.] The real thing was hidden behind the white page somehow.
Was not a big problem for me as I followed on the deck from the website.
If it were just one person, I chalk it up to someone moving the focus to another tab in their browser by accident. Since it was quite a few people, it sounds like what was happening was that the GoToWebinar software was not properly getting the focus on the presentation window in some cases.
I have some sympathy for Citrix in this case. This sounds like a configuration-specific bug, which is really hard to find. That said, even though we will report it, I’m not expecting a lot of help from Citrix, based on past results of reporting bugs to them.
Until there is some implied warranty rule for software, as there is for every other product and services sold in the US, this kind of nonsense will continue.
Tags: Articulate, Citrix, e-learning, GoToWebinar, software bugs, software quality, webinar
Posted in software quality, software testing | No Comments »
Wednesday, August 24th, 2011 by Rex Black
We have lots of terms in software testing, but we’re not always clear what they mean. A common confusion is about the terms bug, defect, fault, failure, anomaly, incident, false positive, error, and mistake. Reader Rodrigo Cursino asks:
I read [Foundations of Software Testing]… You defined bug as a synonymous with fault or defect. If we exercise the part of software that contains that bug we’ll have a failure.
Now I’m reading your article “The Bug Reporting Processes” and I’m a little confused. Should the correct [title] be “The Failure Reporting Processes”. It’s because we report failures not bugs. The developers, based on the steps to reproduce the failure will be able to debug it and find the bug.
Rodrigo, you are right. This is a case of common usage prevailing of terminological rigor, I suppose. Many people talk about “bug reporting,” so I used that term in the article and in my book Critical Testing Processes.
If we want to be rigorous in our terminology, here’s the way to think about the sequence of events:
- The programmer makes a mistake (also called an error). This can be a misunderstanding of the internal state of the software, an oversight in terms of memory management, confusion about the proper way to calculate a value, etc.
- The programmer introduces a bug (also called a defect) into the code. This is the programmatical manifestation of the mistake.
- The tester executes the part of the software that contains the bug.
- If the test was properly designed to reveal the bug, the test can cause the buggy software to execute in such a way that the behavior of the software is not what the tester–who is closely observing the behavior–would expect. This difference between expected behavior and actual behavior is called an anomaly.
- The tester then investigates the anomaly to determine the exact failure. The failure may go beyond the obvious, immediately observable misbehaviors associated with the anomaly. For example, data might have been corrupted, another process improperly terminated, etc.
- The results of that investigation are a report, which is commonly referred to in the software business as a bug report, an incident report, a defect report, an issue, a problem report, or various other names.
- Whatever we call it, that report gets prioritized and (in some cases) ultimately routed to a programmer. The programmer then debugs the program in order to repair the underlying bug.
- Ideally, the bug fix (typically as part of a larger test release) comes back to the tester who reported the problem in the first place for a confirmation test. If the confirmation test passes, the report can be closed as fixed. If the confirmation test fails, the report should be re-opened.
Now, a few additional points are worth making:
- In some cases, when a tester runs a test, she observes an anomaly, but not due to a failure. This happens when the anomaly results from a bad test, bad test data, an improperly configured test environment, or simply a misunderstanding on the tester’s part. This situation is referred to as a false positive. Because some reports will inevitably be false positives–testers being human, they will also make mistakes–some people like to refer to them as incident reports. An incident is some situation that requires further investigation, and in this case the programmer will investigate whether the incident was really caused by a failure.
- Bugs (or defects) can be introduced into work products other than software. For example, a business analyst can put a bug into a requirements specification. Bugs in requirements specifications and design specifications (and code, for that matter) are ideally detected by reviews. When a bug is detected by a review (or by static analysis), notice that the bug is what is actually detected; the software is not executing, so no failure occurs.
- Some people use the word fault instead of bug or defect. I don’t like that term, and avoid it. When I talk about a fault, perhaps it sounds like I’m talking about something that is someone’s fault; i.e., implications of blame can arise. Bugs happen for various reasons, and individual carelessness is not at the top of the list. We should see bugs (and bug reports) as a way to understand the quality capability of the software process, not as a way to apportion blame.
So, back to Rodrigo’s bigger point: Does it really matter what we call the report? If you want to be 100% correct in your terminology, then incident report is probably the best name. However, I think that failure report is fine, too. I also think that, because these terms are so widely used, defect report and bug report are also acceptable. However, when using the terms defect report or bug report, it’s important that people keep in mind the sequence of events laid out above, and that the report actually describes the symptom of the bug, not the bug itself.
Tags: bug, defect, error, failure, fault, mistake, software testing
Posted in software quality, software test management, software testing | 2 Comments »
Tuesday, August 2nd, 2011 by Rex Black
Reader Patricia Osorio had another good question about the answer to the following question from our Advanced Test Manager e-learning course. Here’s the question:
A given organization is using reviews for development work products like code, requirements and design specifications; test work products like test plans, quality risk analyses, and test design specifications; and, documentation and help screens. The review processes have been in place for two years and are delivering excellent financial, quality, and schedule benefits.
You are attending a management team meeting. A senior executive raises the need to update the objectives by which the individual contributors are measured on their yearly performance evaluations. He suggests using defect counts from review meetings. He circulates a draft plan. Under the plan, people will be rewarded based on the number of defects they find in reviews. Further, people will be penalized if items they have produced incur too many defects during reviews.
Which of the following is a psychological factor affecting review success and failure that is likely to cause such an initiative to undermine the current success of the reviews?
- Scrutinize the document and not the author.
- Focus all participants on delivering high-quality items.
- Try to find as many defects as possible.
- Assemble the right team of reviewers.
The correct answer is the second option. Instead of focusing on delivering high-quality items, the reviewers will focus on trying to claim as many defects found in reviews as their own, and will raise as defects trivialities, in order to try to maximize their rewards, while the authors will try to refute every defect report, even the most serious, in order to try to minimize their punishments. We have seen this play out in real life with clients, so you must be careful.
Tags: software reviews
Posted in ISTQB certification, software quality, software testing | No Comments »
|
Rex Black is President of RBCS (
