Archive for the ‘software quality’ Category
Thursday, October 13th, 2011 by Rex Black
Last week, Steve Jobs died, and I published an encomium to him in this blog. Today, I was saddened to read of the death of Dennis Ritchie, one of the greatest software innovators that most people have never heard of, but who had arguably as large an impact on our world as Steve Jobs did. I learned much of what I know about how software works by reading his book on C programming and by applying what I learned from that book to programming Unix systems in the 1980s and 1990s.
Shortly after I read about his death, I received an excellent post from Lauren Weinstein, relating his own thoughts and experiences about Dennis Ritchie’s inestimable contributions to the software world. I asked Lauren if I could quote his post, and he graciously agreed. Here follow Lauren’s fine remarks:
A Few Brief Thoughts on the Death of Computer Science Titan Dennis Ritchie http://lauren.vortex.com/archive/000903.html
Dennis Ritchie has died after a long illness ( http://j.mp/pQfU8J [CNET]).
He was only the relatively young age of 70. Unlike Steve Jobs, Dennis was not a household name. He should have been, but he was a quiet and private person, and would have hated the publicity.
I’ve known Dennis since the early days of the UNIX operating system at Bell Labs in the 70s, where he created the ubiquitous “C” programming language, and co-created UNIX with Ken Thompson.
It is impossible to overstate the importance of this work to today’s world. UNIX was almost entirely written in “C” — and UNIX’s direct descendants like Linux power Google, the Web at large (the vast majority of Web servers run Linux), and likely multiple devices in your home and office right now. The current Mac OS. Android. TiVo. The impact of C and Linux are everywhere. Your ability to read these words rests directly to a major degree on Dennis’ work.
But beyond the nitty-gritty of software design, the creation of UNIX was also the inception of what would ultimately become the open source community, and the vast collaborations of the ARPANET/Internet that have led to the global phenomenon we see today.
Such goals were explicit in the design of UNIX. In this 1980s Bell Labs video from my collection, featuring Ken and Dennis explaining the origins of UNIX, Dennis explains how they wanted the system to specifically foster community and fellowship.
I said this would be brief. I’ll close with one personal story. Back in the early UNIX days, on one of my visits to Bell Labs’ main facilities in Murray Hill, New Jersey, I was sitting at a terminal in the 1127 Labs where UNIX was developed, logged in over the nascent Net
back to UCLA. As usual, I was trying to get a bit of coding done even on this trip.
My rapid typing suddenly stopped as I puzzled over a particularly complex C language declaration in the program, that definitely was incorrect as it stood.
Sitting at the terminal next to me was Dennis. So I asked his advice on the declaration — after all, who better to know than the creator of the language?
He thought about it for no more than a few seconds, then immediately (and graciously) provided an elegant solution that, frankly, would not have occurred to me. I got on with my programming.
In later years, I realized that this exchange was probably the closest I’d ever come in my life to asking a question directly to, and receiving a detailed answer from, a true god.
Rest in peace, Dennis.
–Lauren–
Founder:
Member: ACM Committee on Computers and Public Policy
Tel: +1 (818) 225-2800 / Skype: vortex.com
Lauren, thanks for permission to re-post this. Dennis, thank you very much for what you taught me through your book and your work, and for what you did to advance software engineering. May we always be so fortunate as to work in an industry with such generous hearts and brilliant minds as Dennis had.
Tags: Dennis Ritchie
Posted in software quality | No Comments »
Friday, October 7th, 2011 by Rex Black
Long time reader, Patricia Osorio Aristizabal, asked me to clarify the following question:
During a walkthrough meeting for a high-level design, a tester who is participating as a reviewer asks the system architect who created the document why only physical network connections are supported in the design, rather than also supporting wireless connections. The designer replies that they did not have time to address the security concerns raised by wireless access to the system. The tester accepts that answer and the review proceeds.
Which of the following is an objective of a review that was achieved in that exchange?
A. Increased reviewer understanding
B.Early detection of a defect
C. Correction of a defect
D. Analysis of system design
The correct answer is A. The tester now has a better understanding of why the system is built the way it is. The design of the system is not defective, as there is a deliberate reason for not having WiFi support, so C is not the right answer.
Tags: software reviews
Posted in software quality, software testing | No Comments »
Friday, September 30th, 2011 by Rex Black
This week, I’m pleased to feature a bonus bug of the week, courtesy of long-time reader John Singleton. He wrote to tell us:
Rex,
I was posting some comments in response to your blog post, entitled “Bad Agile Implementation Causes Software Test Challenges”. For whatever reason, the Submit button was entirely non-responsive in Firefox. Thankfully, switching over to IE resolved it.
Not griping, just thought you’d want to know. Good, thoughtful response you gave.
Cheers,
John
I responded:
Hi John–
That’s weird. It’s usually the other way around with WordPress (the software we use for our blog): Firefox works when IE doesn’t. I guess we have a bonus bug of the week. 
Glad you liked the post. I’m looking forward to your comments.
Regards,
Rex
So, while this isn’t a very serious bug, it is part of a very serious pattern: Failure of major players in the web world to test browser compatibility. See my previous posts http://bit.ly/mdBIRY and http://bit.ly/mPkD9z. Is this a lack of understanding of basic test design techniques such as equivalence partitioning and (for particularly risky situations) pairwise testing? Or just a lack of caring?
I have to say that this pattern is not universal. Working with a major e-commerce client this week, I was pleased to see that they include testing of most of the major browsers, covering the equivalence partitions.
What do you think about this pattern? Are you in the web world, and, if so, do you test browser compatibility? If not, why?
Tags: browser incompatibility, equivalence partitioning, pairwise testing, web testing
Posted in best practices, compatibility testing, software quality, software testing, test coverage | No Comments »
Thursday, September 29th, 2011 by Rex Black
As those of you who receive our newsletter know, we had a little glitch this week. The newsletter greeted readers with “Dear (Contact First Name)”, when of course it should have given the reader’s first name rather than the parenthesized stuff.
Immediately–given the 14,000+ good testers who read our newsletter–we received many reports of the defect. A massive self-head-smack and “Doh!” action (a la Homer Simpson) ensued at the RBCS mothership. We promptly issued a 20% discount code to our readers as a way of apologizing for–well, someone forgetting their names.
So, who forgot their names? We use ConstantContact for our e-mail, so one logical explanation is a bug in ConstantContact’s mail merge code. We suspect that is the cause, because a test e-mail sent to Laurel Becker, RBCS VP, included the “Dear Laurel” string, but in the wrong place.
That said, there’s the “never attribute to malice that which can be explained by human error” angle to this. Did we accidentally mangle the newsletter template in a way that overrode the mail merge function? It’s certainly a possibility, and we can’t tell from the source newsletter whether that is the case. This is a bug in ConstantContact’s editing tool, because it’s possible to damage these mail merge keywords in such a way that you can’t see the mistake in either the source letter or the test copies.
So, the conclusion at this end is: It’s a mystery. We apologize to our readers again for appearing to have forgotten their names. We’ll keep an eye on this for future newsletters.
In the meantime, this is…uh, yeah, Rex, that’s right…Rex here reporting a possible bug…or just an inattentive copy-paste action somewhere in the process.
Tags: software defect, software quality
Posted in software quality, software testing | No Comments »
Thursday, September 8th, 2011 by Rex Black
This post is part of my all-too-regular series of posts on bugs that have vexed, frustrated, or inconvenienced me, my clients, or the world in general. It is a series that shows no sign of abating, sadly.
This week we had two bugs, one fairly nasty, one less so but still irritating.
To start with the nasty, as some readers will know, we use Articulate to author and host our e-learning. I’ve used other authoring and hosting tools in the past, and found them all relatively immature and buggy. Articulate is better than the others I’ve used, but by no means good enough, as this latest incident illustrates.
Articulate did a “scheduled update” on Saturday. They were nice enough to let us know about it a week beforehand, so we could warn our e-learning customers about downtime. In the past, we’ve had “stealth updates” where they didn’t tell us and then all of a sudden something stopped working; they would then deny that they had done an update and insist that we must have changed something.
However, Articulate apparently wasn’t nice enough to thoroughly regression test the update before they put it into production. As a result, all of our podcast attachments (MP3 audios of the e-learning courses) are corrupted. Re-attaching them doesn’t work, so we had to file a bug report with Articulate.
Did I say they didn’t regression test the update? Wait, I’m wrong, they are regression testing it, using their paying customers to do so. Why bother with equivalence partitioning the support attachments and testing that they all work when you can just promote untested code into production and make your paying customers do free work for you as testers? What’s most frustrating about this incident is that this is certainly not the first time it’s happened, and Articulate is by no means alone in employing this infamous tactic.
Now, on to the irritating. I did a webinar today. Over 400 people showed up, which was great. What wasn’t so great was that a few of them had a weird bug affect them. They reported not seeing the slides, but only the white welcome page with a greeting from me and my picture. As attendee James Heeren, who was affected by the bug, put it:
A funny thing happened when I closed the Webinar Window, a second window appeared with the picture of the graph of defects reported and defects closed. [This was a slide in the webinar presentation.] The real thing was hidden behind the white page somehow.
Was not a big problem for me as I followed on the deck from the website.
If it were just one person, I chalk it up to someone moving the focus to another tab in their browser by accident. Since it was quite a few people, it sounds like what was happening was that the GoToWebinar software was not properly getting the focus on the presentation window in some cases.
I have some sympathy for Citrix in this case. This sounds like a configuration-specific bug, which is really hard to find. That said, even though we will report it, I’m not expecting a lot of help from Citrix, based on past results of reporting bugs to them.
Until there is some implied warranty rule for software, as there is for every other product and services sold in the US, this kind of nonsense will continue.
Tags: Articulate, Citrix, e-learning, GoToWebinar, software bugs, software quality, webinar
Posted in software quality, software testing | No Comments »
Wednesday, August 24th, 2011 by Rex Black
We have lots of terms in software testing, but we’re not always clear what they mean. A common confusion is about the terms bug, defect, fault, failure, anomaly, incident, false positive, error, and mistake. Reader Rodrigo Cursino asks:
I read [Foundations of Software Testing]… You defined bug as a synonymous with fault or defect. If we exercise the part of software that contains that bug we’ll have a failure.
Now I’m reading your article “The Bug Reporting Processes” and I’m a little confused. Should the correct [title] be “The Failure Reporting Processes”. It’s because we report failures not bugs. The developers, based on the steps to reproduce the failure will be able to debug it and find the bug.
Rodrigo, you are right. This is a case of common usage prevailing of terminological rigor, I suppose. Many people talk about “bug reporting,” so I used that term in the article and in my book Critical Testing Processes.
If we want to be rigorous in our terminology, here’s the way to think about the sequence of events:
- The programmer makes a mistake (also called an error). This can be a misunderstanding of the internal state of the software, an oversight in terms of memory management, confusion about the proper way to calculate a value, etc.
- The programmer introduces a bug (also called a defect) into the code. This is the programmatical manifestation of the mistake.
- The tester executes the part of the software that contains the bug.
- If the test was properly designed to reveal the bug, the test can cause the buggy software to execute in such a way that the behavior of the software is not what the tester–who is closely observing the behavior–would expect. This difference between expected behavior and actual behavior is called an anomaly.
- The tester then investigates the anomaly to determine the exact failure. The failure may go beyond the obvious, immediately observable misbehaviors associated with the anomaly. For example, data might have been corrupted, another process improperly terminated, etc.
- The results of that investigation are a report, which is commonly referred to in the software business as a bug report, an incident report, a defect report, an issue, a problem report, or various other names.
- Whatever we call it, that report gets prioritized and (in some cases) ultimately routed to a programmer. The programmer then debugs the program in order to repair the underlying bug.
- Ideally, the bug fix (typically as part of a larger test release) comes back to the tester who reported the problem in the first place for a confirmation test. If the confirmation test passes, the report can be closed as fixed. If the confirmation test fails, the report should be re-opened.
Now, a few additional points are worth making:
- In some cases, when a tester runs a test, she observes an anomaly, but not due to a failure. This happens when the anomaly results from a bad test, bad test data, an improperly configured test environment, or simply a misunderstanding on the tester’s part. This situation is referred to as a false positive. Because some reports will inevitably be false positives–testers being human, they will also make mistakes–some people like to refer to them as incident reports. An incident is some situation that requires further investigation, and in this case the programmer will investigate whether the incident was really caused by a failure.
- Bugs (or defects) can be introduced into work products other than software. For example, a business analyst can put a bug into a requirements specification. Bugs in requirements specifications and design specifications (and code, for that matter) are ideally detected by reviews. When a bug is detected by a review (or by static analysis), notice that the bug is what is actually detected; the software is not executing, so no failure occurs.
- Some people use the word fault instead of bug or defect. I don’t like that term, and avoid it. When I talk about a fault, perhaps it sounds like I’m talking about something that is someone’s fault; i.e., implications of blame can arise. Bugs happen for various reasons, and individual carelessness is not at the top of the list. We should see bugs (and bug reports) as a way to understand the quality capability of the software process, not as a way to apportion blame.
So, back to Rodrigo’s bigger point: Does it really matter what we call the report? If you want to be 100% correct in your terminology, then incident report is probably the best name. However, I think that failure report is fine, too. I also think that, because these terms are so widely used, defect report and bug report are also acceptable. However, when using the terms defect report or bug report, it’s important that people keep in mind the sequence of events laid out above, and that the report actually describes the symptom of the bug, not the bug itself.
Tags: bug, defect, error, failure, fault, mistake, software testing
Posted in software quality, software test management, software testing | 2 Comments »
Tuesday, August 2nd, 2011 by Rex Black
Reader Patricia Osorio had another good question about the answer to the following question from our Advanced Test Manager e-learning course. Here’s the question:
A given organization is using reviews for development work products like code, requirements and design specifications; test work products like test plans, quality risk analyses, and test design specifications; and, documentation and help screens. The review processes have been in place for two years and are delivering excellent financial, quality, and schedule benefits.
You are attending a management team meeting. A senior executive raises the need to update the objectives by which the individual contributors are measured on their yearly performance evaluations. He suggests using defect counts from review meetings. He circulates a draft plan. Under the plan, people will be rewarded based on the number of defects they find in reviews. Further, people will be penalized if items they have produced incur too many defects during reviews.
Which of the following is a psychological factor affecting review success and failure that is likely to cause such an initiative to undermine the current success of the reviews?
- Scrutinize the document and not the author.
- Focus all participants on delivering high-quality items.
- Try to find as many defects as possible.
- Assemble the right team of reviewers.
The correct answer is the second option. Instead of focusing on delivering high-quality items, the reviewers will focus on trying to claim as many defects found in reviews as their own, and will raise as defects trivialities, in order to try to maximize their rewards, while the authors will try to refute every defect report, even the most serious, in order to try to minimize their punishments. We have seen this play out in real life with clients, so you must be careful.
Tags: software reviews
Posted in ISTQB certification, software quality, software testing | No Comments »
Monday, July 4th, 2011 by Rex Black
As long-time listeners–or even brand new listeners, for that matter–of the RBCS webinars know, we use Citrix’s GoToWebinar service for our free monthly webinars. Now, I’ve been fairly satisfied with GoToWebinar. I’ve used one or two of the competing services, and been less happy with those. Of course, webinar listeners (and readers of this blog) might remember I chided Citrix back in May for the ungraceful way the system handles audio drop-outs by the presenter.
So, during the June webinar, webinar attendee Keith Stobie reported an inability to see the presentation using Internet Explorer 9. He said that Chrome (not sure which version) worked just fine. I reported the problem to Citrix on Wednesday of last week. Five days later, I receive the following reply, quoted in its entirety (minus the links provided at the end):
Thank you for contacting Citrix Online Global Customer Support,
Dear Rex Black,
IE 9 has not been tested with any of our products as of yet. we will try to help fix any issues the best we can, but cannot guarantee anything. Hopefully we should get this done as soon as possible.
If you have any additional questions or need further clarification regarding this matter, please feel free to reply directly to this email. For any other product inquiries or technical assistance, please visit us at our Support Centers listed at the bottom of this email. Our Support Centers include Self Help files and our Global Customer Support Contact Information.
Thank you,
Richard Carrel | Global Customer Support
So, I appreciate the reply, though I have to say that five days isn’t quick turnaround for a customer complaint about a browser-based service that’s incompatible with a major vendor’s browser.
More surprising to me is the admission that Citrix hadn’t tested IE9. I don’t keep up with the browser wars, so I’m not sure what share of the browsing action IE9 has, but I’m pretty sure that Microsoft’s IE family of browsers remains at least one of the 800 pound gorillas in the room.
Putting myself in the position of the Director of Quality or VP of Testing or whatever the head-testing-honcho’s title is at Citrix, I understand that there are constraints on compatibility testing. I wouldn’t bother to test four-year-old versions of Opera, for example. But come on, not testing IE9? If I were in charge of testing for any SaaS provider, compatibility would be one of my top quality risks, and testing browser/OS/malware configuration combinations would receive a fair amount of time, money, and attention. Of course, functionality, reliability, performance, and security would also be high on the list of risk categories, too.
Here’s some free consulting advice to my fellow test professionals who work at Citrix: Spend a little time getting ramped up on how to do quality risk analysis and risk based testing. You can find lots of free resources on our web site, especially in the articles and the Digital Library. You’ll notice that compatibility is one of the quality risk categories included in our free quality risk checklist. If you need more help, let me know, as we can provide a one-week risk based testing bootstrapping service that will get you headed in the right direction.
Morale of the story: If you are in charge of testing at any SaaS vendor, and you’re not testing for compatibility, it’s only a matter of time before someone writes a blog post like this one about your product and the degree to which you aren’t testing it.
Tags: software compatibility, software testing
Posted in best practices, risk based testing, software quality, software testing, test coverage | No Comments »
Monday, May 30th, 2011 by Rex Black
I almost titled this blog post, “Why Maintenance Testing Is Good (and Why Pair Networks Should Do More of It),” but decided it was the confluence of events that was the real story here.
Last week was a week for RBCs to get hit with a bunch of costs of external failure, foisted on us by various vendors and service providers. First, we had the problem with the dying audio driver during the Q&A session of our metrics webinar (see here for details). Next, and literally just as I was finishing the blog post describing that reliability bug, the consequences of an apparently-not-well-tested upgrade by Pair Networks (hosts of our website) hit our entire site like a grenade.
At first, rbcs-us.com was completely taken out by the upgrade. With some heroic efforts by our web team, we got most of the site back up and running within a couple hours. However, the store was damaged more severely, due to database hooks apparently. The store is still down, and won’t be back up for a few days. The irony of this situation is that, while we intend to send store discount codes to people inconvenienced by the webinar audio crash, there’s no point doing that until the store is back up. Compounding costs of external failure.
If you’re thinking, “Well, ho-hum, these things happen,” consider this mind-experiment. Imagine Ford Motor Company released an upgrade to engine control software and sent it to all of their dealers to install in cars during their next scheduled maintenance. Imagine that this software caused thousands of cars to stop working completely, and to only be partly repairable with minor efforts, with complete restoration of function requiring a major service. Who do you think would be paying for the service? The customer? Or Ford?
The answer, of course, is Ford. Software and software services, however, remains among the few businesses that are allowed to transfer their costs of external failure, which is a cost-of-quality way of saying “deliver crap products and services to their customers without having to face the consequences.” This isn’t the first time I’ve made this point on this blog (see here), and I’m guessing it won’t be the last.
Anyway, a big Bronx cheer out to Pair Networks for their failure to properly test this update before putting it out there, and an even bigger Bronx cheer out to Pair Networks for their utter failure to even bother to contact us (or presumably anyone else) to express regret for the cost and inconvenience inflicted by their negligence.
P.S. For those of you unfamiliar with the phrase, dictionary.com defines “Bronx cheer” as “a loud, abrasive, spluttering noise made with the lips and tongue to express contempt.”
Posted in software quality, software test management, software testing | No Comments »
Friday, May 27th, 2011 by Rex Black
As the 300 or so people who attended yesterday’s metric webinar know, we had an audio problem about ten minutes into the question and answer session. This failure resulted in me talking to a dead microphone for a minute or two before I realized the problem and fixed it.
The symptom of the bug was two-fold. The red light on the front of the Blue Yeti microphone I was using turned off (i.e., went from illuminated red to uniluminated), indicating a complete loss of power to the microphone. In addtion, naturally enough, the audio display on the GoToWebinar console (which is tucked away toward the bottom of the console, unfortunately) showed no audio and no one talking.
To resolve the problem, I unplugged the USB cable from the PC, then plugged it back in. After resetting the audio configuration on the GoToWebinar console to use the Yeti mike (as GoToWebinar had defaulted back to the Windows default audio), we were back in business. I suspect if I had noticed immediately I could have restored audio within 1 to 2 minutes.
The main culprit is, I suspect, a reliability problem either in Windows XP’s USB drivers or in the audio drivers themselves. I favor the USB drivers as the cause, because the Yeti appeared to have been completely powered down. The failure was abrupt, and cleared itself immediately once I disconnected and re-connected the cable. If the audio drivers had failed, I would expect that clearing the problem would have taken more work. Perhaps someone with a greater understanding of Microsoft’s XP USB drivers could comment on my hypothesis.
I also would suggest to the Citrix people that their user interface is not entirely blameless in this glitch. Surely, if a webinar is active, someone should be speaking. GoToWebinar is very good about warning the presenter when they have disabled screen sharing; e.g., by changing the focus to an application other than the application being shared. Why not have a red warning that comes up saying, “The webinar is in progress, but no audio is currently detected”? Obviously, some pauses, say for a few seconds, are not a problem, but in this case the audio was down for a total of four minutes. Also, depending on the way in which the failure of the microphone manifested itself to the GoToWebinar software–e.g., did the selected and active audio device abruptly disappear from the list of available audio devices?–perhaps GoToWebinar could have put up a warning that the active audio device has become unavaiable. I’m not sure what, but I know I could have used a much more urgent heads-up than the fairly passive indicator that GoToWebinar gave me.
In general, I have been very pleased with GoToWebinar and GoToMeeting–and yes, I have used the major competitors–but this failure to warn the presenter when the audio stops is a signficant usability/ error handling bug, in my opinion. I’d be much less inclined to recommend GoToWebinar as a webinar hosting tool now, after having this experience. If anyone from Citrix is reading, I’d be happy to discuss this with you. Same for you guys from Microsoft, regarding my assertion about the reason for the mike failure.
For those of you who were victims of this glitch and spent four minutes listening to dead air, you’ll be receiving a discount code applicable to any product or service in our store. Other than that little incident, I enjoyed the webinar and everyone’s great participation. The recorded webinar (of the evening presentation, that went without a hitch) will be posted in the next few days.
Tags: software reliability, software usability
Posted in software quality, software testing | No Comments »
|
Rex Black is President of RBCS (
