Archive for May, 2011
Monday, May 30th, 2011 by Rex Black
I almost titled this blog post, “Why Maintenance Testing Is Good (and Why Pair Networks Should Do More of It),” but decided it was the confluence of events that was the real story here.
Last week was a week for RBCs to get hit with a bunch of costs of external failure, foisted on us by various vendors and service providers. First, we had the problem with the dying audio driver during the Q&A session of our metrics webinar (see here for details). Next, and literally just as I was finishing the blog post describing that reliability bug, the consequences of an apparently-not-well-tested upgrade by Pair Networks (hosts of our website) hit our entire site like a grenade.
At first, rbcs-us.com was completely taken out by the upgrade. With some heroic efforts by our web team, we got most of the site back up and running within a couple hours. However, the store was damaged more severely, due to database hooks apparently. The store is still down, and won’t be back up for a few days. The irony of this situation is that, while we intend to send store discount codes to people inconvenienced by the webinar audio crash, there’s no point doing that until the store is back up. Compounding costs of external failure.
If you’re thinking, “Well, ho-hum, these things happen,” consider this mind-experiment. Imagine Ford Motor Company released an upgrade to engine control software and sent it to all of their dealers to install in cars during their next scheduled maintenance. Imagine that this software caused thousands of cars to stop working completely, and to only be partly repairable with minor efforts, with complete restoration of function requiring a major service. Who do you think would be paying for the service? The customer? Or Ford?
The answer, of course, is Ford. Software and software services, however, remains among the few businesses that are allowed to transfer their costs of external failure, which is a cost-of-quality way of saying “deliver crap products and services to their customers without having to face the consequences.” This isn’t the first time I’ve made this point on this blog (see here), and I’m guessing it won’t be the last.
Anyway, a big Bronx cheer out to Pair Networks for their failure to properly test this update before putting it out there, and an even bigger Bronx cheer out to Pair Networks for their utter failure to even bother to contact us (or presumably anyone else) to express regret for the cost and inconvenience inflicted by their negligence.
P.S. For those of you unfamiliar with the phrase, dictionary.com defines “Bronx cheer” as “a loud, abrasive, spluttering noise made with the lips and tongue to express contempt.”
Posted in software quality, software test management, software testing | No Comments »
Monday, May 30th, 2011 by Rex Black
On the heels of the webinar last week, listeners have had lots of comments (all good) and some questions. Here’s an interesting set of questions from listener Stephen Ho. I’ve interspersed my answers in his e-mail, with “RB:” in front to make it easier to follow:
Rex,
Thanks for such Webinar. This webinar did not talk about how to organize and build up a good testing Metrics.
RB: There was a general discussion early on about how to go from an objective to a specific metric and specific targets for that metric. Perhaps you were missing the part about implementing the metrics with specific tools?
However, it provided some interesting points to measure the successful of a testing project, such as: BFE. Just for more realistic, how can we know that a testing metrics is good?
RB: One attribute of a good metric is that it is traceable back to some specific objective. That objective should relate to a process (e.g., finding defects is an objective for the test process), to a project (e.g., reaching 100% completion of all schedule tests is an objective for many projects), or to a product (e.g., reaching 100% coverage of requirements with passing tests is an objective for some products). Another important attribute is that the metric supports smart decision-making and, if necessary, guides corrective action. Yet another important attribute is that the metric have a realistic target.
At here, I have another topic for your interest.
“What is effective & efficiency testing?”
RB: We have to be more specific than this. What are the objectives for testing, as you mean it here? Once you have defined those objectives, you can then discuss effectively and efficiently meeting them. For example, if you define finding defects as an objective, then you can use the DDP metric (discussed in the presentation) as a metric of effectiveness. Cost of quality (which is discussed in various articles in the RBCS web site, such as this one) can serve as a metric of efficiency.
-how to narrow it down to know that our existing testing job is in effective & efficiency ways.
RB: You might want to read the chapter I wrote (Chapter 2) on this topic in the book Beautiful Software. That’s a book worth reading, anyway, because there a number of other good chapters in it.
-What are the right way to measure the performance of a QA?
RB: I assume you’re talking about an individual tester here. If we can define specific objectives for the tester, then we can use the same method to define metrics. Keep in mind the rule about objectives needing to be SMART.
-How can we know that a QA is in competence level?
RB: Check out my book, Managing the Testing Process, 3e, for a discussion about how to use skills inventories to manage the skills of your test team.
-How to increase the productivity of a QA?
RB: This question is too general, I’m afraid. Productive at what? I suggest that you define specific objectives for the test team, and then measure the current efficiency with which those objectives are achieved. At that point, you can make realistic (and measurable) goals for improvement of productivity.
You may have existing webinar or article regarding this topic. If yes, I am thirsty to study your material. Would you direct me how to access to this information. I would definitely provide my feedback to you.
RB: Follow this link for another article on metrics that you might find useful.
Thanks,
Stephen
RB: You’re welcome.
Tags: metrics, software testing effectiveness, software testing efficiency
Posted in best practices, software test management, software test metrics, software test skills, software testing | No Comments »
Monday, May 30th, 2011 by Rex Black
Long-time reader Din asked a good question about the ISTQB Advanced syllabus and the test design techniques covered in Chapter 4. He wrote:
Hi Rex,
As far as the ISTQB syllabus is concerned, topic on boundary value analysis & testing is discussed in Foundation and Advanced level. Just want to check whether deeper discussion on extension of BVA/BVT such as robustness testing and robust worst- case testing (and other related approach) of BVA/BVT are covered in our ISTQB syllabus, especially in Advanced level. I raise this as I came across several academic presentations discussing on BVA/BVT of more than one parameter/variables which I didn’t realize so much before this. Hope to get your feedback.
Thanks.
-Din-
Well, Din, I can’t speak for other training providers–and they probably wouldn’t want me to even if I could!  However, I can say that the RBCS Advanced Test Analyst and Advanced Technical Test Analyst courses–and the corresponding books–go into the topic of boundary value testing (and the related concept of equivalence partition testing) in great depth. This includes multi-value boundary value testing, along with the use of these techniques in combination with other test design techniques such as all-pairs testing, state-based testing, decision-table testing, and so forth.
Tags: ISTQB Advanced, ISTQB Foundation
Posted in ISTQB certification, software test skills, software tester certification, software testing | No Comments »
Friday, May 27th, 2011 by Rex Black
As the 300 or so people who attended yesterday’s metric webinar know, we had an audio problem about ten minutes into the question and answer session. This failure resulted in me talking to a dead microphone for a minute or two before I realized the problem and fixed it.
The symptom of the bug was two-fold. The red light on the front of the Blue Yeti microphone I was using turned off (i.e., went from illuminated red to uniluminated), indicating a complete loss of power to the microphone. In addtion, naturally enough, the audio display on the GoToWebinar console (which is tucked away toward the bottom of the console, unfortunately) showed no audio and no one talking.
To resolve the problem, I unplugged the USB cable from the PC, then plugged it back in. After resetting the audio configuration on the GoToWebinar console to use the Yeti mike (as GoToWebinar had defaulted back to the Windows default audio), we were back in business. I suspect if I had noticed immediately I could have restored audio within 1 to 2 minutes.
The main culprit is, I suspect, a reliability problem either in Windows XP’s USB drivers or in the audio drivers themselves. I favor the USB drivers as the cause, because the Yeti appeared to have been completely powered down. The failure was abrupt, and cleared itself immediately once I disconnected and re-connected the cable. If the audio drivers had failed, I would expect that clearing the problem would have taken more work. Perhaps someone with a greater understanding of Microsoft’s XP USB drivers could comment on my hypothesis.
I also would suggest to the Citrix people that their user interface is not entirely blameless in this glitch. Surely, if a webinar is active, someone should be speaking. GoToWebinar is very good about warning the presenter when they have disabled screen sharing; e.g., by changing the focus to an application other than the application being shared. Why not have a red warning that comes up saying, “The webinar is in progress, but no audio is currently detected”? Obviously, some pauses, say for a few seconds, are not a problem, but in this case the audio was down for a total of four minutes. Also, depending on the way in which the failure of the microphone manifested itself to the GoToWebinar software–e.g., did the selected and active audio device abruptly disappear from the list of available audio devices?–perhaps GoToWebinar could have put up a warning that the active audio device has become unavaiable. I’m not sure what, but I know I could have used a much more urgent heads-up than the fairly passive indicator that GoToWebinar gave me.
In general, I have been very pleased with GoToWebinar and GoToMeeting–and yes, I have used the major competitors–but this failure to warn the presenter when the audio stops is a signficant usability/ error handling bug, in my opinion. I’d be much less inclined to recommend GoToWebinar as a webinar hosting tool now, after having this experience. If anyone from Citrix is reading, I’d be happy to discuss this with you. Same for you guys from Microsoft, regarding my assertion about the reason for the mike failure.
For those of you who were victims of this glitch and spent four minutes listening to dead air, you’ll be receiving a discount code applicable to any product or service in our store. Other than that little incident, I enjoyed the webinar and everyone’s great participation. The recorded webinar (of the evening presentation, that went without a hitch) will be posted in the next few days.
Tags: software reliability, software usability
Posted in software quality, software testing | No Comments »
Friday, May 27th, 2011 by Rex Black
After the test metrics webinar held yesterday–link to recorded webinar coming soon in the Digital Library–we had an attendee ask a good question by e-mail (mailto:info@rbcs-us.com). Linda Li wrote to ask,
Hello Rex,
I just attended your free webinar about test metrics, you mentioned :
DDP=Bugs Detected/Bugs Present.
So I want to know how can I get ‘Bugs Present’? what’s included in Bugs Present? Thank you very much.
You delivered a great presentation, that is really help me much.
Thanks for the kind words about the presentation, Linda. I do hope it provides useful ideas.
As this metric, which is variously called defect detection percentage (DDP) or defect detection effectiveness (DDE), it is mathematically defined as Linda mention:
DDP = bugs found/bugs present
When we’re talking about testing at the end of the software development or maintenance process, we can say that:
bugs present = bugs found by testing + bugs subsequently found in production
So, to calculate DDP for testing, use this formula:
DDP = bugs found by testing/(bugs found by testing + bugs subsequently found in production)
In this equation, test bugs are the unique, true bugs found by the test team. This number excludes duplicates, non-problems, test and tester errors, and other spurious bug reports, but includes any bugs found but not fixed due to deferral or other management prioritization decisions. Production bugs are the unique, true bugs found by users or customers after release that were reported to technical support and for which a fix was released; in other words, bugs that represented real quality problems. Again, this number excludes duplicates, non-problems, customer, user, and configuration errors, and other spurious field problem reports, and excludes any bugs found but not fixed due to deferral or other management prioritization decisions. In this case, excluding duplicates means that production bugs do not include bugs found by users or customers that were previously detected by testers, developers, or other prerelease activities but were deferred or otherwise deprioritized, because that would be double-counting the bug. In other words, there is only one bug, no matter how many times it is found and reported.
To calculate this metric, you need to have a bug tracking system for all the bugs found in testing (and for those using Agile methods, yes, I do mean bugs found during the sprints even if fixed during the sprints). You also need a way to track bugs found in production. Most help-desk or technical-support organizations have such data, so it’s usually just a matter of figuring out how to sort and collate the information from the two (often) distinct databases. You also have to decide on a time window. That depends on how long it takes for your customers or users to find 80 percent or so of the bugs they will find over the entire post-release life cycle of the system. For consumer electronics, for example, the rate of customer encounters with new bugs (unrelated to new releases, patches, and so forth) in a release tends to fall off very close to zero after the first three to six months. Therefore, if you perform the calculation at three months, adjust upward by some historical factor—say 10 to 20 percent—you should have a fairly accurate estimate of production bugs, and furthermore one for which you can, based on your historical metrics, predict the statistical accuracy if need be.
Note that this is a measure of the test processes’ effectiveness as a bug-finding filter. Finding bugs and giving the project team an opportunity to fix them before release is typically one of the major objectives for a test team, as I mentioned in my presentation.
Tags: defect detection effectiveness, software testing metrics
Posted in software test management, software test metrics, software testing | 10 Comments »
Thursday, May 19th, 2011 by Rex Black
Following on my most recent post, another one of the Advanced Test Analyst course attendees took the exam this week. Kelly Rasmussen passed on the following advice. While a bit more targetted to the CTAL-TA exam than the other CTAL exams, there are some good general guidelines, especially regarding Foundation-level questions, the weighting of questions, and the typical e-exam experience.
Just took my exam today and I passed also! Don’t know if you all took the test yet or not, but wanted to share my experience with you. Hopefully it’ll be of some help.
Firstly, I would like to say that there was no lockers [at the Kriterion exam center] to lock up my stuff. I didn’t know this going in so I had my purse and cell phone with me. The receptionist took my stuff and put it under her desk, which I didn’t feel all that comfortable doing but didn’t want to go back to my car and put my sutff up. The exam didn’t take the whole 3 hours. I was able to answer all the questions within 2 hours. Then I used next 30 minutes or so going back to the questions that I marked for review. I wasn’t sure if I was quite ready for my results yet, so I sat there for a minute or two deciding whether to click the submit button or not. lol… Yes, I was very nervous to find out if I passed or not, but I was also tired and wanted to go home. So clicked the button, and luckily, I passed.
As far as the type of questions I got, I was surprised to see so many K1 questions from the Foundation Level. I think my first 7 questions or so were from the Foundation Level, which worked to my advantage. There were plenty of scenario questions, which were tricky in my opinion. [People with extensive testing experience will find that helpful] in deciding on the correct answer. Another thing that I thought was tricky is that you had to choose which test technique would be the best one to use depending on the scenario. So make sure you know when it is better to use one technique over another and of course how to figure out how many test cases one would need to test. You want to get as many of the K3 questions correct as possible since these are worth 3 points. I basically ditto what Jennifer said. Go over everything!!! Pay close attention to chapters 2 and 4. There were a lot of questions from these two chapters!!!
I’ll add that, implicit in Kelly’s comments, and Jennifer’s, is the lesson that extensive pre-exam preparation is required. In addition to attending a class or going through e-learning, be sure to spend lots of additional time studying for the exam.
Tags: ASTQB, ISTQB, ISTQB accredited live training, ISTQB Advanced, ISTQB exam preparation
Posted in ISTQB certification, software test skills, software tester certification | No Comments »
Friday, May 13th, 2011 by Rex Black
As followers of the RBCS Facebook and Twitter micro-blogs will know, last week we offered an ISTQB Advanced Test Analyst course. The corresponding exam has the mouthful-name “ISTQB Certified Tester Advanced Level-Test Analyst,” typically abbreviated “CTAL-TA.” CTAL-TA is also the authorized resume acronym for those who pass the exam, by the way.
Jennifer Parran of Booz Allen Hamilton took an electronic version of the exam at a Kriterion exam center just this week, and she passed on the following details. This is really good advice for anyone taking an ISTQB Advanced exam, especially the Test Analyst and Technical Test Analyst exams.
Hello –
I just wanted to let you all know I took the exam yesterday and PASSED! To give you an idea of what it was like… I took the electronic exam at the Career Technology Center in Falls Church [Virginia]. You have 3 hours once you click the “Start” button and there is a timer counting down on the screen (try not to let that distract you). (Also you may be in the room with other people taking other exams, some of which are allowed to bring books in and they might be making noise flipping back and forth in the book.)
You can mark questions for “review” if you want to review the answer or if you want to skip a question and come back to it later. You can also navigate backwards and forwards so if you answer a question and want to go right back to it, you are able to do so. Once you’re all done and click the “Submit” button (at which point I felt like I wanted to throw up) it takes a couple seconds and then displays your score and that you “passed”. It also sends you an email with this information which I saw later.
Oh yeah…don’t touch the keyboard while taking the exam! I accidentally hit the spacebar and it backed me out of the exam and I had to get the proctor to log back in and bring the exam back up. THANKFULLY it saved all my answers as I only had an hour left but to play it safe – only use the mouse!
The questions were very similar in style to the sample questions [provided in the RBCS live and e-learning courses and exam prep guides] so definitely do all of those. My questions covered every chapter in the advanced syllabus so you definitely want to read the entire syllabus [as discussed in the class]. Some of the questions seemed more foundational but it was hard to distinguish between K1 Foundation and K1 Advanced. I also had questions pertaining to several of the referenced standards so like Rex said in class – you don’t want to be “sad” when you’re reading one those questions and you can’t remember what it is.
Make sure you are very comfortable with the black-box techniques like we practiced in class because there are quite a few questions that are very long and involve a lot of reading [that involved] these techniques so you don’t want to have to spend a lot of time reading the question and remembering how to do the technique. These are of course the K3 questions.
As far as preparation goes, I read through all of the foundation and advanced syllabus’ once and then skimmed over my “highlights” several more times (once right before I took the exam). I did all the sample questions and mock exams. I skimmed over the referenced standards a couple times. I studied a little Monday night, 3 hours Tuesday night, all day Wednesday, and then read over my highlights right before the exam.
Good luck to all of you and Thanks Rex for all the great explanations and preparation!
Jennifer
By the way, for those of you considering taking an ISTQB exam and still mulling over your exam preparation options, RBCS will have an exciting announcement in the next couple weeks on this topic. Watch our Facebook page, Twitter posts, and our newsletter for more information. If you’ve already decided to take an RBCS course, you can find the course schedule on the home page.
Tags: ISTQB accredited e-learning training, ISTQB accredited live training, ISTQB Advanced, ISTQB CTAL, ISTQB exam preparation
Posted in ISTQB certification, software test skills, software tester certification, software testing | No Comments »
Thursday, May 5th, 2011 by Rex Black
I’m running an ISTQB Advanced Test Analyst course this week in DC. This course includes a hands-on set of exercises based on a realistic requirements specification. This allows people to transfer the techniques immediately to real-world projects when they return from training, because they have already done so in the class.
In this particular class, we use a requirements specification based on a real project. This requirements specification was originally written by a senior programmer–not a business analyst or requirements engineer–on the project. I anonymized the requirements and reorganized them, for use in the training course, but other than that did not change their quality or coverage.
We did a review exercise today. People did 30 minutes of preparation, followed by a 30 minute walkthrough. We found over 20 defects, some of which had multiple manifestations in the requirements specification. We concluded that, in a real world situation, the document would need revision to repair these defects, which would cause significant changes, and thus would necessitate a re-review. Basically, this means that these requirements were not an adequate basis for further development or for creation of test cases.
Interestingly enough, one of the attendees, an experienced, professional tester, remarked, “These are the best requirements I’ve ever seen.” Only one participant said she typically sees better requirement specifications.
So, is this the best we can do, in this industry? We must make do with requirements written by people with no real qualifications to do so? We must proceed with development and testing using fundamentally flawed requirements as a basis?
I’d be interested in your thoughts. How do requirements affect you and your testing? Do you receive adequate requirements? If so, why do you think you do? If not, what do you think you don’t?
Posted in software quality, software testing | 5 Comments »
|
Rex Black is President of RBCS (
