Posts Tagged ‘software testing’
Monday, July 4th, 2011 by Rex Black
As long-time listeners–or even brand new listeners, for that matter–of the RBCS webinars know, we use Citrix’s GoToWebinar service for our free monthly webinars. Now, I’ve been fairly satisfied with GoToWebinar. I’ve used one or two of the competing services, and been less happy with those. Of course, webinar listeners (and readers of this blog) might remember I chided Citrix back in May for the ungraceful way the system handles audio drop-outs by the presenter.
So, during the June webinar, webinar attendee Keith Stobie reported an inability to see the presentation using Internet Explorer 9. He said that Chrome (not sure which version) worked just fine. I reported the problem to Citrix on Wednesday of last week. Five days later, I receive the following reply, quoted in its entirety (minus the links provided at the end):
Thank you for contacting Citrix Online Global Customer Support,
Dear Rex Black,
IE 9 has not been tested with any of our products as of yet. we will try to help fix any issues the best we can, but cannot guarantee anything. Hopefully we should get this done as soon as possible.
If you have any additional questions or need further clarification regarding this matter, please feel free to reply directly to this email. For any other product inquiries or technical assistance, please visit us at our Support Centers listed at the bottom of this email. Our Support Centers include Self Help files and our Global Customer Support Contact Information.
Thank you,
Richard Carrel | Global Customer Support
So, I appreciate the reply, though I have to say that five days isn’t quick turnaround for a customer complaint about a browser-based service that’s incompatible with a major vendor’s browser.
More surprising to me is the admission that Citrix hadn’t tested IE9. I don’t keep up with the browser wars, so I’m not sure what share of the browsing action IE9 has, but I’m pretty sure that Microsoft’s IE family of browsers remains at least one of the 800 pound gorillas in the room.
Putting myself in the position of the Director of Quality or VP of Testing or whatever the head-testing-honcho’s title is at Citrix, I understand that there are constraints on compatibility testing. I wouldn’t bother to test four-year-old versions of Opera, for example. But come on, not testing IE9? If I were in charge of testing for any SaaS provider, compatibility would be one of my top quality risks, and testing browser/OS/malware configuration combinations would receive a fair amount of time, money, and attention. Of course, functionality, reliability, performance, and security would also be high on the list of risk categories, too.
Here’s some free consulting advice to my fellow test professionals who work at Citrix: Spend a little time getting ramped up on how to do quality risk analysis and risk based testing. You can find lots of free resources on our web site, especially in the articles and the Digital Library. You’ll notice that compatibility is one of the quality risk categories included in our free quality risk checklist. If you need more help, let me know, as we can provide a one-week risk based testing bootstrapping service that will get you headed in the right direction.
Morale of the story: If you are in charge of testing at any SaaS vendor, and you’re not testing for compatibility, it’s only a matter of time before someone writes a blog post like this one about your product and the degree to which you aren’t testing it.
Tags: software compatibility, software testing Posted in best practices, risk based testing, software quality, software testing, test coverage | No Comments »
Sunday, June 26th, 2011 by Rex Black
Long-time reader Patricia Osorio Aristizabal sent the following question via e-mail (info@rbcs-us.com):
Hi Mr Black
I have a dilemma and I would like to know your thoughts about it.
It is clear for me the difference between error, defect, and failure (IEEE 610). Besides, the convenience of keeping in mind and use these words orally in order to help the project team – may be – to find out where or when an issue (incident) occurs. However, it is not easy to change all the team to use these words without cause resistance and incredulity to the importance of that difference. In your experience, how I could get to change first, my team and then all the organization to standardized the use of that concept? Could you please give some tips to do this change?
For those readers not familiar with the distinction to which Patricia alludes, here are the ISTQB Glossary definitions for each of those terms:
error (or mistake): A human action that produces an incorrect result.
defect (or bug): A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
failure: Deviation of the component or system from its expected delivery, service or result.
incident: Any event occurring that requires investigation. [Note: During testing, any failure that occurs would be an incident, though some incidents ultimately turn out to be false positives caused by things like bad test cases, bad test data, bad test environments, etc.]
So, based on these definitions, the programmer makes an error (or mistake) in their thinking while writing a program, leading to a defect in the code. The code is executed during testing and the defect produces a failure. The tester observes the failure, investigates the incident, and presumably files a bug report.
Certainly, having clear thinking in one’s mind–reflected by understanding the distinctions drawn–can help testers understand what’s going on. However, it can be difficult to convince people to adopt the definitions. As a consultant, I’m familiar with the challenge of trying to motivate people to make changes of any sort.
The general rule for change, in my opinion, derives from a basic rule of sales: People move away from pain much more quickly and reliably than they move toward a desirable situation. In other words, if you want to motivate change, make people aware of the organizational pain–waste, delay, frustration, etc.–associated with the behavior you want to change, and then suggest how to make that pain go away based on your proposed change.
So, Patricia, if you can find a way in which a lack of clarity on these definitions is causing waste, delay, frustration, or other sorts of organizational problems, then you might succeed at motivating changes here.
Tags: software testing, software testing best practices Posted in ISTQB certification, best practices, software testing | No Comments »
Monday, April 4th, 2011 by Rex Black
I received an interesting question from a colleague in Malaysia, Dhiauddin Suffian. He wrote:
Hi Rex, I have one simple question with regard to Fundamental Test Process. As we aware, the process involves Planning & Control, Analysis & Design, Implementation & Execution, Evaluating Exit Criteria & Reporting and Test Closure. My concern is on the Test Planning and Control, since it goes along the way of the whole process. I have no issue on the “Planning” portion. My question is directed to the “Control” part. What are “Control” activities involved in subsequent phases, i.e. “Control” activities that happen in Analysis & Design, Implementation & Execution, Evaluating Exit Criteria & Reporting and Test Closure, respectively. Thanks. Regards, -Din (CTFL, CTAL-TM)-
Test control can be thought of as the test management tasks required throughout the test process in order to keep the testing aligned with the software development process, the needs of the project, and the needs of the organization. These tasks occur as needed, based on the judgement of the test manager or other members of the project team, and can also occur on a planned basis.
For example, we might plan to regularly check our risk analysis to see if we have discovered new risks, or uncovered information that tells us we should revise the risk levels for the existing risks. As another example, if we find that a key piece of testing hardware will be available earlier than we expected, we might re-work our test execution schedule to accelerate the tests that use that hardware.
Yet another example could be if we discovered, during test execution, that a key test staff member will be leaving the team. In this case, if we did a thorough job during test planning, we might have identified a contingency plan for loss of a key staff member. This is a classic project risk, after all, and a good manager should consider all such risks. If we do have a contingency plan, triggering that contingency plan would be an act of test control.
Here’s an analogy: Think of the test plan as a roadmap, with the starting location and the final destination clearly indicated. This roadmap will help you drive to your chosen destination. However, throughout your drive, you should plan to stop at traffic lights, mind your lane and speed, adapt to unexpected events (such as pedestrians stepping into a crosswalk), and even adaptively overcome errors in the roadmap (such as discovering a planned route is closed due to roadwork). While a good test plan makes test control easier–just as a good roadmap makes driving easier–the smart manager remains ever alert to the possible need for test control.
Tags: software testing, test control, test management Posted in risk based testing, software test management, software testing | 2 Comments »
Tuesday, March 29th, 2011 by Rex Black
For those readers outside the US or too young to remember this ad slogan, there is a glue-trap-based insect control device called the Roach Motel. If a cockroach (or other bug) walked onto the floor of the rectangular box (open at the ends, of course), it would become trapped on the glue. The slogan was, naturally enough, “Bugs check in but they don’t check out.”
While it might seem like I’m setting up a story about a software development team that never fixes bugs, I actually have a different story to tell.
A number of years ago, I broke one of my rules–be a conservative adopter of new technologies–and was an early adopter of LinkedIn. However, within a short period of time, I read an article (I believe in Computer World or Information Week) that said LinkedIn was planning on “monetizing” its business model by selling information to recruiters and other interested parties. So, I stopped using it and started declining invites.
Being somewhat lazy–and not having many contacts at risk–I didn’t get around to trying to delete the account for some time. About a year or so ago, I finally got tired of having to decline two or three LinkedIn invites every week, so I went ahead and deleted the account.
It wasn’t easy to figure out how to do it. Usability of that feature didn’t seem to be high on the list. Finally, I did manage to get to the right page and did get a confirmation that the account was deleted. I turns out, I should have saved a screen shot of that confirmation message.
Because, to my surprise, the LinkedIn invites keep on coming. I guess they didn’t test whether this delete feature works, or at least didn’t test it very well. Of course, to some extent, this confirms my decision not to get caught up in the LinkedIn mania. I’m glad I didn’t get a lot of contacts in there before I read that article.
So, if you are a colleague of mine, and you send me a LinkedIn invite, you’re likely to receive an e-mail much like this one:
I hope you are dong well? It’s been a while since we’ve touched base. How are things going with you?
I’ve stopped using LinkedIn because I’m not comfortable with the way they manage personal data and contact data.
In the meantime, I will probably spend some time in the next few weeks trying to figure out how to check out of the Roach Motel of social networking. I’ll post an update if I can find any way to unstick my little paws from the glue-trap floor.
Tags: social networking, software testing Posted in software testing | 1 Comment »
Monday, March 28th, 2011 by Rex Black
Here’s a question about equivalence partitioning.
My name is Nhat Khai Le and I’m taking ISTQB Advanced Technical Test Analyst online course (provided by RBCS).
I faced this sample question:
You are testing an accounting software package. You have a field which asks for the month of a transaction.
You know that months are treated differently in this software depending on the month of the quarter (first month of the quarter is treated differently than the second which is treated differently than the third.)
How many test cases, minimum, would you need to test this field to get equivalence coverage?
A. 4
B. 3
C. 5
D. 7
I chose answer B, i.e 3, but the answer is C, i.e 5.
The reason the answer is C is because there are three months in each quarter, each of which is different, plus the two months immediately preceding and following the quarter. It’s important to remember that equivalence partitioning includes partitions outside the “normal” range when using it for testing.
To help visualize the answer, see the illustration below:
 Equivalence Partitioning for the Months in the Quarter
Tags: software testing Posted in software testing, test coverage | 3 Comments »
Sunday, March 27th, 2011 by Rex Black
Reader Gianni Pucciani has a good question about a question in the Advanced Software Testing: Volume 2 book:
I have another doubt for a question in Advanced Software Testing Vol. 2. It is about the first question in Chapter 7, Incident Management. The book says that the correct answer is C “Insufficient Isolation”. What does it mean? I had chosen B “Inadequate classification information”, because all the rest was not making sense to me. For B, I could justify it saying that more information could be added to the incident report, e.g the error message displayed by the application.
Here is the question from the book:
Assume you are a test manager working on a project to create a programmable thermostat for home use to control central heating, ventilation, and air conditioning (HVAC) systems. In addition to the normal HVAC control functions, the thermostat also has the ability to download data to a browser-based application that runs on PCs for further analysis.
During quality risk analysis, you identify compatibility problems between the browser-based application and the different PC configurations that can host that application as a quality risk item with a high level of likelihood.
Your test team is currently executing compatibility tests. Consider the following excerpt from the failure description of a compatibility bug report:
1. Connect the thermostat to a Windows Vista PC.
2. Start the thermostat analysis application on the PC. Application starts normally and recognizes connected thermostat.
3. Attempt to download the data from the thermostat.
4. Data does not download.
5. Attempt to download the data three times. Data will not download.
Based on this information alone, which of the following is a problem that exists with this bug report?
A. Lack of structured testing
B. Inadequate classification information
C. Insufficient isolation
D. Poorly documented steps to reproduce
The reason that the answer is “C” is because we don’t see any evidence of the tester trying some different scenarios to see if the data downloads properly. The testing is clearly well-structured and carefully thought out, and the steps to reproduce are well-described. The classifications are not given, so we have no way of saying, based on this information alone, whether those classifications are correct.
Tags: bug isolation, software testing Posted in software test metrics, software testing | 2 Comments »
Monday, February 7th, 2011 by Rex Black
Reader Gianni Pucciani has another good question about the Advanced Software Testing: Volume 2 book. Specifically, he’s concerned with question 2 from Chapter 8:
Which of the following is a best practice for retrospective meetings that will lead to process improvement?
A. Ensuring management commitment to implement improvements
B. Allowing retrospective participants’ to rely exclusively on subjective assessment
C. Requiring that every project include a retrospective meeting in its closure activities
D. Prohibiting any management staff from attending the retrospective meeting
Gianni writes, “I had marked A, but also C. Where is the mistake? I have a feeling on it, but I would like you to confirm. Is C not correct because it is an organizational best practice, and not a best practice for retrospective meetings. A logic trick basically , is that correct?”
Actually, Gianni, the reason C is not correct is because merely having retrospectives does not guarantee process improvements. In fact, I’ve encountered a few situations were organizations were good about having retrospectives, but not so good about management commitment, and thus no improvements occurred.
Tags: retrospectives, software testing Posted in best practices, software quality, software testing | 3 Comments »
Thursday, January 13th, 2011 by Rex Black
From time to time, I get questions about the books I’ve written. I’ve never found a way (at least, one that I thought worked properly) to handle those questions efficiently. Now I have an idea, and we’ll see if it works. If you are a reader of one of my books, and have a question about something in that book, you can send the question to info@rbcs-us.com with the subject line “Book Question for Blog”. Put your question in the body of the e-mail, watch the blog, and within a 2-3 days you should see an answer.
Tags: software testing Posted in software testing | No Comments »
Tuesday, December 28th, 2010 by Rex Black
Throughout 2010, I’ve spent months doing a lot of traveling, and talking to a lot of testers and test managers around the world. I’ve been to various spots in North America, China, Malaysia, New Zealand, Australia, Turkey, and Germany. No matter where I go, I hear two comments fairly consistently from test managers and staff alike: 1) Management is pushing for increased productivity; and, 2) training budgets are tight. For people to improve productivity, they have to improve their skills. So, how can the smart test manager build the skills of her test staff without breaking the bank? Let’s evaluate various options.
To start, you need a skills management plan. First, you perform a task analysis. In a task analysis, you examine the tasks that your staff performs as part of their regular (and perhaps irregular) duties. From this analysis, you then create a list of skills that someone would need to effectively and efficiently perform those tasks.
Second, you create a skills inventory. In this step, for each of the skills you identified, you assess what skills level a perfectly qualified person would have, for each of the positions in your teams. (This assumes that you have some degree of specialization in your teams, in that people are not considered interchangeable, but rather are assigned tasks based on their positions.) You then assess your current team against these skills.
Third, based on this information, you can now perform a gap analysis for the skills in your current team. In other words, what’s the gap between your current team and the perfect team? This tells you where skills augmentation is needed, and thus where training can have a positive return on investment.
Finally, your skills management plan must address how people will apply the skills you intend to improve. This requires an opportunity to put those new skills to real-world use, on real tasks, within a few weeks (at most) of the person obtaining those new skills. It’s a classic worst-practice of training to send people to training courses and then assume that somehow, magically, that training will someday translate into increased effectiveness and efficiency. In such cases, these new skills often molder unused so long that, by the time you need them, the skills are forgotten.
Okay, if you’ve followed these four steps, you now have a specific list of skills that you want to improve, for each member of your team, along with a plan for how to utilize those improved skills. Time to select training options.
The training options you have available constitute both a spectrum and an a la carte menu. The options are a spectrum in that the degree of investment ranges from high to low. The options are also an a la carte menu in that you can certainly select multiple options, not just one.
The first option is live, instructor-led courses. This can involve either sending one or more staff members to a public course, or having the course run at one or more sites in your company. The advantages of live, instructor-led courses are the immediate attention of the instructor (including direct interaction when questions arise and discussions of the application of the concepts to specific situations) and, for on-site courses, the possibility of making the course a hands-on workshop focused on your specific skills gaps. In addition, for some staff members, having them devote their time entirely to training for a continuous period improves their focus and retention. The effectivity of knowledge transfer is maximized, but so is the cost.
A second option, closely related, is what is called a virtual course. In such a course, the instruction happens synchronously, as with a live course. The course is instructor-led. However, the instructor leads the course via a webinar or similar virtual classroom. This can cost less per attendee, but some less self-directed attendees can lose focus over time.
The third option is e-learning. This typically involves some kind of asynchronous, browser-based interactive application. The course should include some kind of presentation (e.g., animated slides) accompanied by a recorded audio lecture. Such courses should also include exercises and some kind of regular check of comprehension of the material. The latter is important because, since the instructor cannot monitor attendee comprehension directly in real-time, the attendees must check their own comprehension. Typical ways to check comprehension include multiple choice questions about the material covered in the last few minutes of the e-learning lecture.
Some types of e-learning are called blended e-learning. Blended e-learning combines webinar-type facilitation sessions with an asynchronous e-learning course. The facilitation is instructor-led, and typically includes anywhere from two to six such sessions. In these sessions, the instructor reinforces key ideas from the course. Facilitation provides attendees with an opportunity to ask questions, and also provides structure that helps to keep less self-directed attendees engaged.
Pure asynchronous e-learning is typically considerably less expensive than live, instructor-led training, sometimes as much as half or a third as much. Indeed, you can often purchase e-learning course enterprise licenses that allow the training of an unlimited number of attendees for a relatively small fixed cost. The addition of facilitation sessions adds cost, but savvy training customers can find ways to balance the cost of facilitation against the benefit.
A fourth option is self-study. In such a situation, an attendee uses books, articles, blogs, podcasts, videos, and web-site materials to learn. The range internet options makes self-study truly attractive, and no one can argue with the price. Buying one or two books and spending a few hours availing oneself of free internet resources is cheap. Of course, the risk is that the attendee will spend time reading what are effectively sales pitches or, worse yet, really bad ideas.
The fifth option is cross-training and other forms of on-the-job training. In such programs, you assigned someone a task—and a mentor—that will allow them to expand their skills. Obviously, the cost of such an approach is low, though remember to take into account the efficiency costs on both the person learning the new skill and the mentor. Even when other training options are used, I suggest that every skills growth initiative should include this option as the last step of cementing the new skills. For example, if someone takes an e-learning course, they can then be assigned a mentor and a task that involves one or more of the new skills they have acquired.
As managers, the economic exigencies of the current economy require that we become more effective and efficient in our use of all our resources. In software testing, people are indeed the most important resource, because software testing work is brain-work. Training, in one form of another, is an essential part of becoming more effective and efficient. To use train your staff properly, start with a proper understanding of what they need to know—and what they currently don’t know. Next, select options such as instructor-led training, e-learning, self-study, and cross-training to ensure proper skills transfer and the application of those skills to real-world problems. If you develop and execute a smart skills-growth plan that covers these elements, you can expect significant improvements in your team’s abilities over the next six to eighteen months.
Tags: software test training, software testing Posted in best practices, software test skills, software testing | 2 Comments »
Thursday, December 23rd, 2010 by Rex Black
As readers of this blog will know, I’ve spent a lot of time in this blog (and in articles, books, and consulting work) on the topic of risk based testing. I recently spent some time in Japan, working with various teams in Sony to implement better risk based testing. Part of that time was spent working with Atsushi Nagata, who is helping teams in Sony put risk based testing into action. Together, he and I have written an article describing some ground-breaking work that they’ve done on risk based test results reporting. That article was just published in ST&QA magazine. You can read it (and comment on it) by clicking here.
Tags: risk based testing, software test management, software testing Posted in risk based testing, software test management, software test metrics, software testing | No Comments »
|