Archive for the ‘software test skills’ Category
Friday, October 7th, 2011 by Rex Black
I received a request from Martin for help on how to apply orthogonal arrays. He wrote:
Hello,
I am working on putting together testing at work using an orthogonal array. This is the first time I’ve used this method. I read about it in detail in Rex’s books. But I’m having difficulty remembering all the steps. I need a little help to jog my memory. Could you please send me the high-level, bulleted, steps for creating the orthogonal array with my test case data?Thank you.
Sincerely,
Martin
Sure, Martin, you can take a quick listen to the key parts of my webinar on pairwise testing techniques. This webinar addresses how to use orthogonal arrays and other pairwise tools. After you listen to that, I’d suggest you go to www.pairwise.org to find a free downloadable tool. Last I heard, the US Department of Commerce had funded a project to create a pairwise tool (your tax dollars at work!) that will actually do classification trees as well, and I think you might find it there.
In general, Martin, if you are looking for a refresher on test design techniques, a good first stop is the RBCS Digital Library.
Tags: software test design, software testing Posted in best practices, software test skills, software testing | No Comments »
Monday, June 27th, 2011 by Rex Black
Just in time for tomorrow’s and next week’s webinar–and this week’s webinar-style virtual Foundation bootcamp–I can share some advice on audio adjustments from long-time webinar attendee Avner:
…I experienced a personal issue with the gotowebinar sound. The problem manifests as untenable echo on the presenters voice. I don’t know why by default the audio started with this effect ON. After a few minutes of experimenting I was able to find the solution. It’s an easy fix but if you don’t know where to go, it’s a deal breaker. I.e. I think I’ll go to lunch. Included is a screen print of how to fix it [see embedded picture below]. Please share this with Mr Black or whoever needs to know. This has happened to me before on other seminars where I was not able to fix the problem & had a good lunch instead.
Regards & thank you again for an informative webinar,
Avner Uzan
Avner, thanks for sharing this tip. Audio for these webinars can indeed be tricky, both for the presenter and for the listener. I’m sure you’ve helped a few people catch webinars they would other miss…for lunch.
Posted in software test skills, software testing | No Comments »
Monday, May 30th, 2011 by Rex Black
On the heels of the webinar last week, listeners have had lots of comments (all good) and some questions. Here’s an interesting set of questions from listener Stephen Ho. I’ve interspersed my answers in his e-mail, with “RB:” in front to make it easier to follow:
Rex,
Thanks for such Webinar. This webinar did not talk about how to organize and build up a good testing Metrics.
RB: There was a general discussion early on about how to go from an objective to a specific metric and specific targets for that metric. Perhaps you were missing the part about implementing the metrics with specific tools?
However, it provided some interesting points to measure the successful of a testing project, such as: BFE. Just for more realistic, how can we know that a testing metrics is good?
RB: One attribute of a good metric is that it is traceable back to some specific objective. That objective should relate to a process (e.g., finding defects is an objective for the test process), to a project (e.g., reaching 100% completion of all schedule tests is an objective for many projects), or to a product (e.g., reaching 100% coverage of requirements with passing tests is an objective for some products). Another important attribute is that the metric supports smart decision-making and, if necessary, guides corrective action. Yet another important attribute is that the metric have a realistic target.
At here, I have another topic for your interest.
“What is effective & efficiency testing?”
RB: We have to be more specific than this. What are the objectives for testing, as you mean it here? Once you have defined those objectives, you can then discuss effectively and efficiently meeting them. For example, if you define finding defects as an objective, then you can use the DDP metric (discussed in the presentation) as a metric of effectiveness. Cost of quality (which is discussed in various articles in the RBCS web site, such as this one) can serve as a metric of efficiency.
-how to narrow it down to know that our existing testing job is in effective & efficiency ways.
RB: You might want to read the chapter I wrote (Chapter 2) on this topic in the book Beautiful Software. That’s a book worth reading, anyway, because there a number of other good chapters in it.
-What are the right way to measure the performance of a QA?
RB: I assume you’re talking about an individual tester here. If we can define specific objectives for the tester, then we can use the same method to define metrics. Keep in mind the rule about objectives needing to be SMART.
-How can we know that a QA is in competence level?
RB: Check out my book, Managing the Testing Process, 3e, for a discussion about how to use skills inventories to manage the skills of your test team.
-How to increase the productivity of a QA?
RB: This question is too general, I’m afraid. Productive at what? I suggest that you define specific objectives for the test team, and then measure the current efficiency with which those objectives are achieved. At that point, you can make realistic (and measurable) goals for improvement of productivity.
You may have existing webinar or article regarding this topic. If yes, I am thirsty to study your material. Would you direct me how to access to this information. I would definitely provide my feedback to you.
RB: Follow this link for another article on metrics that you might find useful.
Thanks,
Stephen
RB: You’re welcome.
Tags: metrics, software testing effectiveness, software testing efficiency Posted in best practices, software test management, software test metrics, software test skills, software testing | No Comments »
Monday, May 30th, 2011 by Rex Black
Long-time reader Din asked a good question about the ISTQB Advanced syllabus and the test design techniques covered in Chapter 4. He wrote:
Hi Rex,
As far as the ISTQB syllabus is concerned, topic on boundary value analysis & testing is discussed in Foundation and Advanced level. Just want to check whether deeper discussion on extension of BVA/BVT such as robustness testing and robust worst- case testing (and other related approach) of BVA/BVT are covered in our ISTQB syllabus, especially in Advanced level. I raise this as I came across several academic presentations discussing on BVA/BVT of more than one parameter/variables which I didn’t realize so much before this. Hope to get your feedback.
Thanks.
-Din-
Well, Din, I can’t speak for other training providers–and they probably wouldn’t want me to even if I could! However, I can say that the RBCS Advanced Test Analyst and Advanced Technical Test Analyst courses–and the corresponding books–go into the topic of boundary value testing (and the related concept of equivalence partition testing) in great depth. This includes multi-value boundary value testing, along with the use of these techniques in combination with other test design techniques such as all-pairs testing, state-based testing, decision-table testing, and so forth.
Tags: ISTQB Advanced, ISTQB Foundation Posted in ISTQB certification, software test skills, software tester certification, software testing | No Comments »
Thursday, May 19th, 2011 by Rex Black
Following on my most recent post, another one of the Advanced Test Analyst course attendees took the exam this week. Kelly Rasmussen passed on the following advice. While a bit more targetted to the CTAL-TA exam than the other CTAL exams, there are some good general guidelines, especially regarding Foundation-level questions, the weighting of questions, and the typical e-exam experience.
Just took my exam today and I passed also! Don’t know if you all took the test yet or not, but wanted to share my experience with you. Hopefully it’ll be of some help.
Firstly, I would like to say that there was no lockers [at the Kriterion exam center] to lock up my stuff. I didn’t know this going in so I had my purse and cell phone with me. The receptionist took my stuff and put it under her desk, which I didn’t feel all that comfortable doing but didn’t want to go back to my car and put my sutff up. The exam didn’t take the whole 3 hours. I was able to answer all the questions within 2 hours. Then I used next 30 minutes or so going back to the questions that I marked for review. I wasn’t sure if I was quite ready for my results yet, so I sat there for a minute or two deciding whether to click the submit button or not. lol… Yes, I was very nervous to find out if I passed or not, but I was also tired and wanted to go home. So clicked the button, and luckily, I passed.
As far as the type of questions I got, I was surprised to see so many K1 questions from the Foundation Level. I think my first 7 questions or so were from the Foundation Level, which worked to my advantage. There were plenty of scenario questions, which were tricky in my opinion. [People with extensive testing experience will find that helpful] in deciding on the correct answer. Another thing that I thought was tricky is that you had to choose which test technique would be the best one to use depending on the scenario. So make sure you know when it is better to use one technique over another and of course how to figure out how many test cases one would need to test. You want to get as many of the K3 questions correct as possible since these are worth 3 points. I basically ditto what Jennifer said. Go over everything!!! Pay close attention to chapters 2 and 4. There were a lot of questions from these two chapters!!!
I’ll add that, implicit in Kelly’s comments, and Jennifer’s, is the lesson that extensive pre-exam preparation is required. In addition to attending a class or going through e-learning, be sure to spend lots of additional time studying for the exam.
Tags: ASTQB, ISTQB, ISTQB accredited live training, ISTQB Advanced, ISTQB exam preparation Posted in ISTQB certification, software test skills, software tester certification | No Comments »
Friday, May 13th, 2011 by Rex Black
As followers of the RBCS Facebook and Twitter micro-blogs will know, last week we offered an ISTQB Advanced Test Analyst course. The corresponding exam has the mouthful-name “ISTQB Certified Tester Advanced Level-Test Analyst,” typically abbreviated “CTAL-TA.” CTAL-TA is also the authorized resume acronym for those who pass the exam, by the way.
Jennifer Parran of Booz Allen Hamilton took an electronic version of the exam at a Kriterion exam center just this week, and she passed on the following details. This is really good advice for anyone taking an ISTQB Advanced exam, especially the Test Analyst and Technical Test Analyst exams.
Hello –
I just wanted to let you all know I took the exam yesterday and PASSED! To give you an idea of what it was like… I took the electronic exam at the Career Technology Center in Falls Church [Virginia]. You have 3 hours once you click the “Start” button and there is a timer counting down on the screen (try not to let that distract you). (Also you may be in the room with other people taking other exams, some of which are allowed to bring books in and they might be making noise flipping back and forth in the book.)
You can mark questions for “review” if you want to review the answer or if you want to skip a question and come back to it later. You can also navigate backwards and forwards so if you answer a question and want to go right back to it, you are able to do so. Once you’re all done and click the “Submit” button (at which point I felt like I wanted to throw up) it takes a couple seconds and then displays your score and that you “passed”. It also sends you an email with this information which I saw later.
Oh yeah…don’t touch the keyboard while taking the exam! I accidentally hit the spacebar and it backed me out of the exam and I had to get the proctor to log back in and bring the exam back up. THANKFULLY it saved all my answers as I only had an hour left but to play it safe – only use the mouse!
The questions were very similar in style to the sample questions [provided in the RBCS live and e-learning courses and exam prep guides] so definitely do all of those. My questions covered every chapter in the advanced syllabus so you definitely want to read the entire syllabus [as discussed in the class]. Some of the questions seemed more foundational but it was hard to distinguish between K1 Foundation and K1 Advanced. I also had questions pertaining to several of the referenced standards so like Rex said in class – you don’t want to be “sad” when you’re reading one those questions and you can’t remember what it is.
Make sure you are very comfortable with the black-box techniques like we practiced in class because there are quite a few questions that are very long and involve a lot of reading [that involved] these techniques so you don’t want to have to spend a lot of time reading the question and remembering how to do the technique. These are of course the K3 questions.
As far as preparation goes, I read through all of the foundation and advanced syllabus’ once and then skimmed over my “highlights” several more times (once right before I took the exam). I did all the sample questions and mock exams. I skimmed over the referenced standards a couple times. I studied a little Monday night, 3 hours Tuesday night, all day Wednesday, and then read over my highlights right before the exam.
Good luck to all of you and Thanks Rex for all the great explanations and preparation!
Jennifer
By the way, for those of you considering taking an ISTQB exam and still mulling over your exam preparation options, RBCS will have an exciting announcement in the next couple weeks on this topic. Watch our Facebook page, Twitter posts, and our newsletter for more information. If you’ve already decided to take an RBCS course, you can find the course schedule on the home page.
Tags: ISTQB accredited e-learning training, ISTQB accredited live training, ISTQB Advanced, ISTQB CTAL, ISTQB exam preparation Posted in ISTQB certification, software test skills, software tester certification, software testing | No Comments »
Tuesday, April 12th, 2011 by Rex Black
Here’s another good question from a reader. Tingting Ren asks:
2.10. Sample Exam Questions
3. Which of the following is not always a pre condition for test execution?
A. A properly configured test environment
B. A thoroughly specified test procedure
C. A process for managing identified defects
D. A test oracle
I think the answer is C but in the book it gives B.
This is a good question. The answer is “B” because oftentimes testers use exploratory testing and logical (high-level) test cases when running tests, rather than concrete (low-level) test cases that spell out every action to be taken and result to be observed. For example, see Whittaker’s book How to Break Software.
The answer cannot be “A” because the test environment must be configured correctly in order for the results to be reliable. The answer cannot be “C” because we must have an agreed-upon way–a bug tracking system, ideally–for handling failures that we observe when running tests. The answer cannot be “D” because we must have a way of distinguishing expect results from unexpected results.
Posted in software test skills, software testing | No Comments »
Tuesday, March 22nd, 2011 by Rex Black
I received an interesting e-mail from long-time reader, John Singleton:
I’m so proud of my 6-year old, Josh. This weekend, we were playing Wii. Those of you likewise hooked on Wii will know that pushing the Home button on the controller pauses the current game and allows the user to access some configurations and such. It also displays the battery level for all connected Wii controllers. This time, the battery icon had only one bar, with the red color to get your attention. Josh said, “Hey Dad, have you ever noticed that when you push Home, it shows the battery is full for just a second before it shows that it’s empty?”
I had him show me, and sure enough. There on the Home screen, it shows the battery meter as blue and full for just an instant before it refreshes with the red, almost-empty icon. I think I spouted off something geeky about how it probably shows a default value for just a moment while it queries for the actual battery value.
I don’t know if I mentioned, but my son also has some special needs, including Sensory Processing Disorder, which tends to make one much more acutely aware of any kind of sensory stimulus. I wonder if this episode is similar to the kinds of dynamics that have led people to hire individuals with Aspergers’ or Autism Spectrum Disorder for software testing. Or maybe he’s just quirky, like his dad…
Regardless, my heart swells to about ten times its normal size when I hear my six-year-old finding obscure software defects in robust commercial products!
Test on!
( http://en.wikipedia.org/wiki/Sensory_processing_disorder)
An interesting set of questions raised here, John. Certainly, my children are very adept at digital devices, but they don’t seem to show that level of awareness for problems. Is testing an innate skill? Do certain traits which are thought of as “disorders” in some contexts actually provide skills in other contexts?
Comments, anyone?
Tags: software testing skills Posted in software test skills, software testing | 5 Comments »
Tuesday, December 28th, 2010 by Rex Black
Throughout 2010, I’ve spent months doing a lot of traveling, and talking to a lot of testers and test managers around the world. I’ve been to various spots in North America, China, Malaysia, New Zealand, Australia, Turkey, and Germany. No matter where I go, I hear two comments fairly consistently from test managers and staff alike: 1) Management is pushing for increased productivity; and, 2) training budgets are tight. For people to improve productivity, they have to improve their skills. So, how can the smart test manager build the skills of her test staff without breaking the bank? Let’s evaluate various options.
To start, you need a skills management plan. First, you perform a task analysis. In a task analysis, you examine the tasks that your staff performs as part of their regular (and perhaps irregular) duties. From this analysis, you then create a list of skills that someone would need to effectively and efficiently perform those tasks.
Second, you create a skills inventory. In this step, for each of the skills you identified, you assess what skills level a perfectly qualified person would have, for each of the positions in your teams. (This assumes that you have some degree of specialization in your teams, in that people are not considered interchangeable, but rather are assigned tasks based on their positions.) You then assess your current team against these skills.
Third, based on this information, you can now perform a gap analysis for the skills in your current team. In other words, what’s the gap between your current team and the perfect team? This tells you where skills augmentation is needed, and thus where training can have a positive return on investment.
Finally, your skills management plan must address how people will apply the skills you intend to improve. This requires an opportunity to put those new skills to real-world use, on real tasks, within a few weeks (at most) of the person obtaining those new skills. It’s a classic worst-practice of training to send people to training courses and then assume that somehow, magically, that training will someday translate into increased effectiveness and efficiency. In such cases, these new skills often molder unused so long that, by the time you need them, the skills are forgotten.
Okay, if you’ve followed these four steps, you now have a specific list of skills that you want to improve, for each member of your team, along with a plan for how to utilize those improved skills. Time to select training options.
The training options you have available constitute both a spectrum and an a la carte menu. The options are a spectrum in that the degree of investment ranges from high to low. The options are also an a la carte menu in that you can certainly select multiple options, not just one.
The first option is live, instructor-led courses. This can involve either sending one or more staff members to a public course, or having the course run at one or more sites in your company. The advantages of live, instructor-led courses are the immediate attention of the instructor (including direct interaction when questions arise and discussions of the application of the concepts to specific situations) and, for on-site courses, the possibility of making the course a hands-on workshop focused on your specific skills gaps. In addition, for some staff members, having them devote their time entirely to training for a continuous period improves their focus and retention. The effectivity of knowledge transfer is maximized, but so is the cost.
A second option, closely related, is what is called a virtual course. In such a course, the instruction happens synchronously, as with a live course. The course is instructor-led. However, the instructor leads the course via a webinar or similar virtual classroom. This can cost less per attendee, but some less self-directed attendees can lose focus over time.
The third option is e-learning. This typically involves some kind of asynchronous, browser-based interactive application. The course should include some kind of presentation (e.g., animated slides) accompanied by a recorded audio lecture. Such courses should also include exercises and some kind of regular check of comprehension of the material. The latter is important because, since the instructor cannot monitor attendee comprehension directly in real-time, the attendees must check their own comprehension. Typical ways to check comprehension include multiple choice questions about the material covered in the last few minutes of the e-learning lecture.
Some types of e-learning are called blended e-learning. Blended e-learning combines webinar-type facilitation sessions with an asynchronous e-learning course. The facilitation is instructor-led, and typically includes anywhere from two to six such sessions. In these sessions, the instructor reinforces key ideas from the course. Facilitation provides attendees with an opportunity to ask questions, and also provides structure that helps to keep less self-directed attendees engaged.
Pure asynchronous e-learning is typically considerably less expensive than live, instructor-led training, sometimes as much as half or a third as much. Indeed, you can often purchase e-learning course enterprise licenses that allow the training of an unlimited number of attendees for a relatively small fixed cost. The addition of facilitation sessions adds cost, but savvy training customers can find ways to balance the cost of facilitation against the benefit.
A fourth option is self-study. In such a situation, an attendee uses books, articles, blogs, podcasts, videos, and web-site materials to learn. The range internet options makes self-study truly attractive, and no one can argue with the price. Buying one or two books and spending a few hours availing oneself of free internet resources is cheap. Of course, the risk is that the attendee will spend time reading what are effectively sales pitches or, worse yet, really bad ideas.
The fifth option is cross-training and other forms of on-the-job training. In such programs, you assigned someone a task—and a mentor—that will allow them to expand their skills. Obviously, the cost of such an approach is low, though remember to take into account the efficiency costs on both the person learning the new skill and the mentor. Even when other training options are used, I suggest that every skills growth initiative should include this option as the last step of cementing the new skills. For example, if someone takes an e-learning course, they can then be assigned a mentor and a task that involves one or more of the new skills they have acquired.
As managers, the economic exigencies of the current economy require that we become more effective and efficient in our use of all our resources. In software testing, people are indeed the most important resource, because software testing work is brain-work. Training, in one form of another, is an essential part of becoming more effective and efficient. To use train your staff properly, start with a proper understanding of what they need to know—and what they currently don’t know. Next, select options such as instructor-led training, e-learning, self-study, and cross-training to ensure proper skills transfer and the application of those skills to real-world problems. If you develop and execute a smart skills-growth plan that covers these elements, you can expect significant improvements in your team’s abilities over the next six to eighteen months.
Tags: software test training, software testing Posted in best practices, software test skills, software testing | 2 Comments »
Sunday, December 19th, 2010 by Rex Black
I spend a lot of time traveling the world and talking to IT executives, managers, and staff members. Even with the economy struggling, a surprising number of managers are hiring new staff. I did a webinar on hiring in November, and over 300 people registered to attend. You can listen to the recorded version of the webinar here.
Hiring is great, and it’s sure better to hire than to fire. However, with hiring comes the possibility of the dreaded hiring mistake, the opening scene of one of the manager’s worst nightmares. Since interviews are so important in selecting the right people to hire—and, ideally, deflecting the hiring mistakes—you need to be able to conduct effective interviews. What are the important elements of an effective interviewing process?
One element is a good job description. In addition to the obvious sections of this document, it should answer for following questions:
- What tasks and responsibilities are involved?
- What and how much experience?
- What specific skills are needed, and what is the career path?
- What training, education, certification, or licenses are required?
- What is the start date?
- If unusual, what are the hours, the dress code, and the travel requirements?
The first four of these sections are easy to complete if you have done a task analysis and skills inventory for your team. Finally, avoid a classic worst-practice of job descriptions by distinguishing between required and desirable qualifications.
With the job description in place, you can start interviewing candidates. (I assume that your HR department will handle the actual recruiting activities that bring candidates and their resumes your way.) To make the process efficient—to be blunt, to minimize wasted time on in-person interviews with unqualified candidates—I recommend using a phone interview (perhaps more accurately called a “phone screen”) to start.
This brings us to an important point that applies to the interview hiring process. While we were all taught to be polite when growing up—and you should be polite in this entire process—we do need to turn off the politeness instinct that causes us to pull back and redirect questions when we sense that the other person is uncomfortable. Remember, the objective of the hiring process is to hire the most qualified candidate, not to make all the interviewees completely comfortable.
So, in the phone screen, you should explore the person’s experience and qualifications in a polite but incisive way. In particular, weed out people who pad their resume or inflate their experience. If a buzzword or acronym is on a resume, check that the candidate has meaningful mastery of the subject. Carefully evaluate all claimed expertise and experience in the phone interview. Be especially skeptical if a skill is listed without any description of a particular job where the skill was applied. Also, you may want to verify degrees, certifications, and licenses if these are important. If the candidate passes the phone screen, then you can schedule an in-person interview with yourself and others on the team in which the person will work. Key managers who will work with the candidate are often included. Again, I assume your HR team can help you set up the interview participants and schedule.
In the in-person interview, include a mix of qualification questions, behavioral interviewing, and audition interviewing. Qualification questions are those with correct and incorrect answers; e.g., “What programming language is primarily used to write the Linux operating system?” Pick skills and knowledge that relates to the actual work the successful candidate will perform, then develop a set of good qualification questions for them. Don’t make these questions so hard that no one gets them right; the objective is not to pose the riddle of the sphinx to the candidates, but to measure their level of skill.
Behavioral interviewing is concerned, obviously enough, with how a person will behave on the job. Behavioral questions are open-ended, and often require candidates to relate their past experience to the job you are considering them for. For example, here are three possible behavioral interview questions:
- Tell me about ways that past managers have enabled you to do your best work.
- How will what you learned on project XYZ help us here at our company?
- Of all the jobs you’ve had, what was your most enjoyable job, and what did like the most?
Depending on the culture, workstyles and values of your company, the right answer for any of these questions could be the wrong answer for other companies.
You should also include audition interviews. An audition interview is where you set up an actual work task—or a scaled down version of it—and ask the candidate to perform it. For example, if you are hiring a test engineer, you could ask the candidate to create a test based on a requirements specification or user story from a past or current project. As another example, if you are hiring a test technician, you could give the candidate a real test case–written to the expected level of detail as most test cases–and have the candidate test a real system. While audition interviews might sound complicated, they’re actually easy and fun once you get the hang of them. I have hired some really good people—and not hired some people I might otherwise have mistakenly hired—based on their audition interviews.
Let me conclude with some cautionary notes on avoiding classic, all-too-common interviewing mistakes. One of these mistakes is scaring off the candidates. This can happen when people deliberately intimidate, stress out, or just plain weird out candidates in interviews. It can also happen when interviewers who are having a bad day vent their frustrations; you should be honest about the good and bad aspects of the company, but don’t painting a bleak and bitter picture. Another classic mistake, which I mentioned earlier, is being afraid to ask tough questions. Probe for weakness in the skills and experienced claimed in the resume and the interview. Keep your ears open for fudging, vagueness, attempts to redirect the question, and using incorrect technical terms. Another classic mistake is to break the law. Be sure you know what questions and topics you can’t bring up. Again, turn to your HR department to help you (and others who will be involved in interviewing) understand what topics are acceptable in an interview.
Successful teams are built by smart managers who have mastered the art of effective interviewing. Effective interviewing starts with a good job description, as that document defines clear requirements for the position. Effective interviewing should also include a good phone screen. The interview process should include qualification questions, behavioral interviewing, and an audition interview. Don’t scare off the candidate or break the law, but do ask polite yet challenging questions. By including these essential elements of effective interviewing, you can be a smarter hiring manager, too.
Tags: hiring, interviewing, software testers Posted in best practices, software test skills, software testing | No Comments »
|