I had a query come in about a sample exam question in our Advanced Test Manager course. Shukti asked me to confirm the answer to the following:
A given organization is using reviews for development work products like code, requirements and design specifications; test work products like test plans, quality risk analyses, and test design specifications; and, documentation and help screens. The review processes have been in place for two years and are delivering excellent financial, quality, and schedule benefits.
You are attending a management team meeting. A senior executive raises the need to update the objectives by which the individual contributors are measured on their yearly performance evaluations. He suggests using defect counts from review meetings. He circulates a draft plan. Under the plan, people will be rewarded based on the number of defects they find in reviews. Further, people will be penalized if items they have produced incur too many defects during reviews.
Which of the following is a psychological factor affecting review success and failure that is likely to cause such an initiative to undermine the current success of the reviews?
- Scrutinize the document and not the author.
- Focus all participants on delivering high-quality items. [Correct]
- Try to find as many defects as possible.
- Assemble the right team of reviewers.
Here’s why that answer is correct. Bonuses and other financial incentives/disincentives are based on the assumption that people are basically rational economic actors who will behave in a way to maximize their financial situation. (Now, we can have a whole separate discussion about whether this assumption holds perfectly in all situations, but it really doesn’t need to be perfect, as long as more often than not it is true.)
So, in this scenario, what will the reviewers be focused on? Not on delivering the highest quality items, but on finding the maximum number of bugs in each item and “claiming” those bugs for themselves (i.e., squabbling with other reviewers over who should get credit.) What will the authors be focused on? Arguing about every single bug that reviewers report, trying to insist that the document is perfect. None of these behaviors is supportive of increased quality, and the authors’ behaviors are directly contrary to that goal.
In short, a really bad idea. I wish I could say that I never saw organizations make this kind of mistake with process metrics (i.e., mistaking a software process metric for a people metric), but unfortunately it is all too common.
Tags: metrics, people metrics, process metrics, reviews
Rex Black is President of RBCS (