As part of the continuing series of video blog entries, I’m throwing out to the wider software testing and quality community a question: How do we balance the desire–especially in Agile and Lean/Agile projects–to enjoy the efficiency of lightweight record-keeping, while at the same time having enough visibility into project, process, and product status, including testing and quality metrics?
For a little video context on my question, check out: How to Balance Metrics, Lean, and Agile when Measuring Software Testing and Quality
I’d be interested in hearing from people as to how their projects are achieving balance here, and also in anecdotes about when they are not achieving balance.
Tags: Agile, Lean, software metrics, software quality, software testing
Rex Black is President of RBCS (
I think some of the premises are confused.
1. Lean does not mean lightweight.
2. Lean means reducing waste along the value stream. Unless an effort has been made to tie tracking and measuring to value, they are waste. In most cultures, they compound things by giving a false sense of security. Temporary measurements can be useful to explore explicit problems, but you don’t measure everything in sight. Tracking bugs is usually not lean, because it costs more to file the bug report than to fix the bug. One should track only those bugs whose cost exceeds a certain threshhold — unless you are gathering data for a focused Kaizen exercise.
3. If you’re going to measure something, let the machine do it fore you without human intervention. That, in some sense, is the core Lean principle: to give the machine human intelligence to be able to detect known problems, involving actual humans only when problems arise — as with automated test batches. However, it cannot really support the kind of kaizen thinking and innovation that can arise from finding new problems — as is central to exploratory testing.
3. There is a fine line between observing and measuring. Agile is all about making everything transparent so you can fix it. As Yogi Berra said, you can observe a lot just by watching. This doesn’t necessarily mean using formal metrics. Too many engineers (and manager) try to use metrics to prove what is common sense to everyone, and then they overlook the real common sense things. They often end straining out gnats and swallowing camels.
4. The most important metric is value, and few projects measure it. They tend to measure only cost and schedule. You should observe value and make sure that your process changes increase long-term overall value. Any other metric is just local optimization unless you have a theory or a preponderance of data to support otherwise.
These principles are only a step away from Lean 101 and Agile 101. Read Taichi Ohno’s “The Toyota Production System” for more of the underlying theory.
Jim makes a lot of good points, all worth considering.
Yes, bloated, beside-the-point metrics programs happen; this is why it’s important that every metric be traceable back to one or more strategic or tactical objectives.
Yes, people do fall prey to “local optimization” where they improve a process only to limit impede the large objective; this is why considering the possible negative implications of measuring something is important.
Yes, automation of metrics gathering is important, so that people focus their energies on the real work, not the measurement of the work.
Yes, failing to have a balanced set of measures (e.g., only measuring schedule and cost) happens; this is why having quality metrics is so important, because otherwise key project decisions will only be made based on cost and schedule metrics, which is one of the problems my current client is trying to address by having testing and quality metrics.
It’s important to remember that testing and quality metrics are not just about assessing project and product status. We also need them to assess the effectiveness and efficiency of the test process (see http://www.rbcs-us.com/blog/2010/08/12/assessing-software-testing-processes/) and the quality capability of the overall development process.
Without metrics, it’s easy to draw conclusions that sound reasonable but are wrong. I see from time to time in my consulting work. People make assertions, and they sound very reasonable, but the data (once we unearth it) says otherwise.
Finally, while I value the practice of getting good ideas where you find them, I think we need to be very careful as software professionals about trying to find useful lessons in manufacturing. We’ve seen the inherent difficulties of doing so before, in the application of Total Quality Management, Six Sigma, and other quality techniques that grew from manufacturing; there are some good ideas there for software professionals, but also some ideas that lead you astray. In the case of Toyota in particular, their confidence on their manufacturing processes to assure quality in their software may well be misplaced (see http://www.rbcs-us.com/blog/2010/07/15/what-testers-can-learn-from-toyota%e2%80%99s-woes/)
Agile and metrics
First of all, I think that agile is a very dangers word. Too often I hear the “Agile” world shouting down basic good test practices with frames and buss words. Too often I hear Scrum Masters and Scrum coaches who do not have their fingers down in the mud say again again, do TDD and that it.. no need for follow up processes and metrics.
I agree, Lean means reducing waste.. or cut the crap other will say. Normally I sum this up to keep it simple stupid, but with that I mean be smart and don’t waste other people’s time.
From my daily life in 4 Scrum teams where we get our requirements features from User Stories – I’ve found out the following about metrics:
1) First of all we do need metrics to cover the User stories. Normally the User story is cut down to smaller features and we need metrics to keep track on this on a Business level, so that we can prove that a User story is Done.. Only with test we can provide this information, and with some metrics on this high level we and the team can actually use this as a pointer on how well we are doing – we can call this a form of Test Driven Development (not TDD on unit test level, but still test as a driver). And for me these test gives much more values than the Unit Test TDD.. And the customers don’t really care about unit test or how we code. All they care about is that we give them the features they pay for, and the only way we can prove that is test the features on high level.. If you use the Agile Quadrant which I use a lot, I can say we need metrics from the quadrant 2 supporting the team while building the product.
2) Do we need metrics from Quadrant 1, which is the unit test level, technical but supporting the team. Yes I think we need metrics here as well, but most development tools gives us a lot here, we can see, test coverage and lots of other things using the tools which can help us becoming better on this low level as well— but we need to be very clear here. We can have 100% test coverage, and we can have no defects when we run these test, but when we show the customer the system they turn the fingers down and say.. no this is not what we ordered – people do make mistakes, and people do misunderstand each other, and developers do make decisions which are not 100% agreed on… this is just the way we human are build. But we do need metrics on this Agile Quadrant level 1, mostly to become better developers, and to get a higher quality in the code.. Cleaner code you can say.
3) On the Agile Quadrant 3 level we are in the exploratory test, UAT and usability level – this is where the product is finished and the user or other are actually trying out the product. And we need metrics to help us here as well – We need usability checklists, and we need a way to let the user/customer come with new features, changes (yes we are agile) ect ect. These metrics will then be used as input for the product backlog or to fast changes in the features we want to release.
4) And finally we are in the Agile Quadrant 4, where its technical test on the finished product, like performance, stress, load, security ect that we test. And we need these data both for use on the product we have just finished, but also for future use so we can compare results.
My main observation is that in the traditional projects I have worked on, we did have lots of metrics, but we couldn’t use them that much as time changes. The same metrics, maybe a little bit less, are much more useful in the agile developments world as we might make use of the knowledge already in next sprint – meaning we learn quicker.. and that is also the point of going agile – the learning organization.. There are two things we human are very good at.. Making mistakes and forget, and by use of metrics we can help on the last isue atleast.
Steen, thanks for this reply. I think the way you’ve illustrated the use of various metrics in Agile is useful. I also agree strongly with your main point, which is that people make mistakes and then forget those mistakes, and it’s only through the use of metrics can we learn to do better. That’s exactly right: Without some way of measuring process capability, we cannot effectively and efficiently improve that process, since, without metrics, we are only making subjective guesses about what improvements to make.