Short Term Measurement Error

Short Term Measure Error

This is a counter warning to my article on magical numbers. Magical Numbers are a useful abstraction – a way of measuring less without learning less. Without finding these numbers, you end up investing too much in collecting detailed measurements that make the real story hard to fathom, but Magical Numbers allow you to collect less, but find out more (because the link between the number and the outcome is clear).

So what could go wrong? The most prevalent problem I see in business is mistaking a short term measure for a long term improvement.

So let’s think about retrospectives. The best kind of outcome from a retrospective is a concrete action containing an experiment along with a clear measure that will tell you whether the experiment is a success. So far, so good. You run your experiment for a week and collect up your first data point. You see a two-fold improvement, break out the champagne and party poppers, and head for the nearest pub.

You have fallen for the short-term measure error – because you haven’t allowed the experiment to run long enough to eliminate noise in your numbers. This is the first kind of short-term measurement error.

This problem is common in Agile teams, who fall into the trap of making a change at the start of an iteration and declare success or failure at the end of the iteration – allowing themselves to fall for the single data-point trap. When making a change, it is incredibly likely to be a false positive in the first iteration (due to the Hawthorne Effect) or be a false negative while people adjust to the change. One data point does not give you the full picture.

Let’s rework that example. You have your experiment and your measure. You are going to hand out a weekly award to the person who closes the most bugs. You measure the result each week and you notice that the winner is closing more bugs than anyone did before the change. After 10 weeks you feel confident enough with the improvement that you break out the champagne and party poppers, and head for the nearest pub.

You have fallen for the second kind of short-term measure error. You are measuring a number that itself shows a local improvement in each time period, but you are ignoring a number that is warning you about a long-term negative effect. Namely, the total number of bugs being closed each week is the same, or worse, than it was before the change.

This problem is found everywhere in business. Commonly manifesting itself in punishment systems (miss this deadline and you’re in trouble), and reward systems (performance-related bonuses, employee of the month awards, etc). Using any of these methods will show a short-term improvement, but the net effect is negative in the longer term. This issue is so common that people will swear by their reward programs and show you proof that it works… but that proof will all suffer from the short-term measurement error. The long term effects of rewards are that they damage business.

So beware of measuring the short-term “green line” and double check that you aren’t missing out on a much more fundamental “red line” – even when using Magical Numbers.

Further Reading:

  • The Human Side of Enterprise – Doug McGregor
  • Get Rid of the Performance Review – Sam Culbert
  • Punished by Rewards – Alfie Kohn
  • Drive – Dan Pink