Process

The DITE Cycle: Data Insight Theory Experiment

This is not my idea, though I’ve named it the DITE cycle in opposition to other available alternatives. There are lots of organisations and individuals doing something along these lines and they all give it different names. There are echoes of The Lean Startup, flashes of Spotify’s DIBBs, the velvety touch of Impact Mapping, and whiffs of the golden PDCA cycle (which has been re-invented and re-named many times before).

The DITE cycle sums up all of this into the following four stages, which repeat infinitely:

  1. Data
  2. Insight
  3. Theory
  4. Experiment

The cycle is simple. You have some information. You realise something that the information might suggest. You formalise that realisation into a testable statement. You test it. The data that pops out of the end of your first cycle feeds straight back into the start of the next one. Boom.

That’s the DITE cycle. Data -> Insight -> Theory -> Experiment.

Data -> Insight -> Theory -> Experiment

Start With Small Wins

This cycle emerged quite informally and organically in our organisation. I was keen to land the first official product decision based on real data when I arrived at this organisation and I very quickly landed a small win. A small change had been made that increased the conversion rate of a particular page, but some customers wanted to revert the change because they didn’t like how it looked. This is pretty common when you have a product with a large customer base. Because you only hear from the customers that don’t like a change, the information many decisions are based on can be skewed. Enter the data! By sharing the experiment with our customers, we kept the change. There are two important pieces here. First, we knew that the change had a positive impact on the product. Second, we could explain it expertly.

Although this was the smallest of wins, it was a major milestone. We repeated the small-wins a few times to help everyone adjust to the change in temperature and we published them on a shared wiki so everyone could refer back to them.

Bigger Victories

Once a number of small wins have been recorded, you can repeat the exact same process on something bigger. This gets tricky if you don’t bring people along on the journey. You can’t use the DITE cycle aggressively because you’ll be defeated by stealth change. For DITE to work, you need to follow the cycle and have a theory before you make the change. The worst possible subversion of the process is making the change first and then justifying it by selecting data. If you don’t see the change you expected, but some other change instead, don’t jump to any conclusions; you’ve just gained an insight and need to write a theory so you can run an experiment to test it. Don’t try to short-circuit the cycle.

You can only conclude the theory under experiment… not any other!

You need to write down the theory and ensure you are measuing the right things before you make any changes. Once the impact of the change can be measured, you learn something. If the change had a positive impact you get a double bonus; you learned something and you got an improvement. If the change is neutral, you can at least scrub-out the theory. If the change is negative, you’ve learned something that’s actually very valuable – in fact, this is one of the best outcomes because it typically means you’ve disproved something that people would otherwise have taken-for-granted was true.

Publish the result of every experiment whether the impact on the test metric is positive, neutral, or negative.

Once you get into the swing of using the DITE cycle, look for the following big victories:

  • The first product change that gets rolled back because the experiment disproves the theory and you want to recover from the negative effect of the change
  • The first feature request that you don’t do because of a prior learning

DITE by Default

Eventually, you should find that you are following the DITE cycle by default. Customers, execs, and individual contributors will all be asking for the information generated by this cycle. The theory is a great directing vision for change and sets a clear goal for the work in process. The return on investment comes either in the form of a correct theory providing the desired outcome, or the increased understanding and expertise from disproving the theory.

So, remember:

  • Use your data to gain insights
  • Use that insight to form a theory
  • Test that theory with an experiment
  • Use the experiment to generate data