I rather flippantly posted this waterfall vs agile graph online this week, along with the equally flippant comment: “Are we still discussing it? Waterfall loses to Agile even based on Waterfall-style metrics”.
So I thought I had been at least acknowledge that this was rather off-hand and also provide The Longer Version™.
The reason that this graph is striking isn’t that as far as success goes, Agile rises three times higher than Waterfall on this graph – although perhaps that is striking if you haven’t worked with both Agile and Waterfall. The thing that surprises me isn’t actually shown on the graph.
If you are involved with any Agile communities online, you’ll notice a common thread of discussions with a tone that demands statistics that prove that Agile is any better than the established and proven old ways. People will bemoan the lack of empirical evidence or scientific tests that prove Agile is better.
The thing is, you can’t really scientifically test any method. As soon as you start measuring people, the chances are the measurements will improve no matter what you do because as soon as people know they are being measured they behave differently. Depending on what you measure, you’ll get different results. You could try testing people without their knowledge, but this isn’t very ethical and you would still have trouble trying to design a good test.
Do you try and create two teams, one Agile, one Waterfall and make them write the same software to see which is better? How do you know the teams are comparable? If you try the test by using the same team, whatever method they try second will turn out better because now they have experience in building the software once – and if you make them build something else how do you ensure it is a fair test?
I haven’t even scratched the surface on the reasons why it is impossible to get clear and unchallenged results when testing two methods of working in an environment where people, knowledge and thinking are the key components. If we were testing production, it would be easy to measure many aspects including cost, quality and time to make the products – but the human race does not yet know a good scientific way to measure two methods of producing “knowledge-work”.
But let’s go back to our graph, accepting all of the imperfections therein. The one thing it does tell you is this:
If you need to justify your choice of methodology, you would have to make a really strong case if you wanted to use the method that has just a 14% chance of success. It isn’t a case of justifying Agile over Waterfall, it is case of explaining how you will ensure that your Waterfall process will somehow beat these numbers.
I’m not saying everyone has to do Agile – let’s face it, if a methodology came along tomorrow that gave you a better than 50% success rate over the long term you’d be very interested – but the illusion of Waterfall being the stable, established, and successful way of building software needs to be abandoned.