There are an increasing number of articles appearing about how a software development method that has gained significant traction may be dangerous… “Scrum didn’t work for us” or “Lean Start Up nearly killed us” – or many other dramatic examples.
So how did we get here?
Agile was a special little niche for a while and there were lots of success stories. Some independent studies found it had a good success rate. Agile companies were leap-frogging traditional organisations. Agile got so good, in fact, that it was no longer possible for the big dusty companies to ignore it.
When agile methods went mainstream, the reports started flooding in about failed adoptions. Of course they did! Those old command and control companies are able to smother any software development method in bureaucracy.
Here is how you go mainstream with a software development method.
Stage one. A team experiments and creates an evolving method that works really well for them. So far, 100% success rate.
Stage two. The team share how they are working with the community. It resonates with a few other teams who continue the ideas, adapting them in their own unique ways. So far, still very close to 100% success rate… as it would likely have been without the new idea – because these teams are successful teams.
Stage three. The ideas spread. People get jobs at companies who are “adopting that idea” and teams of mixed enthusiasm use the idea under the leadership of keen experts. Quite a few people are won over by what they see. The success rate starts to dip a little because the ideas are compromised. “You can do that idea, but you can’t do that bit because we have never allowed more than one deployment a year…”. It is still around 80% though.
Stage four. The idea is getting noticed. The adoption may still be less than 40%, but everyone has heard of it now. The companies who made money training and certifying old ideas start to offer services for the new ideas. The idea begins to calcify because the next wave of adopters are demanding a manual. The success rate doesn’t immediately suffer because although it “doesn’t feel like the real idea” it still works for a lot of people.
Stage five. It goes mainstream. Adoptions in traditional organisations are dictated by managers who know they won’t be fired for picking this “industry standard” idea. The teams are pushed into the new way of working by being sent on a course ran by the training companies that recently made the switch. The ideas are compromised in multiple ways, because the organisation has no intention of actually changing anything. Teams get frustrated by being forced to use the idea, and then by not really being allowed to use the idea. They pass audits on the idea because the parts of the manual that can be audited on a check-list have been implemented. The heart of the idea can’t be audited because it is more philosophical or principles-based. It takes a while (because they don’t measure often on 3-year projects) – but the failure rate is getting ready to explode. The later you adopt, the more likely it is that you are the kind of organisation that can make anything fail (either entirely, or by blowing way too much money on it, or by taking far too long).
This is not a new concept. Kuhn describes a similar problem in science where the new model is never adopted by the incumbents. You just wait for them to die out so the next generation will be free from their old ways. These failure cases that emerge from dysfunctional organisations don’t matter. These organisations will die out and be replaced with new ones founded on the new ways of thinking.