Paradigm Shifts In Software Development

If you think the term “agile” has been misappropriated in the software development industry, spare a thought for Thomas Kuhn who created and subsequently lost control of the term Paradigm Shift.

Kuhn popularised the term in his science history essential The Structure of Scientific Revolutions (1962) to describe how one science restructures in the wake of inexplicable anomalies. The very short version goes like this:

  1. “Pre-Paradigm” (or old-paradigm) – Most work undertaken during the pre-paradigm stage assumes the model is correct. Research simply refines the already known. Anomalies are largely assumed to be a mistake in the experiment or in the execution of the experiment.
  2. “Crisis” – Anomalies are eventually recognised as being a discovery that doesn’t fit with the paradigm (i.e. it is being invalidated).
  3. “Paradigm” – A paradigm shift occurs and a new model becomes the accepted one.

There is a major point to bear in mind here. The new paradigm is not “correct” and the old one “incorrect”. The new one is simply more correct than the one it replaces – and the new paradigm owes its existence to the old one as it may never have existed without the combination of the old paradigm and the recognition of anomalies. Each iteration moves our understanding to a more correct model of our world.

And here is another fundamental point. These revolutions, or paradigm shifts, often take an entire generation in science, because people clutch the old model long after it has been invalidated. In order for the new model to become the paradigm for a particular science, the old generation and their old beliefs must be replaced with a new generation who are more ready to accept the new model.

Now let’s get back to software. The connection may seem very obvious and I apologise if I am over-stating the case; but this is exactly why so many companies fail when adopting agile methods. The older generation (by which I mean those who are heavily invested in the old model, irrespective of age) cannot accept that teams can self-organise, they can’t learn to trust a team, they don’t want the people doing the work to make decisions about the work. They are used to dictating solutions, not elucidating constraints.

This is why there is friction and frustration between the people who only understand heavyweight top-down methods and those who have grown with roots nourished by the body of evidence that says this isn’t an effective way to manage knowledge work (or professional work as Peter Drucker described it).

Those of us who accept the new model are lucky to have developed our thinking amongst excellent books by Tom DeMarco, Timothy Lister, Kent Beck, Fred Brooks, Robert C Martin, Gerry Weinberg, Jim Benson, Andy Hunt, Steve McConnell, Peter Drucker, John Seddon, Taiichi Ohno, James Womack, Mary and Tom Poppendieck, David Anderson, Covey, Bruner, Rogers, Maslow, Miller… the list of books is massive. We don’t realise how strange our way of thinking must appear to the people who were taught that managing the kinds of work we do is best done by dominant and aggressive behaviour.

But here is the problem. We can’t keep waiting for entire generations to die out to accept new ways of working. We know this because we know that reducing cycle times provides us with more learning opportunities. If we use the natural attrition method of getting the industry agile, it will be another whole generation before the next method emerges. We all know deep down that agile is not the final version of the truth – it is simply one more paradigm in the right direction (if you think that it is the panacea for all software development, you are going to be the problem for the next generation in our industry – no better than those who frustrate you now).

So we need to reduce the cycle time for change in our industry. The only way we can do this is by being sympathetic to those fixated on the old model. We cannot wait for them to die out – we have to invest in them and help them to discover the new model and its benefits. If we can bring them along, we can accelerate the adoption of new ways of working. This should be easier in software development than some other industries because change is so commonplace.

So think of one person who you can help up into the agile model and spend some time with them to help them make the transition. I really want to know what will be next after agile and lean – but unless we speed things up we won’t get to see it.