Building software to last 1,000 years

Our cities do this, why not our software?

In Vernor Vinge's Zones of Thought novels, set in the far future, there is the notion of a 'programmer archaeologist'. In the words of one of the characters:

Basically, when hardware performance has been pushed to its final limit, and programmers have had several centuries to code, you reach a point where there is far more signicant code than can be rationalized.

After this point, writing new code wasn't just unhelpful — it was unnecessary. These programmer-archaeologists merely searched around for tools from the long history of programming that might help them with their current problems, analysing and adapting them for their current needs.

Vinge was right about the direction, but wrong about the timeframe. We are closer to this world than you might think. Programming is increasingly about reading, testing, understanding, and then properly reusing existing code.

I was recently told a story about a hidden-away finance system that needed to be converted from calculating with Pound Sterling into Euros. They contemplated rewriting it, but decided instead to create another system to ‘wrap-around’ the current program and manage the currency conversions.

When the project got started, however, they realised that the original system was itself a wrapper around an even earlier system — a system which calculated with the pre-decimal currency pounds, shillings & pence.

The natural reaction of most is to say — well, another quote from Zones of Thought:

"We should rewrite it all," said Pham.

"It’s been tried," corrected Bret, just back from the freezers. "But even the top levels of fleet system code are enormous. You and a thousand of your friends would have to work for a century or so to reproduce it." Trinli grinned evilly. "And guess what—even if you did, by the time you finished, you’d have your own set of inconsistencies. And you still wouldn’t be consistent with all the applications that might be needed now and then."

Just one year after this was published, Joel Spolsky would famously write Things You Should Never Do:

The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. […] When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work. You are throwing away your market leadership. […] You are wasting an outlandish amount of money writing code that already exists.

Spolsky, like Vinge, was wrong about the timescale — but in a different direction. Shortly after this GDS launched and performed one of the most dramatic rewrites in recorded history. This took intense will, herculean effort, years of work, and ultimately was very successful.

But by virtue of its success, it has expanded itself, created more code, infiltrated more services. Could we imagine rewriting GOV.UK now? Possibly, but not for much longer. Could a private enterprise entertain such an ambitious project, even as GOV.UK in 2010? Certainly not.

We are passing an event horizon, gradually, imperceptibly. Major rewrites are moving from ambitious, to being great works of major governments, to the edges of inconceivable.

---

To handle this new reality, we will need to find new kinds of architecture.

Consider this map of the City of London:

This is the product of 1,000 years of accumulated history. Architects and planners have periodically looked at this and thought things like "what a mess!” and “we should rebuild it all!" It's been tried. One notable plan came after the Great fire of 1666, from a man name Christopher Wren:

Out of his work, came St Paul's Cathedral — but not a remapped city. A later attempt came out of post-war architectural modernism, with ideas like this:

A dramatic vision, at its core about segregation by speed. Out of this came our London Underground Railway, offering fast transport across the city, and a plan to thread the streets with a network of raised walkways and podia to raise the slower pedestrian above motor traffic.

The result is another half-renewal. The underground has been a phenomenal success, whereas the raised walkways exist only in small pockets in the centre of the city, relatively unknown. People just didn't want to walk that way. City Planning Officer Peter Wynne shared in the documentary Elevating London:

I think it took rather a while for the shortcomings to kick in, and people to realise that you were often better off taking existing societies and adapting and refurbishing them rather than starting from scratch on a clean piece of paper.

In a 2004 essay Martin Fowler takes inspiration from the strangler fig to propose a more progressive approach to rewrites. The strangler fig is a plant that reaches down through the boughs of existing trees, gradually constricting and suffocating them, eventually taking their place. The analogy to software is that instead of rewriting an application straight out, you begin creating a new application around the edges of the first, progressively replacing more and more until none of the original is left.

(Source)

This story is charmingly human. The fig wins. The credits roll. The end. Reality is more complicated — sometimes the fig loses. Sometimes the trees live for hundreds more years, reinforced and protected from storms by the surrounding roots. Even where the fig wins, the tree has lent it its essential form — living on in structural memory.

Which is not to say Fowler was wrong. GDS used progressive rewrite techniques extensively. But we should expect that over time these rewrites will take longer and longer, outliving first jobs, then careers, then people. They will start bold and full of vision… but never complete. Software will go from tree to forest, with its continual, chaotic renewal. A perpetual stew, the perpetual rewrite.

---

They have recently begun, after decades of hiatus, adding new sections to the raised walkways of the City of London. Below you can see a new section snaking along a wide road called London Wall, named as it follows the ancient defensive walls of the city. Each structure is, like the fig, vying to replace the former. Walls of defence, roads of commerce, streets of communication.

(Source)

Looking at this photograph you can see rewrite after rewrite, hand after hand, incomplete vision after incomplete vision. This is not a disaster, it is a huge success. We used to think the great authors of the city would be architects, but they turned out to be a new profession: the urban planner, one of the most complex and interdisciplinary professions there is.

This is what the software of the future will look like. Not because we will have failed, but because we will have succeeded.

The software engineer of the future will have to adapt too.

---

Next time we’ll be talking about language. Has the programming language of the year 2100 already been written? How fragmented will the landscape be? What forces will shape this world? Through these conversations we’ll continue to work our way towards this profession of the software archaeologist, and speculate on what their lives may be like. If you’d like to stay up to date:

I know I love it when people share things with me that they enjoyed and think I will too. So if you enjoyed this, please do consider forwarding & sharing it!

Kay