Version Hell Revisited
A surprisingly large proportion of the issues a working software engineer deals with on a daily basis turn out to be cultural. My theory is that this is because software ultimately amounts to people problems, and that it's all about coordination costs and human factors, as I've written about previously. This is not to say that we don't deal with the hard problems, but perhaps the "soft" in software hints at the lingering craft aspect even as we try to escape from Deadwood and leap into our industrial revolution.
This is one reason that I've found that, similarly, a large proportion of the best software engineers I've encountered in my now 25 years experience have not been traditional computer scientists. The linguists, historians and even that rare anthropologist, some of whom picked up the profession on the side, or out of expedience as the web has allowed, have been the real dark matter of technology. The web with its radical simplicity, has allowed mass amateurization to prevail. Now mind you, you do need your stereotypical straight ahead just-the-facts engineers with blinkered and relentless nerdy application, and, increasingly as software becomes more professionalized, the guilds and dark factory mills have started appearing. But hold that thought for now, I have another story to tell, indeed you may call it a technology folktale.
I came across this memo written a long, long time ago, in a far faraway land, (to be truthful, it was a Friday afternoon and I was fed up). Certain names have been changed to protect the guilty, and don't assume the anachronistic references mean that it is of recent vintage. Consider it perhaps a glimpse at the way the software sausage is made and we all know the scandal of slaughterhouses although we now call them meatpacking plants in polite society.
The context is some wrangling where at least 6 teams were pointing fingers at each other about who was responsible for uploading some tools that The Company was using. The flow of data through the system had at length been established, and it was now a coordination problem more than a design and architecture problem. The problem was that different teams were responsible for different components. The email chain had been playing out for months it seemed. I was stuck in the middle making arbitrary decisions mainly by virtue of having written a core piece of the system. I kept asking and everyone kept dodging my increasingly pointed questions. And so I wrote the following:
Previously in the same vein: Version Hell
The Memo
From: Koranteng@Toli
To: A long list of interested parties
Date: Friday afternoon, a long, long time ago
Subject: Toolchain versioning Re: provide version number Re: Signing Re: [snip]
I want to tease out the various strands because I'm finding it hard to serve the competing masters here.
Let me pitch it as a folktale, it's a pandemic and we all need comforting narratives.
Master Johnson, my director, would be paraphrased as follows:
Our current process is the moral equivalent of taking a USB stick dropped by our Vendor in our parking lot, and placing that software in our august data centers to sign our Crown Jewels. Now this may be what we're reduced to, but don't blindly use any old tools we are given, let's at least make sure that we track and bless known versions of this toolchain.
There were stronger words said at the outset back when we had our deep dive on our last integration project late last year.
Master Security Team pitched in, again paraphrasing...
Well, in this sorry business, we've normally done integrations of these tools for a given year. We're already overtaxed and busy and we burn weekends to get the integration working. But once we've got it working, we've got clean hands. We don't touch it anymore unless there's an error using them. We're not happy with this but ye olde signing tools ultimately come from the Ministry of Information team. We're middlemen here and the Signing and Software Release teams are the ones that integrate these tools into The Company's IT solutions.
Master Ministry of Information Team is also in a pickle. He receives vendor builds and has to produce release builds with the help of Butler Jenkins. These blasted tools are just one piece of the overall puzzle (you should see what else is on his plate). His main requirement is to be able to reliably sign and update this software for the lifetime of The Product, a decades long quest.
His junior sister, Miss Android Team actually works closely with Messers Qualcomm, Broadcom and Intel, and gets periodic updates from Vendor Google who changes signing requirements at a pace of its own choosing, those Mountain View people move at the speed of the web.
I'll omit the other players who are also similarly overtaxed, although I should make special mention of Mistress Software Release Team who actually owns the official release process and is always playing catch up integrating with Information and with Signing and other things I'm not even aware of.
Meanwhile I'm sitting like Ananse the Spider tending to my farm in the land of Signing and perhaps you can see the roots of my dilemma:
Each build could have a new version of the dismal signing tools.
Per Master Johnson, I can't blindly just sign with any old version of the Signing tools I am presented with. That man signs off on my annual review, so I cross him at my peril.
Per Mistress Software Release Team, I have a frozen API in place which doesn't specify the toolchain version. In any case, there is a long chain between Android, Information, Software Release to Signing, and the lead time to change things is measured in the cost of "Projects". We are booked for months, nay years ahead and agility is a problem here at The Company - The Company's initiatives on agility and the cloud native adoption notwithstanding.
Anyway my solution, as it is, is that I will support "blessed" versions of the toolchain. All that is left is for someone to upload these versions.
Now I'm stuck in the middle of all this, and so far it has been yours truly that is arbitrarily uploading versions of the toolchain based on what I've seen coming through in our test environment. I actually don't know what is a significant version and I don't really want to be in this loop. Further I am making decisions above my pay grade without any domain knowledge of these dismal signing tools.
The details matter and someone needs to own this and solve my riddle.
Again- Who is it that uploads the "blessed" versions of the toolchain to Signing?
- More fundamentally, who decides what is a blessed version of the toolchain?
The accuracy of metadata is inversely proportional to the square of the distance between the data and the metadata.
I am clearly furthest from the data, so somebody closer needs to make the call. Perhaps I'm reading from the wrong playbook, but I'm merely trying to add value here.
Koranteng
--
Chief Toli Monger
The Company
[snip distressing long chain of emails going back 5 months of people passing the buck]
Postscript
I'd like to think that my late Friday email did have the clarifying effect I intended with its visions of Ruby's Postulate and The Tower of Babel. There were certainly calls for action, and immediate action at that. I learned that I had underestimated the extent of the dysfunction at The Company. There were actually at least 8 other teams that I might have added to my folktale, and they each had their own sorry origin story. The list of people inside The Company who I learned had read my missive was impressive and growing.
There were ruffled feathers however. People who should have dealt with the underlying organizational and communication issue were upset that I had revealed it so starkly. Lots of meetings were called. Action items were issued and so forth. I was advised to let others handle things at this stage. I was also reminded of John Kenneth Galbraith writing in The Great Crash of 1929.
... the rite of the meeting which is called not to do business but to do no business...
One of the oldest, most important - and unhappily, one of the least understood - rites in American life.
Men meet together for many reasons in the course of business. They need to instruct or persuade each other. They must agree on a course of action. They find thinking in public more productive or less painful than thinking in private. But there are at least as many reasons for meetings to transact no business. Meetings are held because men seek companionship or, at a minimum, wish to escape the tedium of solitary duties. They yearn for prestige which accrues to the man who presides over meetings, and this leads them to convoke assemblages over which they can preside.
Finally there is the meeting which is called not because there is business to be done, but because it is necessary to create the impression that business is being done. Such meetings are more than a substitute for action. They are widely regarded as action.
Everyone remembers the story of The Emperor's New Clothes, I should note however that Hans Christian Andersen didn't write the sequel about what happened to that little boy and his family six months later when The Authorities could finally deal with them.
Slightly related, the Gambian proverb goes "Words are like bullets, once you release them you can't call them back.
It is said that a prophet is never recognized in his own country and the verdict was still out on if I was to become The Company's John the Baptist. I won't tempt fate and leave it to the historians to make that determination. In closing, I'll again turn to Belloc
It is always a relief to believe what is pleasant, but it is more important to believe what is true.
— Hilaire Belloc
Office Tales
I'd previously pointed to this juxtaposition, a survival guide to life in any workplace
- Office life by Keith Waterhouse
- Office Politics by Wilfrid Sheed
- The Organization Man by William H Whyte, Jr
- Managing Humans by Rands in Repose
- The Bad Child's Book of Beasts by Hilaire Belloc
Belloc's cautionary tales and bestiary of observed pathologies are essential for office life.
Soundtrack for this note
File under: software, technology, complexity, data, metadata, design, architecture, best practices, coordination, organization, social, work, office, observation, decisions, process, programming, versioning, humour, toli,
