Version Hell Revisited

A surprisingly large proportion of the issues a working software engineer deals with on a daily basis turn out to be cultural. My theory is that this is because software ultimately amounts to people problems, and that it's all about coordination costs and human factors, as I've written about previously. This is not to say that we don't deal with the hard problems, but perhaps the "soft" in software hints at the lingering craft aspect even as we try to escape from Deadwood and leap into our industrial revolution.

This is one reason that I've found that, similarly, a large proportion of the best software engineers I've encountered in my now 25 years experience have not been traditional computer scientists. The linguists, historians and even that rare anthropologist, some of whom picked up the profession on the side, or out of expedience as the web has allowed, have been the real dark matter of technology. The web with its radical simplicity, has allowed mass amateurization to prevail. Now mind you, you do need your stereotypical straight ahead just-the-facts engineers with blinkered and relentless nerdy application, and, increasingly as software becomes more professionalized, the guilds and dark factory mills have started appearing. But hold that thought for now, I have another story to tell, indeed you may call it a technology folktale.

I came across this memo written a long, long time ago, in a far faraway land, (to be truthful, it was a Friday afternoon and I was fed up). Certain names have been changed to protect the guilty, and don't assume the anachronistic references mean that it is of recent vintage. Consider it perhaps a glimpse at the way the software sausage is made and we all know the scandal of slaughterhouses although we now call them meatpacking plants in polite society.

The context is some wrangling where at least 6 teams were pointing fingers at each other about who was responsible for uploading some tools that The Company was using. The flow of data through the system had at length been established, and it was now a coordination problem more than a design and architecture problem. The problem was that different teams were responsible for different components. The email chain had been playing out for months it seemed. I was stuck in the middle making arbitrary decisions mainly by virtue of having written a core piece of the system. I kept asking and everyone kept dodging my increasingly pointed questions. And so I wrote the following:

Previously in the same vein: Version Hell

---

The Memo

From: Koranteng@Toli
To: A long list of interested parties
Date: Friday afternoon, a long, long time ago
Subject: Toolchain versioning Re: provide version number Re: Signing Re: [snip]

I want to tease out the various strands because I'm finding it hard to serve the competing masters here.

Let me pitch it as a folktale, it's a pandemic and we all need comforting narratives.

Master Johnson, my director, would be paraphrased as follows:

Our current process is the moral equivalent of taking a USB stick dropped by our Vendor in our parking lot, and placing that software in our august data centers to sign our Crown Jewels. Now this may be what we're reduced to, but don't blindly use any old tools we are given, let's at least make sure that we track and bless known versions of this toolchain.

There were stronger words said at the outset back when we had our deep dive on our last integration project late last year.

Master Security Team pitched in, again paraphrasing...

Well, in this sorry business, we've normally done integrations of these tools for a given year. We're already overtaxed and busy and we burn weekends to get the integration working. But once we've got it working, we've got clean hands. We don't touch it anymore unless there's an error using them. We're not happy with this but ye olde signing tools ultimately come from the Ministry of Information team. We're middlemen here and the Signing and Software Release teams are the ones that integrate these tools into The Company's IT solutions.

Master Ministry of Information Team is also in a pickle. He receives vendor builds and has to produce release builds with the help of Butler Jenkins. These blasted tools are just one piece of the overall puzzle (you should see what else is on his plate). His main requirement is to be able to reliably sign and update this software for the lifetime of The Product, a decades long quest.

His junior sister, Miss Android Team actually works closely with Messers Qualcomm, Broadcom and Intel, and gets periodic updates from Vendor Google who changes signing requirements at a pace of its own choosing, those Mountain View people move at the speed of the web.

I'll omit the other players who are also similarly overtaxed, although I should make special mention of Mistress Software Release Team who actually owns the official release process and is always playing catch up integrating with Information and with Signing and other things I'm not even aware of.

Meanwhile I'm sitting like Ananse the Spider tending to my farm in the land of Signing and perhaps you can see the roots of my dilemma:

Each build could have a new version of the dismal signing tools.

Per Master Johnson, I can't blindly just sign with any old version of the Signing tools I am presented with. That man signs off on my annual review, so I cross him at my peril.

Per Mistress Software Release Team, I have a frozen API in place which doesn't specify the toolchain version. In any case, there is a long chain between Android, Information, Software Release to Signing, and the lead time to change things is measured in the cost of "Projects". We are booked for months, nay years ahead and agility is a problem here at The Company - The Company's initiatives on agility and the cloud native adoption notwithstanding.

Anyway my solution, as it is, is that I will support "blessed" versions of the toolchain. All that is left is for someone to upload these versions.

Now I'm stuck in the middle of all this, and so far it has been yours truly that is arbitrarily uploading versions of the toolchain based on what I've seen coming through in our test environment. I actually don't know what is a significant version and I don't really want to be in this loop. Further I am making decisions above my pay grade without any domain knowledge of these dismal signing tools.

The details matter and someone needs to own this and solve my riddle.

Again

• Who is it that uploads the "blessed" versions of the toolchain to Signing?
• More fundamentally, who decides what is a blessed version of the toolchain?

I'll end by invoking my friend Sam Ruby's Postulate

The accuracy of metadata is inversely proportional to the square of the distance between the data and the metadata.

I am clearly furthest from the data, so somebody closer needs to make the call. Perhaps I'm reading from the wrong playbook, but I'm merely trying to add value here.

Koranteng
--
Chief Toli Monger
The Company

[snip distressing long chain of emails going back 5 months of people passing the buck]

Postscript

I'd like to think that my late Friday email did have the clarifying effect I intended with its visions of Ruby's Postulate and The Tower of Babel. There were certainly calls for action, and immediate action at that. I learned that I had underestimated the extent of the dysfunction at The Company. There were actually at least 8 other teams that I might have added to my folktale, and they each had their own sorry origin story. The list of people inside The Company who I learned had read my missive was impressive and growing.

There were ruffled feathers however. People who should have dealt with the underlying organizational and communication issue were upset that I had revealed it so starkly. Lots of meetings were called. Action items were issued and so forth. I was advised to let others handle things at this stage. I was also reminded of John Kenneth Galbraith writing in The Great Crash of 1929.

... the rite of the meeting which is called not to do business but to do no business...

One of the oldest, most important - and unhappily, one of the least understood - rites in American life.

Men meet together for many reasons in the course of business. They need to instruct or persuade each other. They must agree on a course of action. They find thinking in public more productive or less painful than thinking in private. But there are at least as many reasons for meetings to transact no business. Meetings are held because men seek companionship or, at a minimum, wish to escape the tedium of solitary duties. They yearn for prestige which accrues to the man who presides over meetings, and this leads them to convoke assemblages over which they can preside.

Finally there is the meeting which is called not because there is business to be done, but because it is necessary to create the impression that business is being done. Such meetings are more than a substitute for action. They are widely regarded as action.

Everyone remembers the story of The Emperor's New Clothes, I should note however that Hans Christian Andersen didn't write the sequel about what happened to that little boy and his family six months later when The Authorities could finally deal with them.

Slightly related, the Gambian proverb goes "Words are like bullets, once you release them you can't call them back.

It is said that a prophet is never recognized in his own country and the verdict was still out on if I was to become The Company's John the Baptist. I won't tempt fate and leave it to the historians to make that determination. In closing, I'll again turn to Belloc

It is always a relief to believe what is pleasant, but it is more important to believe what is true.

— Hilaire Belloc

Office Tales

I'd previously pointed to this juxtaposition, a survival guide to life in any workplace

• Office life by Keith Waterhouse
• Office Politics by Wilfrid Sheed
• The Organization Man by William H Whyte, Jr
• Managing Humans by Rands in Repose
• The Bad Child's Book of Beasts by Hilaire Belloc
  Belloc's cautionary tales and bestiary of observed pathologies are essential for office life.

Soundtrack for this note

File under: software, technology, complexity, data, metadata, design, architecture, best practices, coordination, organization, social, work, office, observation, decisions, process, programming, versioning, humour, toli,

66 Ways to Franco

It was the kind of thing that you found yourself doing in the middle of the night, musing on an idle question born of two of your concerns: musical obsession and software anthropology. Counting the ways to Franco was an excursion into the realm of metadata, matters of syntax, and a contemplation of the hive mind of the web. The initial insomniac impulse was to create a playlist; your search, however, found a surprising 38 variants of the name in your library and you couldn't locate the song that had triggered your nocturnal foraging.

You remembered that at the beginning of the year you only had a couple of his albums in your collection but the music blogs of this world wide web had sprang to life and you'd steadily filled the gaps while reading them. In this golden age of music distribution, all one needs is a vague memory and an internet connection to fulfill one's aural titillation. Let's see, a cursory glance shows that the mp3 collection now stands at 25,494 songs, 186 GB, or 99 days of non-stop music - probably a third of which was acquired over the past year. Now you do still spend a lot on your musical vices, but you can imagine that it is highly unlikely that $8,500 (at iTunes or Amazon pricing) would have left your insubstantial wallet during this Great Recession. No, your collection grew by osmosis. Moving on...

Your count was akin to last year's discussion about the bewildering number of ways people mangle song and artist names. That led to the sight of those Top 100 ways to write Guns N' Roses – Knockin' on Heaven's Door. What a strange glimpse of the musical Tower of Babel we now have. Take a lexical curio like Guns N' Roses (call it typographical eccentricity), add a few apostrophes and you'll realize that the children of Mr Special Character and Mrs Structured Data are blessed ones. This is the minutiae that software people - and that unwashed sub-clan, the database denizens, have to deal with on a regular basis. Whole businesses have been founded on making sense of such messy data. Information retrieval is the general term of art, and metadata, well, metadata is the data about data. It's one of the hard problems in your line of work.

So what do we have then? Through a set of historical accidents over the past decade - notably the piecemeal standardization of ID tags in the mp3 file format, the mp3 file format itself, the rapid adoption of internet led by the web, improvements in storage technologies, the development of new portable music players and the decentralized mass digitization and distribution of music, we can behold the glorious results of a democratic exercise of mass data entry. During this time, millions of ordinary people took to their computers and ripped their cd collections. Millions more downloaded and shared this great social bounty. True, the more prudent laggards waited until there was commercial affirmation and legally sanctioned avenues for their digital music. Throughout, however, this mountain of music had to be labeled.

The inevitable curators came along fairly early on in this process - the online databases, GraceNote and then Freedb, to help automate things and identify the cd once you loaded it on the computer. Their altruism however didn't stem the tide of data entry (someone after all had to have entered it once and we all know how that goes). Offerings like MusicBrainz have emerged in recent years as repositories of high quality music metadata, ostensibly on a mission to bring accuracy and fingerprinting to digital music. The commercial services too now loom large as major distribution points, but they too license their listings from some of these databases, and it shows: errors everywhere. More puzzling is that the record companies haven't been of much help, they too don't pay attention to the details of the names of the artists they supposedly promote, nor indeed the titles of the songs. They, like iTunes and Amazon stores, aren't perfect at information hygiene. You see typos and plain wrong labeling of music. You don't have to take the word of an opinioniated metadata curmudgeon, the proof is in the existence of a vibrant ecosystem of tag editing software, free and even commercial. Imagine that, people pay for a product to help them label their music.

Given that even the big boys don't label their music accurately, it has been a true free-for-all and that's not even taking into account the trend of the past couple of years of digitizing (and sharing) all the lost vinyl. I'll only note that the amount of African music that is being rediscovered is frankly startling. The labour of love of those who find and clean dusty grooves, scan album covers, digitize and share their musical memories is a true surplus for society.

It turns out that all these musical curators and aggregators have only been partially successful. It really is a problem with the human factor. When you have humans doing data entry you'll have errors. When you have on order of millions doing data entry, you'll have large numbers of errors. These glitches appeal to me, truth be told. People label things to remember them and the patterns they use are worthy artifacts. The variants, I'll suggest, are emblematic of both folk memory and the mass creation of semi-structured data. If I could, I'd write an ode to the mp3 tag. In the meantime, I set about to count the ways to Franco.

Last.fm was my weapon of choice - a music recommendation system with attitude. For one, they have been gathering data from all and sundry for years now - they call it audio scrobbling. They gather data about what people listen to, crunch away, and make contextual recommendations. They deal with huge amounts of data and the attendant complexity. One early strategy to work around the inadequacies of ID tags in mp3s was to simply escape from their confines and allow users to tag artists, songs and albums on the website and watch the free-form folksonomie emerge. Web-savvy as they are they organized these musical objects each with its url and wiki page. People like to discuss songs, albums and artists. Few algorithms can handle the notion that Orchestra Baobab recorded two albums in 1975 under the Orchestre Bawobab moniker. You need a human intervention to account for this kind of peculiarity. And so they did. It's a simple application of collective wisdom: watch what users do and organize around it. They can even offer radio stations based on tags. At a certain point also, they tried a fingerprinting technique that would process their users' libraries to allow them to normalize the metadata associated with a piece of music. Taking this further, they can simply ask and allow people to correct spellings or suggest alternatives for artists that performed under different incarnations. Once a critical mass is reached you can automatically apply this feedback to tend to this garden of metadata. This is the business of web scale identifiers.

And so we come to Franco. Not the fascist Spanish dictator, whose despicable legacy is still paradoxically a touchstone for some American neocons. No. For most Africans, Franco is all you need to say to signify great, pulsing guitar-driven music, rumba, soukous, social commentary and old time good fun. All these are the elements of Franco. It's all the same good, liquid music and great memories of excursions on the dancefloor, a long career spanning four decades. It's been twenty years since François Luambo Makiadi passed away, time enough to count.

And so I counted:

There are at least 66 ways to get to Franco.

I must admit looking at the list gave me pause. How do people remember a musician? How do we remember a piece of music? Only about 9,000 users of last fm seem to listen to Franco and yet here we find world class variety. He appears in 66 different guises to the world if you exclude the 3 obvious mistaggings and misspellings. The labelers in a pool of 1.5 million Guns N' Roses listeners could only mangle that band's spelling 56 ways. What was so special about Franco, I wondered, as I looked at the list? The obvious answer is that much of his music is not available commercially in digital form. Instead, a lot of his listeners are typing up the records and cassettes.

Capitalization concerns surface immediately. Some people like ALL CAPS, others are lower case freaks. That is par for the course in a world where search engines basically ignore case. (Sidenote: looking at the top search trends of the year, it appears that users use lower case in search engines, few bother these days with "Britney Spears" when "britney spears" will do. In this age of mobile phones, it seems that the shift key is being used less). Sidenote: we won't digress onto Camel Case discussions at this stage.

Most people think of Franco as synonymous with the band he founded: OK Jazz. But how do people deal with the abbreviation? How does one spell OK? Is it rather "Ok" or "O.K."? Opinions are varied. The OK Jazz band was so named because they began as the house band in the OK Bar in Leopoldville (now Kinshasa). It should be simple then: Franco & OK Jazz say - assuming you go with no dots.

Matter of punctuation however open up a can of worms; we stray onto typographical concerns, with pronounced eccentricity in the choice of separators. For delimiters, we see dots, semi-colons, slashes, commas and hyphens.

People have different conventions for conjunctions: it's a case of ampersands (&) for the many and fully spelled out for the few.

We have the language issue, the band was named in French so we'd expect "et", but some labelers are English-speakers hence we get a few instances of "and", "with" and "featuring" showing up. Incidentally the folks in the forums at MusicBrainz will regale you with tales about the epic wars over the conventions for dealing with the word "featuring". Briefly, some people spell it out, others contract to "feat" or "feat." - with the punctuation, or even further to "ft.". Suffice to say that it was much like the egg-cracking debate in Gulliver's Travels.

Lest you think that punctuation doesn't matter, let me interject the following anecdote. Tony! Toni! Toné! were named as such in their debut album, "Who?". In later albums, they were called Tony Toni Toné - with the exclamation points removed. Which is the more accurate name for the band I ask? Could you spin a story from the missing exclamation points? Well I'll engage in mindless speculation on this typographical mystery - stay with me. It's obvious: they changed record label and lost the rights to their name (much like The Jackson 5 had to become The Jacksons when they left Motown). They were shrewd in their negotiations and the price they paid was the removal of the exclamation points. The transformation was from 3 ejaculations (those 3 exclamation marks) to a sedate sentence, perhaps indicating a newfound maturity. Truth be told, the music was better without the exclamation histrionics yet it clearly is the same band. For what it's worth, Last fm and Amazon all normalize the name without the exclamation point. Anyway, I won't pursue this tall tale further. Back to Franco and OK Jazz...

The OK Jazz band was an orchestra with a shifting cast - in Congo and much of Africa after the second world war, there was a great flowering of such orchestras as proving grounds and incubators - so you have some renderings going with Orchestre OK Jazz. Again native language comes into play, for the English it's orchestra instead of the French orchestre.

Then there are the nicknames and honorifics. Simply put, Africans love titles. OK Jazz acquired the "Tout Puissant" prefix (almighty, literally all powerful). Franco acquired the appellation, "Grand Maître" (Grandmaster). Add in the grammatical concerns and you expand the choices; is it "le tout puissant" or "son tout puissant", ergo is it "the almighty" or "his almighty"? Le Grand Maître Franco & le T.P.O.K. Jazz perhaps? Franco Luambo Makiadi & OK Jazz? Sometimes also, Franco's full name is spelled out as if to underlie the vastness of his catalog. And when doing this, the name order varies.

At its most extravagant you'd get something like Le Grand Maitre Franco Luambo Makiadi & Le Tout Puissant Orchestre OK Jazz.

Even then, would you contract to TPOK Jazz? Or T.P. OK Jazz? Or T.P. O.K. Jazz? The plot thickens.

With such a long career, there was inevitably a shifting of focus; on some albums it was the band that was the lead, on others it was Franco who took the spotlight, and at other times other members took center stage (Sam Mangwana, Vicky, Taby Ley Rochereau and so forth). The names varied accordingly. It was all Franco, and it was all good, if you don't mind my saying. Trust me, pick almost anything he recorded and you'll be a happy camper.

But to get to my point. The music I'd been looking for at the midnight hour was the following album:

Over his career, there were a number of occasions where Franco had to pay obeisance to his patron, that murderous dictatorial rogue, Mobutu, kleptocrat without equal. The music on those few albums were not the best that he produced in his illustrious career. True, the tracks were danceable but they weren't ecstatic as usual. Some have even detected elements of irony in some of the songs - subversive dog-whistles that undercut the dictator's purpose and propaganda. I imagine some Africanist historian writing the definitive study of this phenomenon, perhaps something titled Musical Resistance in Dictatorial Times in 20th Century Congo: Rumba as Social Subversion. Interestingly enough, as you can see, the word Franco doesn't appear on the billing of the album, it is just plain old Luambo Makiadi. Twenty five years after its release, I couldn't find Candidat Na Biso Mobutu when I searched my iTunes and Winamp libraries for Franco's music, and it figures: he didn't need the dictator's bloodstains attached to his musical name. Franco was a smart man, he knew all about branding. He is sorely missed.

File under: whimsy, metadata, data, music, Franco, naming, syntax, structure, standards, software, technology, observation, Africa, toli