Showing posts with label best practices. Show all posts
Showing posts with label best practices. Show all posts

Tuesday, February 03, 2026

Category Mistake

I must say it was quite gracious
The reception to my latest mistake
Indulgent and polite, they were, my family gave me a break
Tolerant, that time, in the face of yet another faux pas

For that morning, I noticed that the oats weren't at all popular
Huh? Was it another case of the princess and the brown sugar?
I wondered quizzically what the subtle difference was in the flavor
Worried, perhaps, that my taste buds had been afflicted by the Corona

But it was a more mundane class of category error
A substitution - for the spice bottles looked so similar
Oops, instead of cinnamon I had picked up the paprika
Aha, that explained those curious flecks of red powder

...

A class of category error that I should really avoid
Stems from keeping my bottle of eye drops
And hand sanitizer in the same bag

When you're tired and your eyes are watery,
You reach for the small bottle, and you apply the drops and...
Well, you get the picture

...

A kettle is not an iron
- size, shape and steam to the contrary,
It is unwise to mistake the one for the other

#ModernProverbs

...

Error is a poor friend but an excellent teacher


Presentation Pete - Scared Pete (Office Life)

Category Errors, a playlist


Call it poetic licence that I branded a garden variety mistake as a category mistake in one of the examples above, truth be told there is a full spectrum of mistakes. In any case, a soundtrack for this note (spotify version) File under: , , , , , , , , , , ,

Writing log. October 26, 2022

Tuesday, August 12, 2025

Sensor Calibration

The problem is often couched as one of recognition
The difficulty, as it were, of sensor calibration
For if you can't measure a signal as a matter of first principle
If you can't detect accurately, you might as well be invisible

The auto-focus systems in cameras that can't detect those darker than blue
The pulse oximeters giving false hope - the all clear, to those with a darker hue
Training data, darker skin tones - unusual, light exposure
Biometrics, facial recognition, fingerprint sensors

Architectures of participation and control
Resigned to playing the tenuous outsider role
In this rigged game of life (and death) with ever changing rules
Arbitrary boundaries, when lines are drawn, borders can be cruel

For those excluded from the system, then, a matter of quiet advocacy
A liberation struggle of sorts, forever teaching others how to see
File under the banner of cultural sensitivity in technology
Sensor calibration and relief from the burden of invisibility


disassembled


Sensor Calibration, a playlist


A soundtrack for this note, fodder for sensitive souls (spotify version)

See previously Cultural Sensitivity in Technology and Empire State of Mind

File under: , , , , , , , , , , , , , , , , , ,

Writing log: September 21, 2022

Tuesday, February 27, 2024

Root Cause Analysis

Time for the post mortem, as you know, some take it seriously
The need to revisit, the urge to explain what happened previously
For an ounce of prevention is said to be worth a pound of cure
Let's get to the bottom of things and figure out the root cause

Oftentimes it's a simple mistake, a moment of inattention
Or sometimes it's really just idle exploration
One minute you wonder, what does this button do?
Then the thing happens that you can't recant
Oops, you realize you just shut down the power plant

Human fallibility tends to be the root cause

Ah right, power. It's quite fitting in this era of modernity
That one can't sing too highly of the virtues of Electricity
So essential, we almost always overlook this august substance
We only rue its wonder when confronted by its absence
And now we've lost power and everything must stop
Oops, lights out. In Ghana we call it dumsor

Power failures are prime candidates for the root cause

The next affliction, sadly, is all too common
Like ants, human beings just like to burrow
When in a mad rush to lay down some pipes, it's nigh inevitable
So busy that we never checked to see what could be an obstacle
Dig: bureaucracy got in the way, they were moving too slow
Oops, the contractor cut the critical cable with his backhoe

All too often, cable cuts tend to be the root cause

Things fall apart, they say,
   equipment sputters, machines fail
They blow hot and cold, or crack when used,
   there's wear and tear
Material scientists make a roaring trade
   as do structural engineers
That, sadly, alchemists never overcame nature's challenge
   is the lesson learned
Oops, the widget broke,
   a reminder that no condition is permanent

In this industrial age, hardware failures are a likely root cause

Sometimes you're just too popular,
   so crowded no one can get in or breathe
Congestion is the operative word,
   in matters of scale, a crowd changes things
Your service is the flavor of the month,
   and now you've become essential
Oops, you're completely unprepared for when you go viral

Woe is me, lack of capacity is frequently the root cause

And then we come to the bad actors,
   forever on the attack
Always probing for an opening,
   for vulnerabilities in your stack
And that's even before we consider
   the gremlins and parasites
Iconoclastic beasts with distinctive manners
   and singular appetites
Every complex ecosystem in history
   has had to deal with grifters
Oops, your hospital is held to ransom
   by a band of sneaky hackers

Always protect yourself, a lapse in security is invariably the root cause

There's more in this vein,
   mankind has never built a system without error
From the Tower of Babel to that fancy car,
   or even that blasted word processor
The raw materials of life,
   whether it's the design or the initial conception
Imposing one's will,
   it might be a flaw in the ultimate implementation

You probably have your own experience and area of expertise
Your own rules of thumb about these puzzling mysteries
Let me tell you something
   from my profession of software engineer
If you only knew,
   to defend a system in depth is an exercise in fear
How close we come to catastrophe,
   partial or complete, every day
Trust me, you really don't want to see how the sausage is made

Now one could argue about the order
   of this short list of failure modes
It is only in retrospect that one is truly able to diagnose
The human burden is to keep moving
   in the face of systemic error
To mitigate the worst,
   to build the fail-safes and systematic procedures

Spare a thought for the moron in a hurry,
   for one day it could be you
That, through omission or commision,
   will be blamed for the miscue
We haven't scratched the surface
   of how much the human factor has an impact
Fall back to folk wisdom,
   suffice to say that curiosity killed the cat

And what of the wider world,
   say failed love affairs, or even wars?
It's only human to search for simple answers
   and the root cause
Our prophets and philosophers have long emphasized
   moral suasion and the golden rule
You could do hardly do worse than social living
   and the mosquito principle

Focus on best practices, usability, and layers of protection
Try to put a process in place and make it official!
Make sure that it takes many, many big red buttons
   to launch that nuclear missile
If there's any moral to this tall tale of root cause analysis
Take heed, and wherever possible, make use of the checklists


wide load coming through


After
  • How Complex Systems Fail (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety) by Richard I. Cook
  • a quip about network outages from Sean Donelan

Root Cause, a playlist


A soundtrack for this note (spotify version)
See also: The Dining Philosophers Problem, Resilience and Adaptability, and Version Hell Revisited

This belated entry on failure modes is part of the Toli Technology Series

File under: , , , , , , , , , , , , , , , , , , ,

Writing log: April 3, 2022

Tuesday, February 22, 2022

The Least of Us

For most of history, it has been left to sovereign nations
To raise concern about the plight of refugees
But it was Matthew, from his gospel writer's station,
Who best reported on how to treat the least of these

The laments of note throughout history have been of the great migrations
The tales of exodus, grit, and survival that prevail in our mythmaking
Think back, if you will, to the great origin stories that take pride of place
Much less, however, is made about the fate of the internally displaced

This semantic framing glosses over the very real trauma for those others
How can it be left to insurance actuaries and international lawyers
To assess the level of pain of a wound, even psychic, in human beings?
A temporary inconvenience is their prized, arbitrary reading of things

For every action, there is an equal and opposite reaction
Goes the third proposition of Newtonian physics
On the human scale, and in the realm of human affairs,
The notion is couched as blowback that damns the spirit

And the counter effects are known to resound with appalling severity
That hurt fosters revenge, is the brutal lesson of cultural memory
In the good book it was written that justice is in the eye of the beholder
The ellipses will be filled, history's silent treatment is a mere placeholder

A word to the wise, discard social living at your peril
The mosquito principle teaches that man can be feral
Buyer beware, take heed of the treatment meted out to the least of us
First they came for others, but you may well be the next one deemed treasonous

There is no worse feeling than alienation,
   it leaves the most bitter residue
To walk among people who actively celebrate the hurt inflicted on you
Wrath and ignorance, the mob brings out the worst aspects of our culture
As goes the proverb, try not to die when you are surrounded by vultures

A marshmallow test of sorts is posed in humanity's curriculum
About the interplay in society to gain equilibrium
Whether to exercise restraint or give in to insatiable appetites
The message from your leader:
   a duty of care, tolerance, and equal rights

equality rosewood park

...

No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend's or of thine own were: any man's death diminishes me, because I am involved in mankind, and therefore never send to know for whom the bells tolls; it tolls for thee.

— Devotions upon Emergent Occasions by John Donne

The Least of Us, a playlist


A soundtrack for the least of us. (spotify version) I'm conflicted about how to place this internal displacement, originally conceived for a covidious series as it could well fall under the banner of Social Living, do let me know what works better.

File under: , , , , , , , , , , , , , ,

Writing log: March 6, 2021

Tuesday, November 30, 2021

The Dining Philosophers Problem

The dining philosophers problem is well known in computer science
Dealing with matters of access, it has analogues in our headlines
Humanity's curriculum involves many rituals and traditions
Perhaps, then, it's worth explaining the problem's formulation

The diners that are gathered here today are five in number
Imagine, if you will, your favorite group of philosophers
They are initially presented in a state of silence at the venue
Around a round table, hungry they are, for spaghetti is on the menu

The central conceit is that each philosopher must alternately think and eat
Implicit also is that they are vegetarians or vegans, for there is no meat
The rules of the social game adds the constraint of manners as the main course
That they can only dig into their spaghetti if they have both left and right forks

After a philosopher finishes eating, but before launching into thought,
They must put down both forks to make them available for others.
Forks, in this arrangement, can only be held by one philosopher at a time.
The problem's construction teaches that sharing is caring at mealtime

To add realism to the scenario, we can assume there is an unlimited supply of food
And that our philosophers can eat long past the point that would normally be good
With these core ingredients: philosophers, spaghetti, forks (but no knives)
The problem is meant to illustrate the challenges of living parallel lives

An utter conundrum, this problem when it was presented to me at university
That, while I could discern that it had applications in the study of concurrency,
The problem was going from this set of instructions to designing an architecture
Full disclosure, I damn near failed this course on algorithms and data structures

The enduring challenge, if you think about it, is how to avoid deadlock
The fear is of resource starvation such that the actors find themselves stuck
We want a world in which no philosopher will starve and continue forever
To eat and think deep thoughts; a metaphor, this immortal philosopher

The Last Philosophers

Timing is everything in the choreography of actors
Finding a strategy to manage this human infrastructure
The fundamental issue at hand, the term of art, is mutual exclusion
A stark concept, therein lies the difficulty with the algorithm

The conventional solution involves a resource hierarchy
I found it quite unfair because it clearly lacks neutrality
But it can be proven to work, if one heeds the instructions and laws
It applies an ordering to the distribution of shared resources

Another strategy that has been advanced is arbitration
This one I tried out, as I'm a firm believer in conversation
Once you designate a mediator, and open yourself up to abide by their resolution,
As the song goes, it takes behavior to get along, diplomacy can be the solution

The philosophical issues are many with middlemen, think of Caesar's tax collectors
And of the oligarchs and monopolists who insert themselves as toll collectors
Who are the designers of these systems in humanity's curriculum?
Who gets to make the rules? And who pays soul insurance premiums?

Game theory can come into play, think of the prisoner's dilemma
The shortest path to success might be to seize resources - state capture
Write the ruler's rules, and steal power like a conqueror
Rehearse a coup drill following the lead of many a dictator

I found that the problem had all the makings of a cautionary fable
Earlier, you might recall, I held my fire at that philosopher's table
Where rage, and its contemporary uses, were the bone of contention
'Twas decidedly, I'd explained, a case of normalcy prohibition

I had been thinking about this modernity of parallel lives
Where some have the great privilege of their insatiable appetites
A state of the world in which no progress is possible
We would do well to study the Mosquito Principle

equality rosewood park

See previously:

The Dining Philosophers Problem, a playlist


A soundtrack for this treatise, bookended by the great divas (spotify version)
...

Timing is everything
Observers are worried

File under: , , , , , , , , , , , , , , ,

Writing log: March 28, 2021

Saturday, April 03, 2021

Empire State of Mind

A memory. An open day at the Electrical Engineering department at Imperial College for accepted candidates, the professors wanted to demonstrate their new face recognition software to the group. I demurred (shyness), but they insisted that I be the volunteer.

So I sat and watched for 15 minutes as nothing worked. "Smile", asked the graduate students. "Frown", asked the professors. Same result: no detection. Others were summoned. Invisible man.

It's a minor point, but I can't forget the sheepish grins. As I told the professors and their harried students: "You probably didn't train your algorithms much with many faces like mine."

Cultural sensitivity in technology is one of my perennial themes. And it's hard work even if you acknowledge your blind spots. The anecdotal failures continue to pile up. What was true back in 1991 is much the same in 2021, software and hardware are far more sophisticated and performant but face the same blindspots (provenance of training data, applicability to real world scenarios, ethical framing etc.)

The challenge for software engineering - which is still a craft, is to move beyond curve-fitting phrenology (and Deadwood) into its industrial revolution.

Sidenote: recruiters these days are all "big data, machine learning, cloud yada yada". Buzzword fatigue is an occupational hazard.

I miss the great mass amateurization and view source ethic of early web development and yet the developer tools and frameworks these days almost feel like a golden age is within grasp.

Obligatory citations:

Incidentally, Imperial College was the venue of my worst interviewing experience (and there have been many) - the low point of which was walking past the open door as I left the interview, and walking straight into the wall. This was after having flubbed almost every question I'd been asked.

The laughs and looks exchanged by the professor and the secretary as I turned around, shuffled back muttering an apology (why?) as I rubbed my sore head and headed out the door. Even English reserve and politeness could not deal with my Buster Keaton imitation.

I was admitted to Imperial College a week later.

I occasionally regret not having gone to Imperial or Cambridge for university. Without a doubt, I would be a stronger engineer and yet I suspect the eclectic toli monger you see before you would be repressed.

Guide to Lagos 1975 005 3m 191 revolutionary  copier

Imperial Visions, a playlist


A soundtrack for this note (spotify version). I've been neglecting the Toli Technology Series for years now, albeit I occasionally make a few gnomic pronouncements on Twitter, consider this some throat clearing to prompt a reboot.

File under: , , , , , , , , , ,

Tuesday, February 02, 2021

The Skeptic's Credo

If you've ever read Goethe's theory of color, you'll know that there are often ideas that are beautiful, ingenious, and indubitably wrong.

Neither the scaffolding of an idea, the trimmings around a thought, nor its progenitor or credentialed bearer should disguise its essential worth even though any of the foregoing often do, indeed, mask the underlying truth. The bogus wear many disguises.

Both the most simple idea and the most arch concept can be praiseworthy, and fodder for much food for thought and insight, and yet both, the nigh-judious liminal thought and the complex, counterintuitive koan, can be completely and thoroughly mistaken.

We rightly treasure the insight of the boy who remarked that the Emperor "hasn't got anything on", injecting a dose of reality, but understand, all too well, the process that made the gathered multitude ignore the evidence of nude hubris on display in the fable. We live in our own fables.

Incidentally, I should note that Hans Christian Andersen didn't write the sequel about what happened to that little boy and his family six months later when The Authorities could finally deal with them after the wardrobe malfunction. Suffice to say that it wasn't pretty. My idle speculation is that he tipped off his contemporary, Dostoevsky, with the germ of Crime and Punishment - there's a PhD, or alternatively a Hollywood script, in fleshing out that crossover concept.

Suffice to say that the most brilliant of thinkers can turn into cranks of the grandest order when they step out of their lane. The shape of an idea may transcend borders figuratively but its core rarely strays far from home.

This is not to demean the polymath, or to prefer Isaiah Berlin's hedgehog to the roving fox, but merely an observation that critical thinking is always in order. The point is that delusions are not the province of the uneducated or uninformed. All ideas must thus be taken with a grain of salt and shown to relate to reality. If conception and perception must be married, let it be by wisdom.

The Last Philosophers

The skeptic's credo is one of expectant ambivalence infused with an element of pessimism. Always mark your beliefs to market, you almost always hear the skeptic muttering, and with a touch of weary righteousness. The weariness comes from disappointment - an occupational hazard of skepticism, and the righteousness from the fraught history of one's confounding beliefs being proved right.

Scientists often claim to follow the skeptic's method, indeed their august Method - the one that they exhalt in their droning cocktail hour conversation, lends itself to the trappings of authority. The callouts to verifiability, testability and various isms are often worn as badges of invincibility even though they are mere cloaks of fallibility. Reason and inquiry, couched in specific forms lauded by the academy, are attributed the virtues of rigor and given free entry into society's discourse.

One need not go to Issac Newton, who would have sold you on his fever dream of the South Sea bubble, to pursue this point. Charles Darwin and the Brontë sisters would have talked you into spending your fortune on the Railway Manias of the mid 1800s. Even Charles Mackay, the man who wrote the book on Extraordinary Popular Delusions and the Madness of Crowds, himself fell prone to the similar mania. Gullibility is immortal. And the less said of the later, high-minded Victorian dabblers in phrenology and mesmerism - their parlor room séances speak for themselves, and were lampooned in real time. You've no doubt read their books. In our own time, the script may have changed but the servants of empire and of capital are still spouting their dismal wages.

As an engineer, albeit with a literary bent, I am well versed in our own idées fixes and can recite the STEM mantras as eloquently as others. For engineers too bring their blindspots to the table, substituting their programmatic interventions and sundry heuristics for intuitive common sense. The truth is, that peacock of our tribe, the self anointed tech visionary, is often just another blowhard, and should be judged (and ridiculed) accordingly.

The good books would guide us towards skepticism, and the oracles of all religions weigh in on credulity and inveigh against uncritical thinking and behavior. For Muhammed, the saying was "Trust in God but always tie up your camel at night". His earlier counterpart, the man from Nazareth, was a searching critic of authority on this earthly plane, and perhaps the fiercest advocate for performing due diligence in one's life. He would enjoin us to prepare for the kingdom of God and urge us towards the house on the rock (sidenote: perhaps the history of Christianity's spread betrays an inclination for mergers and acquistion, but I digress).

A difficulty with the parable of the wise and foolish builders is that it is all well for Jesus to advocate so incisively for building on solid foundations, but what if the home inspector assured you that the staged house you've just viewed, the one that was actually built on sand, was the real rock solid deal? In human affairs, who really has time for due diligence? And with Murdoch's real estate agent regaling you on the charming features of the McMansion, with Greenspan's Put Banker promising low interest rates come what may, the pressure is on to seal the deal, to stretch one's budget, and take on the liar loan. And this is not academic pedantry, the poor folks who lived in the Grenfell Tower had no reason not to trust the efficacy of the cladding, of all things, that wrapped what became their deathtrap. It is cold comfort to tell their ghosts that a house is not a home.

aryeetey on the line 1998

We are trusting simians not too far removed from the savanna and modernity makes us mostly morons in a hurry. It requires a lot of effort to be vigilant about the many Potemkins we face in life. We resort to heuristics and rules of thumb, and can be taking in by confidence artists of all sorts in the shell game of life. The gremlins and parasites of society prey on us and can easily turn the brightest of us into useful idiots.

If skepticism is the notion that that true knowledge is always uncertain, it simply imposes a frame on its bearer of always questionning, and this can be a wearying approach to life.

The early skeptic of most nostrums (capitalism, what have you) runs the risk of being branded as the designated driver: at once necessary for soul insurance and sober hindsight, but, frankly, buzzkill during humanity's weekend in Vegas. Aficionados of doubt are rarely celebrated by History, there is no cult of doubting Thomas.

Pyrrhonism, the total skepticism of yore, didn't have many adherents because its doctrine of radical skepticism proved immediately unpopular. While it was attempted as an all-encompassing philosophy of life, it has never prospered. It seems that some measure of faith, and, possibly, a considerable amount of that ineffable substance, is required in all human institutions that reproduce themeselves successfully.

It it the plight of the skeptic to forever inhabit the terrain of uncertainty, to vacillate in those borderlands of fate always on the verge of temptation by seductive manias.

The skeptical genes are obviously useful for humanity in our decision making but, paradoxically, they are not mandated, nor necessary, for our species's survival; they are certainly not under selective pressure to be chosen like a peacock's wing or our finely attuned eyesight. Accordingly, we see a wide spectrum of trust cultures around the world. In an era of ease or nostalgia, our propensity for skepticism may even fall prone to atrophy. Walter Bagehot, in Lombard Street, would remark that "All people are most credulous when they are most happy".

How then to hold on to one's skepticism when a bubble is in full flow when, as Keynes noted, "the market can remain irrational longer than you can remain insolvent". My own favorite reading reading comes from Andrew Odlyzko who offers a rich library on bubbles, gullibility and manias. Forewarned is forearmed. Dan Davies also gives insight on cultivating the skeptical inclination or, as he put it, Avoiding Projects Pursued By Morons 101. The selling of the second Iraq war remains an important case study. I fall back to that critical recommendation he makes about the vital importance of audit.

More classically perhaps, I harken to Diderot:
A thing is not proved just because no one has ever questioned it. What has never been gone into impartially has never been properly gone into. Hence skepticism is the first step toward truth. It must be applied generally, because it it the touchstone.
The aphorists of yore prescribed a healthy dose of skepticism, and for good reason, but in their infinite wisdom, gave no guidance on the scale in question. It is lost in the veil of time the names of those who calibrated their formulae let alone their methodology. Further, their measuring sticks didn't have to contend with our fraught modernity. All too often, we only realize the appropriate measure of skepticism with hindsight and too late for optimal decision making. Fierce competitors they may be, I fear that the skeptics are doomed to run the race with a handicap. Ultimately, the tales the skeptics weave revolve around their conceptions of self, and they often live with the letdowns proffered by society and History. And so I remain a student of dissimulation and the strange architecture of misdirection. My fear is that the game is rigged, that we are all marks being cooled off. And the band played on.

To return to Goethe's notion referenced at the outset, pleasing as it may be to believe the "colors are the deeds and sufferings of light", I wouldn't make operational decisions based on that poetic insight. I have to hold fast to my inner skeptic and return to Hilaire Belloc on wishful thinking to give fortitude:
It is always a relief to believe what is pleasant, but it is more important to believe what is true.


The leakage specialist's 100% herbal solution

This note is part of a series on Shell Games. See previously: Shame Cultures

Next: A Taxonomy of Useful Idiots

File under: , , , , , , , , , , , , ,

Saturday, July 11, 2020

Top Public Health Interventions

A list of the top public health interventions in history...

  • Sewer pipes (Mesopotamia 4000 BC)
  • Soap (for washing hands circa 2800 BC)
  • Emperor Shennong of China mandating drinking boiled tea (2737 BC)
  • Mosquito nets (from Cleopatra's time through present)
  • Vaccination (Jenner 1796, Pasteur 1880)
  • Pasteurization (Pasteur 1864)
  • Antibiotics (pace Fleming's penicillin 1928 onward)
  • Existence of unions (1827 onward)
  • Child labor laws (starting with the Cotton Factories Regulation Act of 1819 in England to present)
  • Birth control pills (Djerassi 1951, and Pincus and Rock producing Enovid, the first birth control pill approved in 1957)
  • Tobacco control (1952 to present, from the 1964 US Surgeon General report through the Tobacco Control Act of 2009)
  • Seat belts (1955 to present, the crucial milestone was 1966 with the National Traffic and Motor Vehicle Safety Act in the USA)
  • Occupational safety regulations (kicked into high gear in the US with Occupational Safety and Health Act of 1970)
  • Water fluoridation (widespread after 1945)
  • Iodized salt (1924 onward)
  • Ban on use of lead in gasoline (regulation starting 1973 - banned in US with Clean Air act in 1996)
  • Antiretrovirals, especially in African countries to manage HIV (1996 onward)
  • Board certification for doctors (1917 onward)
  • FDA approval for food and medicine (onset with Food and Drug Act in USA 1906, effective after 1938's Food, Drug, and Cosmetic Act )
  • Helmets (although a 1600 BC invention, their public health application came in sports after 1896 for American football and transportation circa 1914 with the motorcycle helmet)
  • Gun control (in most countries almost immediately after its invention circa 1000 AD with Yemen and the USA as modern day outliers)
  • Alcohol control (an ongoing and uneasy spectrum of regulations even to the extent of prohibition with religion even weighing in)
  • Oral rehydration therapy (1940s, popularized with ORS sachets especially after the refugees successfully treated during cholera epidemic of cholera after the Bangladeshi Liberation War)
  • Facial masks and shields (briefly after the influenza pandemic of 1918, in East Asia after the SARS pandemic of 2003, in China, Taiwan, South Korea, Vietnam, Mongolia, Japan and the Czech Republic memorably immediately after the start of the novel coronavirus SARS-CoV-2 pandemic January-April 2020, and the rest of the world circa March - June 2020, with notable American exceptionalism - in most enlightened states in the US. The N95 respiratory mask, conceived in 1992, remains a milestone although cloth masks have long had their uses. Whither my Texas neighbors you may ask? The neighbor who was overheard just a month ago, after The Grand Reopening of Texas, shouting "There is no virus" remains under the relentless grip of the Positivity of The Governors even after their July 2020 pivot. My best guess is that it will take 9 months, or perhaps personal experience, for the aversion to evidence and public health to penetrate consciousness. On the other hand, norms can change quickly, as evidenced in other countries, but I won't hold out hope for the year 2020 in the USA at large, I think it's a Vision 2021 project despite the rising body count)

This became a collaborative list with crucial additions from friends and The Wife, whose pointed comment about the birth control pill prompted me to add It's a Man's Man's Man's World by James Brown to the playlist I was conjuring for this list. Indeed oral contraceptives have probably positively affected the greatest numbers of humans lives outside of soap and sewers.

A twitter follower also challenged me that it was Emperor Shennong's courtiers, not the Emperor himself, who mandated boiling tea - that the Emperor just preferred his tea boiled. This again speaks to the Princpal-Agent relationship, and posits that there is a difference between the wishes of the Emperor and those surrounding him. The suggestion is that "Who will boil me this tea?" is the antecedent to "Who will rid me of this meddlesome priest?". I dodged the question as is my wont because, in any case, it was fortuitous that boiling water, with its side effect of killing germs and other beastly organisms, became an integral part of humanity's public health arsenal.

There was some controversy about my inclusion of child labor and the existence of unions, enough that I was even moved to play Devil's advocate

Think about it, would we have Dickens without child labour? What paradise have we lost when the youth of Bangladesh, or Ghana are no longer doing shifts in the textile factory, or planting yams and pineapples in the hills of Aburi? Our modern day Oliver Twists and Little Annie's no longer have the hard knock life, coddled as they are with this modernity, they have school not farms, and they are constantly demanding fondleslabs of mobile entertainment.

Of course, it was facetious to posit a world without Dickens, there was more than enough outrage in 19th century England and the world for Charles Dickens to get novelistic material. The little street urchins and cannon fodder for the worst excesses of capitalism might make the headlines but the the vast majority of capitalism's operation is a mundane grind, a rigged shell game on a tilted playing field - pick your metaphor. Call them the underlying conditions and risk factors that favour capital over labour. The notion of "the weekend", far from being divinely prescribed as days of rest, was in actuality a hard fought victory of labor and trade unions, as were occupational safety regulations. Social welfare and the advances in communal living have had considerable impact beyond the economic sphere and have public health effects.

Looking over the list, the human species has gotten away with much faulty biology and superstition. I suppose that belief in unseen things is what makes the Gospel of Germs such a hard sell that it requires constant marketing and reinforcement. Hilaire Belloc would satirize late 19th century popular conceptions of The Microbe in The Bad Child's Book of Beasts and More Beasts (For Worse Children). Things are not much changed in 2020 than in 1897.

The Microbe by Hilaire Belloc

The Microbe is so very small
You cannot make him out at all,
But many sanguine people hope
To see him through a microscope.
His jointed tongue that lies beneath
A hundred curious rows of teeth;
His seven tufted tails with lots
Of lovely pink and purple spots
.
The Microbe by Hilaire Belloc 2

On each of which a pattern stands,
Composed of forty separate bands;
His eyebrows of a tender green;
All these have never yet been seen -
But Scientist, who ought to know,
Assure us that they must be so...
Oh! let us never, never doubt
What nobody is sure about.

It is interesting that the tiny clumps of viral RNA that are driving our current covidious predicament fit much the same description.

the novel coronavirus SARS-CoV-2

The inverse list is also interesting, and we have a great candidate in Thomas Midgley for the most catastrophic intervention. Midgley invented both leaded petrol, and the CFCs (chlorinated fluorocarbon) used in refrigerators. The collateral damage, and body count, of these two inventions are world historic. Still, knowledge about their effects on the environment was not well known in his lifetime, unlike the inventors of gunpowder or the nuclear bomb. Midgley didn't have an Alfred Nobel come-to-Jesus moment and continued tinkering until his death.

I think we have real time candidates for the negative list either with neglect or by wilful policy decision making. It's slightly morbid to follow the implications of that thought because it gets to valuing abstraction over of flesh and blood. What is the value of a human life? And who gets to coldly decide that on the golf course or hastily fortified bunker?

Back to Basics


Covid-19 is a funny disease as my doctor uncles and aunts put it. It's forcing humanity to go back to basics and causing us all to change our way of life. If you take a look at the coronavirus superspreading timeline, you quickly get a sense of the activities that are now to be avoided, they involve crowds, and places with poor ventilation and sanitation, and where we sing or chant. We basically shouldn't touch or breathe on each other. And these are hard lessons to teach and follow, as I've found even after months of self isolation. The 7 and 9 year olds in my household are not the only ones that need constant reminders about the best practices for survival, frequently washing hands is not obvious despite our best efforts to make it a habit. It's hard to be a good neighbor when you can't do small talk and even conversations over the fence are fraught and wary episodes.

All communal gatherings have the potential for risky distribution of disease even as they might give comfort and social and economic utility. Paradoxically, we need to draw on social living for comfort and resilience, yet in our present, it is social distancing and enforced absence and the foregoing of our rituals and traditions that is our best chance of survival.

We are all in this together and it pays to learn the lessons of The Mosquito Principle, for these clumps of viral RNA, like our close and longstanding companions, mosquitos, don't discriminate. While we wait for a vaccine or effective treatment for Covid-19, our mantra has got to be: stay home if at all possible, maintain social distancing, and wash hands with soap frequently, wear masks when outside and perhaps even indoors when there is poor ventilation or if in the company of others beyond one's core living unit.

Optional: pray.

For faith healing is always an option; epidemiologists will forever be fighting a rearguard action against the placebo effect of faith. The other alternative is the very human hope that you will be saved, a deus ex machina is always a possibility - vaccines do get developed. Indeed this wishful thinking is firmly embedded in American mythology and culture.

And yesterday was the day of our cinema heroes
Riding to the rescue at the last possible moment
The day of the man in the white hat or the man on the white horse
Or, the man who always came to save America at the last moment

Someone always came to save America at the last moment
Especially in B-movies

B-movie by Gil Scott-Heron

In lieu of herd immunity, researchers at the Center for Disease Control (CDC), or perhaps the Surgeon General, would do well to issue the following public warning:

Nostalgia can be a fatal disease.

Observers are worried

chief zaachi physical and spiritual center

A Public Health Playlist


As always, a soundtrack for this note. (spotify version)

Suffice to say that I prefer public health interventions to faith healing, as always there is the fine print: your mileage may vary

File under: , , , , , , , , , , , , , , , , , ,

Wednesday, July 01, 2020

Version Hell Revisited

A surprisingly large proportion of the issues a working software engineer deals with on a daily basis turn out to be cultural. My theory is that this is because software ultimately amounts to people problems, and that it's all about coordination costs and human factors, as I've written about previously. This is not to say that we don't deal with the hard problems, but perhaps the "soft" in software hints at the lingering craft aspect even as we try to escape from Deadwood and leap into our industrial revolution.

This is one reason that I've found that, similarly, a large proportion of the best software engineers I've encountered in my now 25 years experience have not been traditional computer scientists. The linguists, historians and even that rare anthropologist, some of whom picked up the profession on the side, or out of expedience as the web has allowed, have been the real dark matter of technology. The web with its radical simplicity, has allowed mass amateurization to prevail. Now mind you, you do need your stereotypical straight ahead just-the-facts engineers with blinkered and relentless nerdy application, and, increasingly as software becomes more professionalized, the guilds and dark factory mills have started appearing. But hold that thought for now, I have another story to tell, indeed you may call it a technology folktale.

Tower of Babel

I came across this memo written a long, long time ago, in a far faraway land, (to be truthful, it was a Friday afternoon and I was fed up). Certain names have been changed to protect the guilty, and don't assume the anachronistic references mean that it is of recent vintage. Consider it perhaps a glimpse at the way the software sausage is made and we all know the scandal of slaughterhouses although we now call them meatpacking plants in polite society.

The context is some wrangling where at least 6 teams were pointing fingers at each other about who was responsible for uploading some tools that The Company was using. The flow of data through the system had at length been established, and it was now a coordination problem more than a design and architecture problem. The problem was that different teams were responsible for different components. The email chain had been playing out for months it seemed. I was stuck in the middle making arbitrary decisions mainly by virtue of having written a core piece of the system. I kept asking and everyone kept dodging my increasingly pointed questions. And so I wrote the following:

Previously in the same vein: Version Hell


The Memo


From: Koranteng@Toli
To: A long list of interested parties
Date: Friday afternoon, a long, long time ago
Subject: Toolchain versioning Re: provide version number Re: Signing Re: [snip]

I want to tease out the various strands because I'm finding it hard to serve the competing masters here.

Let me pitch it as a folktale, it's a pandemic and we all need comforting narratives.

Master Johnson, my director, would be paraphrased as follows:

Our current process is the moral equivalent of taking a USB stick dropped by our Vendor in our parking lot, and placing that software in our august data centers to sign our Crown Jewels. Now this may be what we're reduced to, but don't blindly use any old tools we are given, let's at least make sure that we track and bless known versions of this toolchain.

There were stronger words said at the outset back when we had our deep dive on our last integration project late last year.

Master Security Team pitched in, again paraphrasing...

Well, in this sorry business, we've normally done integrations of these tools for a given year. We're already overtaxed and busy and we burn weekends to get the integration working. But once we've got it working, we've got clean hands. We don't touch it anymore unless there's an error using them. We're not happy with this but ye olde signing tools ultimately come from the Ministry of Information team. We're middlemen here and the Signing and Software Release teams are the ones that integrate these tools into The Company's IT solutions.

Master Ministry of Information Team is also in a pickle. He receives vendor builds and has to produce release builds with the help of Butler Jenkins. These blasted tools are just one piece of the overall puzzle (you should see what else is on his plate). His main requirement is to be able to reliably sign and update this software for the lifetime of The Product, a decades long quest.

His junior sister, Miss Android Team actually works closely with Messers Qualcomm, Broadcom and Intel, and gets periodic updates from Vendor Google who changes signing requirements at a pace of its own choosing, those Mountain View people move at the speed of the web.

I'll omit the other players who are also similarly overtaxed, although I should make special mention of Mistress Software Release Team who actually owns the official release process and is always playing catch up integrating with Information and with Signing and other things I'm not even aware of.

Tower of Babel

Meanwhile I'm sitting like Ananse the Spider tending to my farm in the land of Signing and perhaps you can see the roots of my dilemma:

Each build could have a new version of the dismal signing tools.

Per Master Johnson, I can't blindly just sign with any old version of the Signing tools I am presented with. That man signs off on my annual review, so I cross him at my peril.

Per Mistress Software Release Team, I have a frozen API in place which doesn't specify the toolchain version. In any case, there is a long chain between Android, Information, Software Release to Signing, and the lead time to change things is measured in the cost of "Projects". We are booked for months, nay years ahead and agility is a problem here at The Company - The Company's initiatives on agility and the cloud native adoption notwithstanding.

Anyway my solution, as it is, is that I will support "blessed" versions of the toolchain. All that is left is for someone to upload these versions.

Now I'm stuck in the middle of all this, and so far it has been yours truly that is arbitrarily uploading versions of the toolchain based on what I've seen coming through in our test environment. I actually don't know what is a significant version and I don't really want to be in this loop. Further I am making decisions above my pay grade without any domain knowledge of these dismal signing tools.

The details matter and someone needs to own this and solve my riddle.

Again
  • Who is it that uploads the "blessed" versions of the toolchain to Signing?
  • More fundamentally, who decides what is a blessed version of the toolchain?
I'll end by invoking my friend Sam Ruby's Postulate
The accuracy of metadata is inversely proportional to the square of the distance between the data and the metadata.

I am clearly furthest from the data, so somebody closer needs to make the call. Perhaps I'm reading from the wrong playbook, but I'm merely trying to add value here.

Koranteng
--
Chief Toli Monger
The Company

[snip distressing long chain of emails going back 5 months of people passing the buck]

Tower of Babel

Postscript


I'd like to think that my late Friday email did have the clarifying effect I intended with its visions of Ruby's Postulate and The Tower of Babel. There were certainly calls for action, and immediate action at that. I learned that I had underestimated the extent of the dysfunction at The Company. There were actually at least 8 other teams that I might have added to my folktale, and they each had their own sorry origin story. The list of people inside The Company who I learned had read my missive was impressive and growing.

There were ruffled feathers however. People who should have dealt with the underlying organizational and communication issue were upset that I had revealed it so starkly. Lots of meetings were called. Action items were issued and so forth. I was advised to let others handle things at this stage. I was also reminded of John Kenneth Galbraith writing in The Great Crash of 1929.

... the rite of the meeting which is called not to do business but to do no business...

One of the oldest, most important - and unhappily, one of the least understood - rites in American life.

Men meet together for many reasons in the course of business. They need to instruct or persuade each other. They must agree on a course of action. They find thinking in public more productive or less painful than thinking in private. But there are at least as many reasons for meetings to transact no business. Meetings are held because men seek companionship or, at a minimum, wish to escape the tedium of solitary duties. They yearn for prestige which accrues to the man who presides over meetings, and this leads them to convoke assemblages over which they can preside.

Finally there is the meeting which is called not because there is business to be done, but because it is necessary to create the impression that business is being done. Such meetings are more than a substitute for action. They are widely regarded as action.

Everyone remembers the story of The Emperor's New Clothes, I should note however that Hans Christian Andersen didn't write the sequel about what happened to that little boy and his family six months later when The Authorities could finally deal with them.

Slightly related, the Gambian proverb goes "Words are like bullets, once you release them you can't call them back.

It is said that a prophet is never recognized in his own country and the verdict was still out on if I was to become The Company's John the Baptist. I won't tempt fate and leave it to the historians to make that determination. In closing, I'll again turn to Belloc

It is always a relief to believe what is pleasant, but it is more important to believe what is true.

— Hilaire Belloc

Office Tales


I'd previously pointed to this juxtaposition, a survival guide to life in any workplace

Office life, office politics, the organization man, managing humans, the bad child's book of beasts


Soundtrack for this note



File under: , , , , , , , , , , , , , , , , , , , ,

Wednesday, April 04, 2007

Crawl Before You Walk

I've been working for the past 18 months in the "emerging web technologies" group at IBM. When I first joined, I was struck by a certain difference in attitude compared to my previous experiences at Lotus. "Eh?", I'd almost hear my colleagues say silently whenever I'd start discussing the merits of any technology. When they responded, there'd be a quite jaundiced sound in their voices. You'd notice the slightly arched eyebrow and perhaps a tightening of the mouth. It wasn't quite incredulity but it was fairly close to wariness. I speculated that this tendency was born of years evaluating countless Next Big Things and was simply the posture of experts and futurologists: "What are you selling this time? Tell me something new...". They've seen standards come and go and the various fashions that sweep the technology world. The group is a little guerrilla outpost of pragmatism, if you will, within Big Blue.

I am fairly cynical about buzzwords and the mechanics of technology adoption, but even my experiences of the Lotus side of things hadn't made me that skeptical. In any case I mention this because I was recently merrily surfing away and came upon the following bullet item in a laundry list of proposed features for Java Server Faces (JSF) 2.0.

Allow for bookmarkable JSF pages. More broadly, if HTTP GET can be used, it should be used.
Quite simply I caught myself saying "Eh? Eh!". My normally sedate eyebrows even arched violently. I couldn't help it.

I haven't closely followed the technology known as JSF, but I think if you wanted to quickly evaluate its prospects as an 'emerging web technology', you could do well to simply dwell on what I've highlighted. Its history, its rate of adoption, its maturity and perhaps even its future prospects are fully encapsulated in that most remarkable passage.

It is striking that it is only in its fifth year of existence that a framework for building web applications is considering allowing bookmarking and might even use the HTTP GET method where applicable.

How can one possibly develop successfully for the web without identifying important resources? That is the first plank of the REST elevator pitch. If you can't bookmark, if you can't cache, if you are invisible to intermediaries, you are willfully ignoring the web.

Technology toli readers might remember an intervention and case study on internalizing the REST style that focused on much the same aspect in the case of WebSphere Portal and its first 3 years of existence. The benefits that I outlined once the portal embraced the web are much the same that will accrue to JSF. Ed Burns and company seem to have staged a REST intervention on JSF. Head nods to them.

My first encounter with JSF was when it was announced at JavaOne in 2002. The name seemed funny ("faces" instead of interfaces?) but I'd seen worse. My snap judgement: yet another framework for building web interfaces, oh well, the more competition the better, we'll see how it pans out.

I note that I've only mentioned JSF once: outlining how its abstractions leaked mightily when it came to html buttons and forms. Three years ago, we had been forced to use it in our forms work because it was deemed "strategic" in the IBM Software Group. I didn't have the clout to affect the decision and, well, I worked to get something to ship to customers. Inwardly however I discounted that project's prospects of success based on my experience with the first releases of JSF. A demo that looks good will get you in the door; an application that works well will actually get deployed and used by Real People™.

In the same piece, I referenced the reaction to the release of Google Web Accelerator and the way it highlighted the large number of web frameworks that were ignoring the basic principles of the web architecture. It seems that those frameworks that were deployed on the broad web and whose flaws were publicly exposed in that episode took that lesson to heart. Their creators have begun internalizing the web style to its core. To take an obvious example, the latest releases of Ruby on Rails seem to be almost religious in the fervour for the gospel of REST.

I suspect that JSF wasn't widely deployed outside of 'enterprises' two years ago hence the outcry that many other frameworks faced to "fix your damn code" wasn't heard. It appears that the message has since been conveyed and that fateful item speaks volumes. REST it is. Sun is going full bore towards embracing the web. And about time too.

Laissez-faire Dynamics


I have described REST, the web style, as being laissez faire distributed computing.

This is a blessing in that its laissez faire approach has the virtue of encouraging participation by as many people as possible. Indeed its embodiment in HTTP might be one of the most successful cases of technology adoption we have seen.

To pursue an argument by analogy on the other hand, I'll note that markets regularly fail. There are the temptations of monopolies seeking rents and that the economics of affluence and attention apply. There are bad faith actors, and you need to create spaces for market lubricants: price discrimination, market makers, arbitrage etc. Income inequality, trickle down economics are widely seen when we fetishize all things laissez faire. Still, there is almost no market which is fully laissez faire; we have laws and regulations even in the freest of markets.

The reigning tension of the low end theory is between participation and control. REST, in favouring participation almost to a fault, enforces very minimal regulations. HTTP is abused daily and advocates of the web style would catch hernias if they dwelled on the extent of the abuse and didn't simply get on with things. You'll only see gentle prodding in my handwaving take at Arrested Development
  • Identify all important resources
  • Use the correct verb
  • Respect browser conventions
  • Layer Stripping and Radical Simplification
There's no coercion in these suggestions. REST is all carrot and no stick. Now the thing is that once you've imbibed the web style and embraced its minimal restrictions, you don't look back and all those externalities begin to accrue. Based on the urgency of that bullet point, I am fully confident that the next few releases of JSF will be bonafide Next Big Things ®.

Most of us had to crawl before we walked, or fall off the bike a few times before we learned how to ride. I only wish that certain technologies would fail faster or at least blunt their noses earlier. Contact with a mass audience on the web seems to be essential in this respect and this presents a dilemma.

Big companies have credibility by virtue of heft and inertia when it comes to purchasing decisions - the technologies they promote get in the door almost by default. Those who sell to big companies also know that, so long as they make the right noises about the strategic direction they are going in, they will get the benefit of the doubt. I would argue that this is only true in times of affluence. When we are all minding our purses, we look more closely at these things and will use what is expedient. Contrast the spread of web native applications like wikis to vertically integrated teamrooms and content management systems. Moreover, the web radically levels the playing field. In the decade or more since the web was adopted as the preferred platform, we have witnessed an inversion of emphasis due to its vastly wider audience and scale.

I'm reminded of the words of a fellow traveller.
I suppose I'd call myself a pragmatist.

I do whatever works. I strive to fail faster. It's the results that matter, and simplicity (stripping layers) is celebrated because time is precious. We don't have time to waste on complexity and buzzwords; we're already behind on inventing the future.
Waiting 5 years before you adopt the native architecture of the web is almost inexcusable. The web won't (and didn't) wait that long. Others will route around you and their dynamism will be adopted by the marketplace. Now it's a game of catch up on that elusive thing known as mindshare and ultimately on cold cash, rueing all those missed opportunities.

The perplexing challenge of the web remains: how do we encourage things to fail faster? Perhaps we should all learn to say "Eh?" more often.

Crawl Before You Walk, a Playlist

A soundtrack for this note
  • The Pharcyde - Passing Me By
    The humourous message of this song from The Pharcyde's hugely influential debut album is that obsolescence awaits if opportunities go wanting. The chorus is wist itself: "she keeps on passing me by".
  • Was (Not Was) - Walk the Dinosaur
    Fun soul music that made the pop charts: "everybody walk the dinosaur".
  • Count Basie - Sleepwalker's Serenade
    It appears that the web style is exploding things at Sun and other big companies. It is only fitting that we end with a swinging track from the essence of swing, the big band at its tightest, the horns, the driving bass, not to mention the Count's percussive piano, provide the counterpoint to the ironic serenade of the sleepwalker. The album title: The Complete Atomic Basie. Eh!

    Atomic Basie


File under: , , , , , , , , , , , , , , , , ,