Tuesday, February 27, 2024

Root Cause Analysis

Time for the post mortem, as you know, some take it seriously
The need to revisit, the urge to explain what happened previously
For an ounce of prevention is said to be worth a pound of cure
Let's get to the bottom of things and figure out the root cause

Oftentimes it's a simple mistake, a moment of inattention
Or sometimes it's really just idle exploration
One minute you wonder, what does this button do?
Then the thing happens that you can't recant
Oops, you realize you just shut down the power plant

Human fallibility tends to be the root cause

Ah right, power. It's quite fitting in this era of modernity
That one can't sing too highly of the virtues of Electricity
So essential, we almost always overlook this august substance
We only rue its wonder when confronted by its absence
And now we've lost power and everything must stop
Oops, lights out. In Ghana we call it dumsor

Power failures are prime candidates for the root cause

The next affliction, sadly, is all too common
Like ants, human beings just like to burrow
When in a mad rush to lay down some pipes, it's nigh inevitable
So busy that we never checked to see what could be an obstacle
Dig: bureaucracy got in the way, they were moving too slow
Oops, the contractor cut the critical cable with his backhoe

All too often, cable cuts tend to be the root cause

Things fall apart, they say,
   equipment sputters, machines fail
They blow hot and cold, or crack when used,
   there's wear and tear
Material scientists make a roaring trade
   as do structural engineers
That, sadly, alchemists never overcame nature's challenge
   is the lesson learned
Oops, the widget broke,
   a reminder that no condition is permanent

In this industrial age, hardware failures are a likely root cause

Sometimes you're just too popular,
   so crowded no one can get in or breathe
Congestion is the operative word,
   in matters of scale, a crowd changes things
Your service is the flavor of the month,
   and now you've become essential
Oops, you're completely unprepared for when you go viral

Woe is me, lack of capacity is frequently the root cause

And then we come to the bad actors,
   forever on the attack
Always probing for an opening,
   for vulnerabilities in your stack
And that's even before we consider
   the gremlins and parasites
Iconoclastic beasts with distinctive manners
   and singular appetites
Every complex ecosystem in history
   has had to deal with grifters
Oops, your hospital is held to ransom
   by a band of sneaky hackers

Always protect yourself, a lapse in security is invariably the root cause

There's more in this vein,
   mankind has never built a system without error
From the Tower of Babel to that fancy car,
   or even that blasted word processor
The raw materials of life,
   whether it's the design or the initial conception
Imposing one's will,
   it might be a flaw in the ultimate implementation

You probably have your own experience and area of expertise
Your own rules of thumb about these puzzling mysteries
Let me tell you something
   from my profession of software engineer
If you only knew,
   to defend a system in depth is an exercise in fear
How close we come to catastrophe,
   partial or complete, every day
Trust me, you really don't want to see how the sausage is made

Now one could argue about the order
   of this short list of failure modes
It is only in retrospect that one is truly able to diagnose
The human burden is to keep moving
   in the face of systemic error
To mitigate the worst,
   to build the fail-safes and systematic procedures

Spare a thought for the moron in a hurry,
   for one day it could be you
That, through omission or commision,
   will be blamed for the miscue
We haven't scratched the surface
   of how much the human factor has an impact
Fall back to folk wisdom,
   suffice to say that curiosity killed the cat

And what of the wider world,
   say failed love affairs, or even wars?
It's only human to search for simple answers
   and the root cause
Our prophets and philosophers have long emphasized
   moral suasion and the golden rule
You could do hardly do worse than social living
   and the mosquito principle

Focus on best practices, usability, and layers of protection
Try to put a process in place and make it official!
Make sure that it takes many, many big red buttons
   to launch that nuclear missile
If there's any moral to this tall tale of root cause analysis
Take heed, and wherever possible, make use of the checklists


wide load coming through


After
  • How Complex Systems Fail (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety) by Richard I. Cook
  • a quip about network outages from Sean Donelan

Root Cause, a playlist


A soundtrack for this note (spotify version)
See also: The Dining Philosophers Problem, Resilience and Adaptability, and Version Hell Revisited

This belated entry on failure modes is part of the Toli Technology Series

File under: , , , , , , , , , , , , , , , , , , ,

Writing log: April 3, 2022

Tuesday, February 20, 2024

Regret Is All

Regret is all, it's only human
An emotion with an intimate audience
The tinge of shame and imperfection
And the taint, for we were never innocent

For some can never descend the mountain of regret
They can't let go of the history or even decide to forget
Forever revisiting the past, vivid scenarios replayed
It's hard to live in the bed of remorse that they've made

Questioning whether the requisite words could have been found
Or whether alternative actions could have been summoned
Standing on the verge, poised between restraint and impulse
Rueing whether different actors might have changed the outcome

Communal regret is often the essence of cultural memory
To find oneself laying down markers of collective identity
Intimations of free will weighed against chaos theory
The direction of the arc of time and missed opportunities

Regret is all, it's a deficiency disease
Testimony to the fallibility of memories
Or sometimes to a surfeit of imagination
The raw materials of life, regret is an imposition


boy on steps of crumbling castles


Regret, a playlist


A soundtrack for this note (spotify version)
Vaguely related: Decision to Forget. The theme is buyer's remorse.

File under: , , , , , , , ,

Writing log: March 31, 2022

Wednesday, February 14, 2024

Safe Harbor

My hope is to disappear into you
Dissolving slowly as you forget my name
When our first encounter becomes a distant memory
And the phantom sensation is all that remains

Streams of thought, the quality of evanescence
Happy to be wrapped in a blanket of impermanence
Ephemeral traces, the comfort of a mood marker
It was in your very self that I found my safe harbor


sumi swirls


Safe Harbor, a playlist


A soundtrack for this note (spotify version)

See previously I Daresay

File under: , , , , , , , ,

Writing log: March 10, 2022

Tuesday, February 06, 2024

Waiting for the Pope

The Pope and Archbishop were coming
And so we waited in the morning sun
Sensible, well pressed skirts for the girls
Boys in sharp shorts, our khaki school uniforms
At the behest of stern teachers and chaperons
We all made an effort to tamp down our ruffian ways
Arrayed, or rather displayed, in formation at the roadside

Earlier, after assembly, we'd marched out of the school gates
Passing through the dusty streets of North Ridge
And made our way to Ring Road, all the while chanting
Oh the heat, the hours passed without a sighting
No water, no snacks, but the novelty of the excursion compensated
Seven year olds will find their own entertainment
Serene and papal, we practiced our hand waves
Then, inspired, tried out our Polish-accented English
"Holy, holy, very holy... I bless you, my children."

In the event, after all that, it was a two minute affair
No popemobile as it turned out, a rather sober procession
Cars bearing dignitaries preceded by the motorcade
Cheers, such cheers, and a mad scramble
We bore witness

A slight man in the flesh, pint-sized
Can't say that he was larger than life
Still, you could have sworn he looked you directly in the eye
Made you feel a connection even at the roadside
Our healthy skepticism of catholicism was almost overcome
The great man theory, we'd learned some kind of civic lesson
We savored the escape and managed to wrangle a further hour of fun
Then slowly, very slowly, made our way back to North Ridge Lyceum


ghana stamp pope john paull II archbishop of canterbury with president Hilla Limann 8-10th May 1980 20 pesewas


Waiting for the Pope, a playlist


A soundtrack for this note (spotify version)
education church or state



See previously Articles of Faith and On Catholicism

File under: , , , , , , , , ,

Writing log: March 28, 2022