Root Cause Analysis
Time for the post mortem, as you know, some take it seriously
The need to revisit, the urge to explain what happened previously
For an ounce of prevention is said to be worth a pound of cure
Let's get to the bottom of things and figure out the root cause
Oftentimes it's a simple mistake, a moment of inattention
Or sometimes it's really just idle exploration
One minute you wonder, what does this button do?
Then the thing happens that you can't recant
Oops, you realize you just shut down the power plant
Human fallibility tends to be the root cause
Ah right, power. It's quite fitting in this era of modernity
That one can't sing too highly of the virtues of Electricity
So essential, we almost always overlook this august substance
We only rue its wonder when confronted by its absence
And now we've lost power and everything must stop
Oops, lights out. In Ghana we call it dumsor
Power failures are prime candidates for the root cause
The next affliction, sadly, is all too common
Like ants, human beings just like to burrow
When in a mad rush to lay down some pipes, it's nigh inevitable
So busy that we never checked to see what could be an obstacle
Dig: bureaucracy got in the way, they were moving too slow
Oops, the contractor cut the critical cable with his backhoe
All too often, cable cuts tend to be the root cause
Things fall apart, they say,
equipment sputters, machines fail
They blow hot and cold, or crack when used,
there's wear and tear
Material scientists make a roaring trade
as do structural engineers
That, sadly, alchemists never overcame nature's challenge
is the lesson learned
Oops, the widget broke,
a reminder that no condition is permanent
In this industrial age, hardware failures are a likely root cause
Sometimes you're just too popular,
so crowded no one can get in or breathe
Congestion is the operative word,
in matters of scale, a crowd changes things
Your service is the flavor of the month,
and now you've become essential
Oops, you're completely unprepared for when you go viral
Woe is me, lack of capacity is frequently the root cause
And then we come to the bad actors,
forever on the attack
Always probing for an opening,
for vulnerabilities in your stack
And that's even before we consider
the gremlins and parasites
Iconoclastic beasts with distinctive manners
and singular appetites
Every complex ecosystem in history
has had to deal with grifters
Oops, your hospital is held to ransom
by a band of sneaky hackers
Always protect yourself, a lapse in security is invariably the root cause
There's more in this vein,
mankind has never built a system without error
From the Tower of Babel to that fancy car,
or even that blasted word processor
The raw materials of life,
whether it's the design or the initial conception
Imposing one's will,
it might be a flaw in the ultimate implementation
You probably have your own experience and area of expertise
Your own rules of thumb about these puzzling mysteries
Let me tell you something
from my profession of software engineer
If you only knew,
to defend a system in depth is an exercise in fear
How close we come to catastrophe,
partial or complete, every day
Trust me, you really don't want to see how the sausage is made
Now one could argue about the order
of this short list of failure modes
It is only in retrospect that one is truly able to diagnose
The human burden is to keep moving
in the face of systemic error
To mitigate the worst,
to build the fail-safes and systematic procedures
Spare a thought for the moron in a hurry,
for one day it could be you
That, through omission or commision,
will be blamed for the miscue
We haven't scratched the surface
of how much the human factor has an impact
Fall back to folk wisdom,
suffice to say that curiosity killed the cat
And what of the wider world,
say failed love affairs, or even wars?
It's only human to search for simple answers
and the root cause
Our prophets and philosophers have long emphasized
moral suasion and the golden rule
You could do hardly do worse than social living
and the mosquito principle
Focus on best practices, usability, and layers of protection
Try to put a process in place and make it official!
Make sure that it takes many, many big red buttons
to launch that nuclear missile
If there's any moral to this tall tale of root cause analysis
Take heed, and wherever possible, make use of the checklists
After
- How Complex Systems Fail (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety) by Richard I. Cook
- a quip about network outages from Sean Donelan
Root Cause, a playlist
A soundtrack for this note (spotify version)
- Oops (Oh My) by Tweet ft. Missy Elliot
- My Mistake (was to love you) by Marvin Gaye
- Reasons by Earth, Wind & Fire
- Mistake by Fela Kuti
- Knockin' at the Wrong Door by The Rollers
- A Few Reasons by Dwele
- Who Can We Blame by Mica Paris
- Lessons by Eric Roberson
See also: The Dining Philosophers Problem, Resilience and Adaptability, and Version Hell Revisited
This belated entry on failure modes is part of the Toli Technology Series
File under: failure, systems, resilience, best practices, human factors, networks, operations, error, technology, software, hardware, culture, observation, perception, design, humour, strategy, Observers are worried, poetry, toli
Writing log: April 3, 2022
