Sensor Calibration

The problem is often couched as one of recognition
The difficulty, as it were, of sensor calibration
For if you can't measure a signal as a matter of first principle
If you can't detect accurately, you might as well be invisible

The auto-focus systems in cameras that can't detect those darker than blue
The pulse oximeters giving false hope - the all clear, to those with a darker hue
Training data, darker skin tones - unusual, light exposure
Biometrics, facial recognition, fingerprint sensors

Architectures of participation and control
Resigned to playing the tenuous outsider role
In this rigged game of life (and death) with ever changing rules
Arbitrary boundaries, when lines are drawn, borders can be cruel

For those excluded from the system, then, a matter of quiet advocacy
A liberation struggle of sorts, forever teaching others how to see
File under the banner of cultural sensitivity in technology
Sensor calibration and relief from the burden of invisibility

Sensor Calibration, a playlist

A soundtrack for this note, fodder for sensitive souls (spotify version)

• Sensitivity by Ralph Tresvant
• You Can't Miss What You Can't Measure by Clarence Carter
• My Sensitivity (Gets in the Way) by Luther Vandross
• You Can't Miss What You Can't Measure by Funkadelic
• We People Who are Darker than Blue by Curtis Mayfield
• Skin I'm In by Cameo
• Choice of Colors by The Impressions

See previously Cultural Sensitivity in Technology and Empire State of Mind

File under: technology, culture, observation, perception, history, race, localization, photography, health, design, software, hardware, sensors, best practices, exclusion, participation, poetry, Observers are worried, toli

Writing log: September 21, 2022

Root Cause Analysis

Time for the post mortem, as you know, some take it seriously
The need to revisit, the urge to explain what happened previously
For an ounce of prevention is said to be worth a pound of cure
Let's get to the bottom of things and figure out the root cause

Oftentimes it's a simple mistake, a moment of inattention
Or sometimes it's really just idle exploration
One minute you wonder, what does this button do?
Then the thing happens that you can't recant
Oops, you realize you just shut down the power plant

Human fallibility tends to be the root cause

Ah right, power. It's quite fitting in this era of modernity
That one can't sing too highly of the virtues of Electricity
So essential, we almost always overlook this august substance
We only rue its wonder when confronted by its absence
And now we've lost power and everything must stop
Oops, lights out. In Ghana we call it dumsor

Power failures are prime candidates for the root cause

The next affliction, sadly, is all too common
Like ants, human beings just like to burrow
When in a mad rush to lay down some pipes, it's nigh inevitable
So busy that we never checked to see what could be an obstacle
Dig: bureaucracy got in the way, they were moving too slow
Oops, the contractor cut the critical cable with his backhoe

All too often, cable cuts tend to be the root cause

Things fall apart, they say,
   equipment sputters, machines fail
They blow hot and cold, or crack when used,
   there's wear and tear
Material scientists make a roaring trade
   as do structural engineers
That, sadly, alchemists never overcame nature's challenge
   is the lesson learned
Oops, the widget broke,
   a reminder that no condition is permanent

In this industrial age, hardware failures are a likely root cause

Sometimes you're just too popular,
   so crowded no one can get in or breathe
Congestion is the operative word,
   in matters of scale, a crowd changes things
Your service is the flavor of the month,
   and now you've become essential
Oops, you're completely unprepared for when you go viral

Woe is me, lack of capacity is frequently the root cause

And then we come to the bad actors,
   forever on the attack
Always probing for an opening,
   for vulnerabilities in your stack
And that's even before we consider
   the gremlins and parasites
Iconoclastic beasts with distinctive manners
   and singular appetites
Every complex ecosystem in history
   has had to deal with grifters
Oops, your hospital is held to ransom
   by a band of sneaky hackers

Always protect yourself, a lapse in security is invariably the root cause

There's more in this vein,
   mankind has never built a system without error
From the Tower of Babel to that fancy car,
   or even that blasted word processor
The raw materials of life,
   whether it's the design or the initial conception
Imposing one's will,
   it might be a flaw in the ultimate implementation

You probably have your own experience and area of expertise
Your own rules of thumb about these puzzling mysteries
Let me tell you something
   from my profession of software engineer
If you only knew,
   to defend a system in depth is an exercise in fear
How close we come to catastrophe,
   partial or complete, every day
Trust me, you really don't want to see how the sausage is made

Now one could argue about the order
   of this short list of failure modes
It is only in retrospect that one is truly able to diagnose
The human burden is to keep moving
   in the face of systemic error
To mitigate the worst,
   to build the fail-safes and systematic procedures

Spare a thought for the moron in a hurry,
   for one day it could be you
That, through omission or commision,
   will be blamed for the miscue
We haven't scratched the surface
   of how much the human factor has an impact
Fall back to folk wisdom,
   suffice to say that curiosity killed the cat

And what of the wider world,
   say failed love affairs, or even wars?
It's only human to search for simple answers
   and the root cause
Our prophets and philosophers have long emphasized
   moral suasion and the golden rule
You could do hardly do worse than social living
   and the mosquito principle

Focus on best practices, usability, and layers of protection
Try to put a process in place and make it official!
Make sure that it takes many, many big red buttons
   to launch that nuclear missile
If there's any moral to this tall tale of root cause analysis
Take heed, and wherever possible, make use of the checklists

After

• How Complex Systems Fail (Being a Short Treatise on the Nature of Failure; How Failure is Evaluated; How Failure is Attributed to Proximate Cause; and the Resulting New Understanding of Patient Safety) by Richard I. Cook
• a quip about network outages from Sean Donelan

Root Cause, a playlist

A soundtrack for this note (spotify version)

• Oops (Oh My) by Tweet ft. Missy Elliot
• My Mistake (was to love you) by Marvin Gaye
• Reasons by Earth, Wind & Fire
• Mistake by Fela Kuti
• Knockin' at the Wrong Door by The Rollers
• A Few Reasons by Dwele
• Who Can We Blame by Mica Paris
• Lessons by Eric Roberson

See also: The Dining Philosophers Problem, Resilience and Adaptability, and Version Hell Revisited

This belated entry on failure modes is part of the Toli Technology Series

File under: failure, systems, resilience, best practices, human factors, networks, operations, error, technology, software, hardware, culture, observation, perception, design, humour, strategy, Observers are worried, poetry, toli

Writing log: April 3, 2022

On AMD, Apple, Intel, IBM and The Great Game of Chips

I posted this 10 days ago in my internal IBM Blogcentral joint but thought it might deserve a wider audience.

• The Great Game of Chips
• The Low End Theory in Hardware
• On Mobile Phones in Nigeria

The Great Game of Chips

Robert Russell points to rumours of Apple to move from PowerPC to Intel. Indeed like the question of who would be the new pope a few weeks ago, the entire blogosphere is abuzz with prognostications. Everybody seems to be concentrating on Apple's strategic outlook and admittedly they are the sexiest company in technology. Intel would be grateful, Microsoft would be curious, IBM would be unhappy but who cares about those companies, right? The Big Apple is where the action is at, right?

The elephant in the room of all these discussions is AMD.

Intel has been falling behind under the repeated onslaught of AMD's Hammer architecture, first with the Athlon which beat them in the race to 1 GHz, then the Opteron favoured by white box manufacturers everywhere, the move to 64 bit computing on the desktop where Athlon 64 rules(now that Windows XP 64 bit edition has been released, that paradigm is legitimized even though Linux and BSD were there earlier), and now in the dual core race in which Intel again don't have a competitive or affordable offering. In all of these things, Intel has played second fiddle to the fleeter AMD. They have made huge architectural mistakes read the Pentium IV which is too big, runs too hot and doesn't perform, rushed and recalled products. If you read ARS Technica, AnandTech or Tom's Hardware this would be the story that enthusiasts would tell.

It is only a huge amount of payola (read the Intel Inside war chest) that has kept people like Dell exclusively in the Intel camp. Indeed all the first tier manufacturers have AMD in their product lines. Now all affiliate programs are smart marketing and moral payola so don't read my words as pejorative. Obviously also, Intel has the edge on process technology and scale which will mitigate the fallout, but it is only a truism to say that they are have been shaken by the AMD onslaught. They, like Microsoft, can turn on the dime and become "hardcore" about a given technological approach. The problem in The Great Game of Chips is that you have those pesky fabs which take 18 months to a 2 years to turn around, thus any mistake you make ties you up for a good 8 to 10 quarters which is an eternity in Wall Street terms. In mitigation for Intel, AMD can't afford any mistakes given their harrowing corporate history and the skepticism that the market's familiarity with Intel entail. However, with their Dresden fab humming along, they've been able to have generate product at will and on time, and more importantly to control pricing: the best chips on the market always command premium pricing which AMD had never been able to have in the past. It doesn't hurt also that they aren't trying to compete with their clients, the chipset manufacturers, like Intel is so the ecosystem around their offerings is chugging along nicely.

The big bang of the web, of the data center, came about because in the hardware arena, AMD was pushing Moore's law at a furious pace to the delight of Taiwanese chipset manufacturers everywhere. Google's server farms in the past used mostly Intel but I'd hazard that the reason those servers were so cheap and disposable was the competition between Intel and the unsung AMD.

The replacement cycle for machines bought during the of the Y2K hype was a little delayed by the bust of 2000-2001 and the current Iraq war uncertainty but in the next 18-24 months the hardware arena is going to be exciting. And AMD has product, Microsoft has even been complaisant with Windows XP 64 on the software front, they too don't want a monopoly on the hardware front. While the dual core dream might be delayed, a nice dual-processor Opteron with 64 bit Linux distribution would serve most small companies very well. If you won't deal with PowerPC, IBM and others will sell you some happily.

So they, Intel, with 18 months before competitive products will come on tap, need a new high profile Dell to staunch the bleeding. Thus I'm sure they are offering huge incentives to Apple, and must be strutting like peacocks to the temperamental Steve Jobs who can afford to give glimpses of flesh under the burkha. If Apple were indeed to make the move, it would make more sense to go with lower cost AMD, much like Sun have done. If a deal is signed, Jobs would be insane to make it an exclusive one even though that is what Intel would want.

The folks like HP and DEC/Alpha who blinked and mothballed the development of their chips to fall on the Itanium sword, or Sun who dilly-dallied about with UltraSparc earlier on in the 1990s must be rueing their decisions over the past decade but those executives are mostly departed to greener pastures (Carly was even rumoured to be up for the World Bank job before Wolfowitz got the nod - again a demonstation about failure being rewarded but such is life).

When I saw the way the Power architecture was going over the past decade, I was a little frustrated that IBM wasn't burning the midnight oil in our PowerPC efforts at the same furious pace as Intel and AMD. But I suppose we've always moved more slowly. But to our credit, we (IBM) have managed to keep the Power architecture and now the Cell in the game and indeed we're patting ourselves on the back like Kingmakers. The problem is that we don't have the notebook processors or megahertz that marketers and users would like and companies like Apple like a few bragging rights. The Fishkill fab is coming along but competitors might paint it as roadkill, certainly as compared to AMD.

I still think that IBM getting out of the commodity PC business in the Lenovo deal was a huge mistake. But anyway I wasn't consulted - I'm just a software engineer not an MBA type. Presumably some bean counter run the numbers and maybe the rhetoric was about not being in commodity businesses. I on the other hand, believe that you really want to be in the high volume market, if only to get the backwash of innovation that only comes with burning the midnight oil for fear of competitors. Otherwise you relax and have grand ideas and 10 year titanic roadmaps ala Itanium. I don't see Motorola, or Nokia getting out of the mobile phone business just because LG and Samsung are now ascendant. To take the analogy further, IBM has played the Ericsonn/Sony card in the chip business and taken the flight to quality of business and "high end server machines". No consumers for us except for laptops. Meanwhile Samsung can't make enough cell phones and is turning over new models every ninety days to keep the world satisfied.

Obligatory disclaimer. The first stock I bought (outside of the widow's investment that is Big Blue) was that snappy underdog AMD. That was when I priced things out realized that the Athlon was a far better deal than the Pentium III and now IV. I also bought my dad a nice IBM Aptiva, made by IBM Canada, that was Athlon-based but some executive decided to mothball Athlons in the IBM PC business so that was a 6 month window for AMD (or was that Intel Inside dollars at work). This particular technology toli does not constitute financial advice. Past performance is indeed no predictor of future performance. You'd be insane to assume that my pontifications about technology and the Kremlin-watching that I write about has any validity in the real world.

Also keep in mind that my first tentative investments were made just before the bubble burst. And if you still think I have insight, come around my way, I have $500 of Global Crossing and $400 of Nortel stock certificates for you. At least with the Nortel variety you'd be able to buy half a glass of bland Miller Lite beer; certainly you couldn't afford Star beer or your favourite designer microbrewed concoction. More likely it will be the cheapest akpeteshie depending on which palm wine bootlegger you go to.

As for the Global Crossing paper, well store that away in pages of the most obscure library book you can find. Future historians need to know all about the hubris of the past.

Anyway I love this Great Game of Chips, consumers and the web have benefitted immensely. I hope it continues apace.

Let me end with a Coca-Cola coffin to go with your Bag of Chips (the Ga people of Ghana will gladly build great fantasy coffins for you.

The Low End Theory in Hardware

[Update] I cross-posted The Great Game snippet to Dave Farber's Interesting People mailing list and got quite a few responses.

Mark Stahlman gently corrected me

1) AMD does *not* have its Dresden Fab 36 humming along and won't until sometime next year. As a result it is capacity constrained and cannot really do any damage to Intel at the moment.

I stand corrected but I guess that is in line with AMD's harrowing corporate history which I did allude to, and also to the time lag it takes for mistakes to be corrected or for capacity to come online in the chip world. In contrast in the web world, services can be deployed by the fleetest within 3 to 6 months. Execution in the hardware world is of more critical importance, you can't deploy busloads of consultants armed with duct tape to patch your dodgy chip.

2) When it come to high-performance memory and I/O capabilities, the current sweepstakes has IBM in first, AMD second and Intel in a *distant* third.

As an Lotus/IBMer I should be gratified that you see things in such blue-tinted lenses and perhaps I too should pat myself on my software back. My assessment is that IBM has been happy to have premium pricing and to concentrate on the high end. That is fine as a strategy I suppose, but I'm interested in the ground level and the low end where issues of scale and innovation prevail instead of the scleroticism of the Ivory Tower.

If I took a walk through Google or Yahoo server farms, I doubt I'd see many high-end Power, Itanium or even UltraSparc machines.

The Low End Theory ^[1] is the lesson of Mr Moore in the data center or Jim Gray's Distributed Computing Economics

3) There is a fundamental battle going on between standard and custom parts... Where will the future volumes be? Which markets -- games, TVs, cellphones, cars, etc. -- will go which way?

This is why I call it the The Great Game. It is like the current Scramble for Congo where everyone has heard about the untold riches, copper, gold, diamonds, coltan etc (the armies of 13 countries have been in that failed state searching for plunder over the past decade, and that's before the UN, and the French got on the ground). The Great Powers want to carve out some loot from their chips (silicon after all being sand). And speaking of coltan or tantalite that you'd find in your mobile phone, the cellphone must be the greatest mass market these days the equivalent of the personal computer.

e.g. within this past year, Nigeria installed the infrastructure for 1 million cell phone lines ^[2]. From zero to over a million in a year is stunning and perhaps emblematic of pent up demand for development. I don't know the numbers for India or China, but that is a sign of a mass-market, driving the kind of innovation we used to see in the personal computer world. On this list, there's a periodic argument about whether the US is falling behind Europe, Japan or the growling tigers. It is hard to say but certainly having spent this past weekend in London, I can testify to the ascendancy of the small form factor.

The problem for the chip makers is that the price point for the mobile chips that are being sold to emerging markets is actually quite low, and the concerns are about power consumption and battery life, things that like Detroit's focus on big Cadillacs and now SUVs might make one have to rejigger the strategy and rely on huge volume. On the desktop chip side, they'll be depending on the replacement cycle, my Y2K machine is getting a little long in the tooth, or the move to 64 bit or to dual core (like the hyperthreading buzzwords foreshadowed).

Glad you like AMD -- so do I but there are still many fundamental issues unresolved as we rapidly deploy the next-generation platform and prepare for global expansion in digital services.

What I like is not necessarily the company, indeed, my stockbroker seeing my play portfolio treats me like a dilettante.

What I like is the effect the competition is having on the fundamentals of the industry. The notion that Intel is aggressively courting all and sundry to not jump ship and is rushing out dual core parts and pricing them at a level that must be painful for their cautious CFOs. The notion that Microsoft will actually sit down and support AMD64 so that those RedHat ants from the village of Linux don't get too much of a head start tickles me. The notion that Apple can show a little leg and give everyone their collective titillation quotient at the beginning of the silly season. The notion that executives at IBM have been sweating for the past few weeks trying to stem the flow is great for me. In the trenches were I live, I deal the occasional insane mandates from on high, I like others to share my pain. Plus the users all benefit. I'm looking for a nice box for my grandma and I'm glad to have a little pricing power.

[1] For the musically inclined The Low End Theory is the name of an album by A Tribe Called Quest (in the top 10 albums of the 1990s IMHO). I need to do a proper appreciation of it at some point because it is worth writing about, and at length.

On Mobile Phones in Nigeria

My original statement was

e.g. within this past year, Nigeria installed 1 million cell phone lines.

Ari Ollikainen lept right in

That's a great oxymoron!

Would you mind explaining to me what you actually mean by that sentence?

Did Nigeria install a mobile phone infrastructure capable of supporting a million cellphone conversations SIMULTANEOUSLY?

Or was a mobile phone infrastructure capable of supporting some statistical model of call holding times much << that 1 million?

Or was it a mobile phone infrastructure with an undefined service spec BUT a million cellphones were sold to the public?

Please illuminate me...

My response...

Ah... the joys of rushed emails while the server restarts. But yes a great oxymoron.

I think it was angling towards your 2nd and especially your last characterizations.

"a mobile phone infrastructure with an undefined service spec BUT a million cellphones were sold to the public?"

The actual quote I had was from a minister of neighbouring Ghana and it was

"capacity for 1 million mobile phone lines had been installed and it was fully utilized indeed they are in a mad rush to upgrade their backbone and deploy more cells"

Knowing a little about the situation in my nearby, smaller Ghana where Telenor (from Norway) where have taken over the running Ghana Telecom and are in fierce competition with a couple of other mobile phone operators Spacefon and another that I forget, there must be an element of over-selling of capacity. Spacefon, when they started out maybe 5 years ago, sold several hundred thousand cell phones having only capacity for 100,000 lines. It was rough going back in 2000-1. 5 years later, they and Ghana Telecom have built out their infrastructure to actually match the usage and the pent up demand.

Depending on who you talk to there are 80-100 million people in Nigeria, and landline penetration is known to be beyond abysmal. Knowing also the fervent economic acumen of the land of the 419 scam, it wouldn't surprise me if well say 3-4 million cell phones were sold on an infrastructure that would statistically service 1 million cell phone lines to 'western' standards.

The thing though is this

1. there was no mobile phone infrastructure in Nigeria up until the last few years
2. for the businessmen and farmers who have adopted these mobile phones in droves (and I'll handwave a few million), even though service is of an order that would make westerners scoff, it is plainly Good Enough and certainly far better than the alternatives (having to drive 4 hours on dangerous potholed roads to meet someone at their house just to discuss price or the slightest thing).

Now that you've asked, I'll dig a little further to elucidate my statistics better and perhaps will blog the results...

Thanks for keeping me on my toes...

Thus I have another Lazyweb request. Does anyone have more precise statistics about the cell phone penetration rates in Nigeria in recent years or knowledge about the details of their build out?

This is part of The Great Game of Technology series.

See also: The Low End Theory of Networks

File under: technology, hardware, chips, amd, intel, apple, ibm, sun, microsoft, hp, strategy, software, observation, ga, ghana, coffins, nigeria, mobile, phones, cell, infrastructure, toli