Availability arithmetic — what a nine actually costs

“Make it highly available” is not a requirement. It is a feeling. Two engineers will hear it and build systems that differ by a factor of fifty in cost, and neither will be able to demonstrate that theirs is the one that was asked for.

A requirement is a number, and the number implies a budget. Before choosing between a restart policy, a second host, or an orchestrator, work out how much downtime you are allowed to spend — because most Docker HA arguments dissolve the moment somebody does the arithmetic.

The budget

Availability is the fraction of time the service is usable. Its complement is the downtime budget, and that is the number worth writing on the wall.

Read the last two rows against how you actually operate. At 99.99% you have four minutes a month. A single host reboot for a kernel update is two to five minutes; one of those blows the entire monthly budget. At 99.999% you have 26 seconds a month, which is less than the time it takes a person to read a page and open a laptop — so no human is in the recovery path at all.

That is the value of the table. It tells you, before any design work, whether the target you were given is compatible with humans, with reboots, and with a single machine.

Components in series multiply

A request passes through a chain, and it fails if any link fails. The availabilities multiply:

proxy 99.9%  x  app 99.9%  x  database 99.9%  x  host 99.9%  x  network 99.9%
= 0.999 ^ 5
= 0.9950  ->  99.50%,  about 3.6 hours of downtime per month

Five components each individually meeting “three nines” produce a service that misses it by a wide margin. This is the single most useful piece of arithmetic in this lesson, because it explains a common and demoralising experience: every team reports their component is healthy, and the service is not.

It also tells you where to spend. Improving the best component in the chain changes almost nothing. The product is dominated by the worst term.

Redundancy in parallel, and why the maths lies

Two independent replicas, each 99% available, give 1 - (0.01 x 0.01) = 99.99%. Four nines from two cheap components.

The word doing all the work is independent. In a real two-host Docker deployment the replicas share:

• the same power feed and the same rack, if you did not check
• the same top-of-rack switch
• the same image, and therefore the same bug
• the same configuration, and therefore the same misconfiguration
• the same database
• the same certificate, and therefore the same expiry date
• the same deployment, applied to both at once

Every shared element is a term the formula does not contain. A correlated failure takes both replicas at the same instant, and the 99.99% you calculated describes a system nobody built.

MTTR is the term you can move

Availability decomposes as MTBF / (MTBF + MTTR) — mean time between failures over the total cycle. On a single Docker host you have almost no leverage on MTBF: the disk, the power supply and the hypervisor fail when they fail.

You have enormous leverage on MTTR, and MTTR is not one number:

MTTR = detect + notify + decide + act + verify

For most single-host incidents, act is the smallest term. Restarting a container takes two seconds. What takes forty minutes is nobody noticing for twenty, the page going to the wrong rotation, and the responder spending ten minutes deciding whether restarting is safe.

That is good news, because those are the cheap fixes:

• Detect — a synthetic probe from outside the host, checking what a user checks. Not a container-level healthcheck.
• Notify — an alert that reaches a person who is awake, with a link to the runbook in the alert body.
• Decide — a written decision rule. “If the /health endpoint has failed for 2 minutes, restart the container; do not investigate first” removes the ten minutes of deliberation.
• Act — a restart policy, so the machine does the common case without waking anyone.
• Verify — the same probe that detected it, confirming recovery.

What one Docker host can honestly promise

$ docker events --since 720h --filter 'event=die' --filter 'event=restart' --format '{{.Time}} {{.Actor.Attributes.name}} {{.Status}}'

1752489011 proj-api-1 die
1752489013 proj-api-1 restart
1753101884 proj-api-1 die
1753101886 proj-api-1 restart

Add up the real costs on a single host over a year:

That totals roughly two and a half to three hours a year, which lands near 99.97% — if nothing goes badly wrong. One four-hour hardware incident drops it to 99.92%. One bad day of debugging drops it below 99.9%.

So the honest statement about a single Docker host with good practices is: three nines is achievable and defensible; four nines is not, because the single largest term is a hardware event you do not control and cannot fail over from.

Sanity check