Host availability vs service availability - measuring the right thing

The monthly report says the host was available 99.99% of the month. The incident log says the service was unreachable for forty minutes. Both are correct, and the gap between them is where most availability arguments happen.

Two different numbers

Host availability is the fraction of time a machine was powered on and responding at the level you tested. Service availability is the fraction of time a user could complete a request successfully.

They are different measurements of different things, and a cluster changes one of them far more than the other. Pacemaker can move a database off a dead node in ninety seconds; it cannot do anything about a certificate that expired on all three nodes at the same moment.

What “up” actually means

“Up” is a ladder, and each rung is a stronger claim than the one below it:

Availability reports are usually built from the top two rungs because they are the cheapest to collect. That is the whole mechanism behind “the host was up and the service was down”.

$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://app.example.com/api/v1/orders/self-test

200 0.184s

Serial dependencies multiply

A request that must traverse several components in sequence succeeds only if all of them work. Their availabilities multiply:

Load balancer   99.99%
App server      99.95%
Database        99.95%
DNS             99.99%
--------------------------------
Product         99.99 x 99.95 x 99.95 x 99.99 = 99.88%

Every component is individually better than three 9s, and the service is below it. This is the arithmetic that explains why adding a component to a critical path is never free, and why “we hardened the database” moved the service number less than anyone expected.

Two consequences follow directly:

• The worst component sets the ceiling. No amount of work on the other three raises the product above the weakest one.
• Fewer serial hops is a design lever. Putting cluster peer names in /etc/hosts removes DNS from the membership path, which removes a whole factor from that product.

Parallel redundancy divides the failure

Redundancy works the other way round. If a request can be served by any of n independent replicas, you multiply the unavailabilities:

unavailability of n replicas = (1 - a)^n

One app server at 99.9%   unavailability 0.001        = 0.1%
Two, independent          0.001 x 0.001   = 0.000001  -> 99.9999%
Three, independent        0.001^3         = 1e-9      -> 99.9999999%

The word doing the work in that formula is independent. Two app servers in the same rack, on the same switch, with the same configuration pushed by the same automation, do not have independent failure probabilities. linux-failure-domains covers how to find the shared factor; the arithmetic above is an upper bound you will not reach.

Where a cluster helps, and where it does not

A Pacemaker or keepalived cluster raises availability against exactly one class of failure: the loss of one node in the cluster. Line the failure classes up against it:

The first two rows are what people buy a cluster for. The rest are what actually takes services down once a cluster exists, which is why the dependency work in linux-dependencies-dns-certs-identity matters more to the service number than adding a fourth node.

The states where the host is up and the service is not

These are the ones that make host uptime a misleading proxy:

• The cluster is deliberately not starting the resource. A fence action failed, so Pacemaker will not start the database anywhere, because it cannot prove the peer stopped. Every node is powered on and quorate. Host availability for the month: 100%. Service availability: zero for the duration.
• The resource is running on a node that cannot serve. The VIP moved, but the switch still has the old MAC in its table, or the backend firewall only permits the old node.
• The service answers, wrongly. A replica that has fallen hours behind still returns HTTP 200. Staleness is an availability failure from the user’s point of view and is invisible to every liveness probe.
• The service answers, slowly. If your objective is a p99 under one second and p99 is nine seconds, the service is not available in any sense the user recognises, even though every request eventually completes.

Measure from the user’s vantage point

The rule is simple to state and unpopular to implement: measure at the point the user is, over the path the user takes.

• A probe running on the host cannot detect a firewall rule, a DNS failure, a routing problem or a load balancer that has removed the node from its pool. It shares too much with the thing it is measuring.
• A probe from another network segment, resolving the real name and using the real certificate, exercises the whole chain. Every hop it crosses is a hop it can report on.
• Aggregate as a request success ratio, not as a time-based average of node states. “99.9% of requests succeeded” and “the nodes were up 99.9% of the time” answer different questions, and the SLA is about the first.

Worked example

A service is fronted by two app servers behind one load balancer, backed by one database:

Load balancer (single)           99.95%
App tier (2 x 99.9%, correlated
  failure rate 0.05%)            99.95%   (not 99.9999%)
Database (single)                99.9%
-------------------------------------------
Service                          99.80%   = 87 minutes/month

The target is 99.9%, which is 43.8 minutes a month. The service misses it, and the table says exactly where to spend: the database is the weakest serial term and the only one with no redundancy at all. A third app server changes nothing, because the app tier is already limited by its correlated failure rate rather than by node count.

That is the point of composing the number rather than measuring one host: the arithmetic names the next piece of work.

Knowledge check