Disaster recovery is not high availability

The failure model insisted that protected is a relation between a mechanism and a named failure, never a property a system possesses, and it left one relation unexamined. Two of the most expensive line items in any infrastructure budget — the cluster and the recovery estate — are habitually justified with the same sentence, that they exist in case something goes wrong. They are not the same thing. They cover different rows of that matrix, they cost money in different shapes, and each is a poor stand-in for the other. This part is about disaster recovery specifically, and it can only begin here, because a large share of estates that believe they have a recovery capability own an availability capability instead, and a smaller share own the reverse.

Two different sentences about the same failure

High availability makes a claim about the user’s experience: the failure happened, and the person using the service did not find out. There is only one way to achieve that. Capacity has to be already running, inside the same consistency domain as the thing that failed, and something has to notice the failure and move the work onto that capacity fast enough that the move hides inside a retry. Everything with cluster, failover, load balancer, virtual IP, quorum or standby in its name is machinery for that one sentence, and its unit of measurement is the interruption that reached a user — a few dropped connections, seconds of elevated latency, a request sent twice.

Disaster recovery makes a different claim: the failure happened, the running capacity could not absorb it, the service stopped, and a deliberate process brought it back — usually elsewhere, and always from a state captured before the failure. Its units are elapsed time to a working service and the quantity of recent work that did not come back with it.

Notice what does not separate them. The tempting sort is by size, and it is wrong in both directions: an estate with a warm second site absorbs the loss of a whole building with availability machinery and no restore at all, while an estate with a perfect cluster is thrown into a full recovery by one mistyped statement. The axis is not scale. It is whether the failure landed inside the set of events the running capacity was built to absorb.

Which turns the subject into two questions asked separately of every service:

1. Is there capacity already running, inside the same consistency domain, that can take the work when this component fails?
2. Is there a state from before the failure, outside the failure’s reach, that this service can be rebuilt from?

An estate can answer yes to both, to one, or to neither, and the answers are independent. What makes the confusion durable is that both get filed under one heading in one document, so nobody has to notice that only one has evidence behind it.

The test: an earlier state, or only the current one?

One question sorts every mechanism in an estate, and the failure model supplies the vocabulary: does this mechanism hold an earlier state, or only the current one?

Availability machinery is built, without exception, out of current-state mechanisms, and that is a requirement rather than an oversight. A standby holding an older version of the database discards transactions when it is promoted; a cluster member serving stale state returns wrong answers. Convergence on the present is the entire job, and an availability design is judged by how completely and how quickly the survivors agree with what just happened. The consequence is that a change which is wrong is carried with the same fidelity and urgency as a change which is right.

Three shapes make the point.

A consensus cluster — three or five members, a replicated log, a majority required before anything commits. It tolerates the loss of a minority of its members, which is what it was built for, and Kubernetes documents two topologies for exactly that, one with etcd stacked on the control-plane nodes and one with etcd on separate machines. Now delete a key instead of killing a node. The deletion is a write: it enters the replicated log, the majority agrees, every member applies it. The stronger the consistency guarantee, the more perfectly and promptly redundancy failed to help.

A streaming replica. The primary produces a stream of changes and the standby applies them as they arrive: synchronous, it is at most one in-flight transaction behind; asynchronous, seconds or minutes. Either way the statement that dropped the table is a change in that stream, and applying changes faithfully is the standby’s definition of correct. The lag is a distance, not a version.

An active/active pair. Both members accept writes and reconcile with each other, which is the most redundant arrangement on this list and the one most often described as leaving nothing to recover from. A deployment computing wrong values writes them on whichever member took the request, and reconciliation carries them to the other, so the estate holds the defect twice, consistently.

The principle underneath all three is what the rest of this part is built on: redundancy multiplies the current state; it never produces an earlier one. Adding members, adding sites, tightening the link from asynchronous to synchronous — each changes the count or the distance, none adds the dimension recovery needs.

Read against the failure model’s rows, the split is clean. The hardware and facility classes — disk, host, rack, site, region — are where the copy sits outside the thing that broke and the content was never in question, and availability machinery answers them well. The classes about people, software and authority — a deleted file, an administrator’s mistake, a release writing bad values, ransomware — are where the content itself is the problem, and there the same machinery does the wrong thing quickly and to every copy at once.

The estate that bought HA and believes it bought DR

This is the common shape, and it is usually built by competent people. Three application nodes behind a load balancer. A database with a synchronous standby in a second availability zone and automated promotion. Storage replicating to a third. A quarterly failover exercise that passes. A dashboard green for a year. Asked whether the business is protected, all of that is real evidence — for question one.

Ask question two and the estate goes quiet in a characteristic way. There are backups, usually: written nightly by a job holding the same credential the application uses, into a bucket in the same cloud account, last restored during a migration two years ago. Coverage against disk, host, rack and zone loss is genuinely excellent. Coverage against a deleted table, a release writing bad values, logical corruption or an actor holding production credentials is close to nothing, because every mechanism above converges on whatever production most recently said.

The symptoms are physical and checkable in an afternoon:

• The only recovery rehearsal anyone can name is a failover. Nobody has restored anything into a working service within memory.
• The runbooks say promote, fail over and shift traffic. The word restore appears in none of them, or once, in a paragraph nobody has run.
• No one can state the oldest recoverable version of the primary dataset in a couple of minutes; finding out means opening a console, not reading a register.
• The backups are configured in the same console under the same identity as the systems they protect, so one authority ends both.
• The disaster recovery plan’s first step is “promote the standby”, an availability step, and there is no step two for when promotion produces a healthy copy of the broken data.

Two further hazards belong here. Availability machinery has a failure rate of its own — split brain, a fencing action that did not happen, quorum lost during maintenance, a cluster-wide certificate rotation that stops every member at once — so complexity bought to reduce outages sometimes causes them, and what fixes it then is a recovery capability. And promoting a standby that has already applied the destructive change recovers nothing; it moves which machine serves the damage.

Two hours of outage in an estate with excellent backups

Now the mirror image, rarer and just as expensive. This estate takes recovery seriously: the repository is verified by reading the data rather than only its structure, off-site copies sit under an object-lock retention window production credentials cannot shorten, and restores are rehearsed onto clean infrastructure, timed, and published. Question two has better evidence here than in most organisations.

Then the power supply in the single database host fails at two in the afternoon. There is no second instance, because redundancy was never bought, so the estate runs the mechanism it did buy: find a target, restore the base copy, replay the log, validate, repoint the application. Two hours later the service is back, having lost the minutes between the last archived segment and the failure.

The instructive part is that nothing malfunctioned. The recovery mechanism performed as designed and as rehearsed; two hours is what that architecture costs, and it was knowable in advance. What was wrong was the expectation, because the estate mistook a recovery capability for an availability one — the previous section’s error, running the other way.

The arithmetic is what makes it painful. Availability budgets are consumed by frequent small events, not by rare large ones. A commitment of 99.9% over a thirty-day month allows roughly forty-three minutes of downtime, so one two-hour restore spends about three months of that budget in an afternoon — and power supplies, kernel panics, full filesystems and failed certificate renewals arrive far more often than fires do. An estate that answers every component failure with a restore will miss a target one more running instance would have met.

There is a second cost with no equivalent on the availability side. Recovery from an earlier state means the work done since that state is gone unless something can replay it, and how much that is follows from the architecture and the schedule — how often a copy is taken, whether a change log is archived continuously, how far the archive had got — never from a figure a product supplies. An estate with excellent restores and no continuous archive is choosing to lose whatever accumulated since the last copy, and that choice deserves to be made rather than discovered.

Neither covers the other’s blind spot, so write both claims down

The two capabilities defend each other’s blind spots badly enough that substituting one for the other is a design error rather than a trade-off. Availability machinery cannot produce an earlier state, so it cannot answer the classes where the content is the problem. Recovery machinery cannot make a failure invisible, because even its cheapest form involves noticing, deciding, restoring and validating, and each of those costs minutes at best.

What makes this tractable is that both questions are answerable per service, in writing, with evidence a reviewer can check. For each service worth naming, record two claims and their evidence: what is already running that would absorb the loss of a component, demonstrated by which exercise; and what earlier state exists, how old its oldest recoverable version is, whose credentials can destroy it, and when a restore from it last produced a working service. Where a column holds a mechanism but no exercise, the claim is a plan rather than a capability.

The register exposes the same two errors wherever it is filled in. A service with a rich first column and an empty second is one bad deployment from an outage nobody has rehearsed; the reverse commits to an availability number its architecture cannot reach. Both are ordinary engineering decisions once written down, and both stay invisible while the two questions share a heading.

Production discipline

1. Ask the two questions separately for every service, and name the evidence for each. Capacity already running inside the failure domain and an earlier state outside it are independent claims; a document answering only the first while titled disaster recovery is the confusion this lesson exists to remove.
2. Apply the earlier-state test before accepting any mechanism as recovery. If it converges on the present — a consensus cluster, a streaming replica, an active/active pair, a mirror — it belongs in the availability column whatever the invoice called it.
3. Never let a failover exercise stand as evidence of recovery capability. A promotion shows that spare capacity takes over and shows nothing about restoring a state from before a change, so an estate whose only rehearsal is a failover has tested one column twice.
4. Write down the second step of every plan that begins with “promote the standby”. Promotion moves the workload onto a copy that faithfully applied whatever the primary did, so the plan needs the branch covering a healthy standby carrying broken data.
5. Size the availability side against the failures that actually recur. Power supplies, panics, full filesystems and expired certificates arrive far more often than disasters do, and answering each with a restore misses a target one more running instance would have met.

Cross-course references

• Kubernetes for Production Sysadmins — Part LXXIII (Control Plane High Availability) builds precisely the machinery this lesson classifies as current-state, with multiple control-plane nodes, an etcd quorum and a load balancer in front of the API server, while Part XCVIII (Disaster Recovery) covers what has to exist when the cluster agrees on a destructive change rather than losing a node, which is the boundary drawn here.
• PostgreSQL for Production Sysadmins — Part XIV (Replication, Slots and Read Replicas) and Part XV (High Availability, Failover and Disaster Recovery) work through the same write-ahead log in both of its uses, streamed to a standby that converges on the present and archived so a recovery target can be chosen, which is the mechanism the middle of this lesson rests on.
• Linux for Production Sysadmins — Part LII (High Availability Fundamentals) treats active/active, active/passive and failure domains as an availability discipline while Part L (Disaster Recovery) treats rebuild against restore as a recovery discipline, so reading the two side by side is the exercise this lesson asks for: one estate, scored twice, once per column.

Quiz