Executing a failover

Secrets, identity and the third parties outside the estate were the last of the preconditions — the things that must be true before a failover can work at all. What is left is the act itself, and almost none of it is technical. It is a sequence of decisions about authority, order and evidence: who may say the word, what must be true before anything moves, what is switched off before anything is switched on, in what order the estate returns, and the moment after which the incident stops being reversible. A rehearsal with a friendly clock and a cooperative primary exercises the commands and skips every one of them.

stateDiagram-v2
    state "Primary serving" as Primary
    state "Failover declared" as Declared
    state "Primary fenced" as Fenced
    state "Recovery tiers starting" as Starting
    state "Service externally validated" as Validated
    state "Recovery site accepts writes" as Writing
    state "Failback required" as Failback
    Primary --> Declared: observable criterion or time threshold
    Declared --> Fenced: authority recorded
    Fenced --> Starting: split-brain impossible
    Starting --> Validated: each dependency gate passes
    Validated --> Writing: traffic and write authority move
    Writing --> Failback: histories now diverge

Fencing precedes every start, validation precedes traffic, and the first write at the recovery site is the boundary after which returning is a failback rather than an abort.

The declaration is a clock, not a judgement

NIST SP 800-34 Rev. 1 divides the execution of a contingency plan into an activation and notification phase, a recovery phase and a reconstitution phase, and activation comes first for a reason easy to miss: nothing in the plan happens until someone invokes it. Two things have to be settled long before the incident — the criteria for activation, and the people who hold the authority to activate — and both are routinely left to the outage they were meant to shorten.

Authority sits with a named role, held by a named person with deputies in a stated order, all reachable through a channel that does not depend on the failing estate. It must be standing: the holder declares without assembling anyone, because a rule requiring a director’s approval adds the director’s response time to every recovery, at the hour when response times are worst. It should not be the repair lead, who holds the estimate — and the estimate is precisely what corrupts the decision.

The criteria work best as observable conditions rather than descriptions of severity. The primary site does not answer from two independent external vantage points. The database has no writable primary. Storage has lost quorum and has not regained it. Each is checkable in a minute, and none requires agreement about how bad things are.

Then the part that does the real work: an elapsed-time threshold. If the service has not been restored by a stated point measured from the start of the incident, the failover is declared regardless of what the current repair estimate says. The threshold converts the decision from a judgement into a default, and the default is what survives contact with a bad night.

It matters because the two ways of being wrong are not comparable. Declare early and the cost is a failback: planned, scheduled, rehearsable, taken at a time of your choosing. Declare late and the cost is the whole duration of the failover added on top of the time already spent, every minute of it inside the outage. The late error is nonetheless the common one, because of how the decision presents itself in the room.

It presents itself as a repair estimate that keeps renewing. Twenty more minutes; then, having learned something, twenty more. Each renewal is honestly given by people who now understand the fault better than they did, which is exactly why the next one is believed. Nobody decides to wait ninety minutes. Ninety minutes is assembled out of four decisions to wait twenty, none of them wrong on its own, and the clock exists because that sequence has no stopping point.

Pre-flight: four questions and the state as found

Once the declaration is made, the pre-flight is short, bounded and not a second investigation. It asks four questions, and it asks them because the answer to any of them can change the plan.

The first is which recovery point, and how old it is. That is the actual data loss this recovery will take, expressed as a time and a set of transactions rather than as a target from a planning document, and the data owner has to accept it on the record. The second is do we have the material: passphrases, the offline kit, credentials and the licence entitlements the previous lesson enumerated. The third is is the target ready: capacity present, dependencies reachable, external allowlists carrying the recovery addresses. The fourth is what is in scope: which services are failing over, in which tier order, and which stay down so their capacity goes somewhere that matters more.

Before anything is touched, capture the state as found — replication lag on each replica, the last archived segment or snapshot identifier the recovery site received, which nodes answer, the published time-to-live on the names about to change, the position of every queue. It costs two minutes now and is unreproducible afterwards, because recovering destroys the evidence of what the estate looked like before it. Failback needs it, the post-incident review needs it, and in a compromise the investigators need it.

Fencing: the primary must be unable to accept a write

Fencing separates a failover from an accident, and it is the step most likely to be skipped, because at the moment it is due the primary looks dead and the work looks redundant.

The PostgreSQL documentation states the hazard. If the primary fails, the standby is promoted and the old primary then restarts, “you must have a mechanism for informing the old primary that it is no longer the primary. This is sometimes known as STONITH (Shoot The Other Node In The Head), which is necessary to avoid situations where both systems think they are the primary, which will lead to confusion and ultimately data loss.” The clause that matters is and then the old primary restarts: the dangerous state is not the primary being down but the primary coming back, the likeliest event in any incident because half the estate is trying to make it happen.

The distinction to hold onto is that unreachable is not stopped. A partition and a dead site produce identical evidence from outside: nothing answers. In the partition case the primary is still running, still writing, and still serving clients inside its own network — batch jobs, internal services, a scheduler — that never traverse the path that failed.

So a fence is a positive act that produces evidence, never an inference drawn from silence. Powering the machine off through out-of-band management and reading back the power state is a fence; so is withdrawing the route, disabling the switch port, revoking the storage initiator, deleting the replication slot or disabling the service accounts at the identity provider — provided the effect can be read back. “We cannot ping it” is not; it is what a broken monitoring path also produces.

Two properties decide whether a proposed fence is worth anything. It must be enforced from outside the primary, since a site you cannot reach cannot be asked to isolate itself; and it must survive that host rebooting, which stopping a service does not and a revoked credential, a withdrawn route or a disabled port does.

The dependency-ordered start and the gate at each tier

The order in which the estate comes back is not a matter of taste and not a generic checklist; it is the recovery dependency graph, read from the bottom. Every service starts after the things it needs at start-up, a stricter relation than the things it needs in steady state, because a process that cannot reach its issuer, resolver or database at the instant it starts often fails permanently rather than retrying.

The shape is usually recognisable: identity and secrets, then network reachability and name resolution, then storage, the data tier, messaging, application services, and finally the edge that admits traffic. What makes it a plan rather than a diagram is the gate between tiers. A gate is a check executed before the next tier starts, asking whether this tier can answer the question the next one is about to ask — not whether its processes are running. The two diverge constantly: a database that has started and is still replaying, a broker that is up and has not loaded its durable state, a certificate authority that is running and cannot yet sign.

Gates are worth the wait because of how failures propagate upward. A data tier that is not ready produces, three tiers later, a symptom that looks like an application bug, and the team debugs in the wrong place for forty minutes before anyone looks down the stack. The gate converts that into a two-minute wait at the correct layer. Each gate carries a deadline, and when one fails the choice — wait, skip the dependent tier, or abort — belongs to the declaration authority, not to the engineer at the keyboard.

TIER=data
ENDPOINT=https://orders.dr.example.net/healthz
DEADLINE=$(( $(date +%s) + 300 ))

until curl -fsS --max-time 5 "$ENDPOINT" > /dev/null; do
  if [ "$(date +%s)" -ge "$DEADLINE" ]; then
    echo "gate ${TIER}: not ready within the deadline" >&2
    exit 1
  fi
  sleep 5
done

echo "gate ${TIER}: passed"

Where the graph shows two branches with no edge between them, start them in parallel. Serialising independent work spends the only resource the incident is actually short of.

Validating from a client position, and the business check

Every health check that runs inside the recovery site is answered by the recovery site. It proves the tier is alive to itself, which is genuine and insufficient: the client is somewhere else and reaches the service through machinery you do not operate — a resolver holding its own cached answers, a path across other people’s networks, a certificate chain validated against a trust store you did not configure.

So the acceptance check runs from a client position outside the estate — an external prober, a connection from a different provider, a device on a mobile network — and exercises the whole path rather than an endpoint chosen for being easy to reach. This is synthetic monitoring’s discipline applied at the moment it is load-bearing.

Then, separately, the business check. Services starting is an infrastructure result; what the organisation needs to know is whether a transaction a real customer would perform completes correctly, end to end. That means an invariant recorded while the data was still known to be correct — a total, a count, a reconciled balance — compared against the recovered system, with the data owner signing off rather than the recovery engineer. The comparison also produces the number the incident report needs: the actual recovery point, as the transactions that are missing.

The point of no return, and the record failback depends on

At some identifiable moment the recovery site accepts a write the primary does not hold and never will. Before it, stopping is an abort: nothing has diverged, and the estate goes back by undoing the steps taken. After it, stopping is a failback — a separate operation with its own data movement, outage window and risk, planned later and executed deliberately. The team needs to know at all times which side of that line it is standing on.

Other steps are irreversible in the same practical sense and usually come earlier. Promoting a replica is one: the PostgreSQL documentation describes what follows as a degenerate state in which “a standby server must be recreated, either on the former primary system when it comes up, or on a third, possibly new, system”. Issuing a certificate at the recovery site, rotating a credential, asking a partner to change an allowlist entry, letting an external party consume recovery-site output — each commits something outside your control. The runbook should name these steps and announce them as they are reached, because a team that crossed the line silently keeps proposing to “just roll it back”.

Which brings the record. Failback is a reconciliation, and its only input is knowing which site was authoritative for which data at which time. Without the timestamp of the first write accepted at the recovery site you cannot tell whether a row on the old primary is stale state to discard or a write that arrived through a path nobody fenced. So a scribe who is not typing commands — the people typing cannot also observe — keeps one authoritative log in UTC: who declared and when; the recovery point, by identifier; what was fenced, by which mechanism, and the evidence it took; the time each gate passed; that first accepted write; every manual change outside automation; every external contact.

Communication runs on a cadence rather than on events: a fixed interval chosen in advance, an update whether or not there is news, and every update naming the time of the next. Silence is filled by speculation and by people asking for status, and answering them individually consumes the responders faster than the incident does. Keep the broadcast channel separate from the working one, put a named person on the broadcast, report facts rather than estimates, and confirm the channel does not authenticate through something currently being recovered.

Production discipline

1. Give the declaration a named owner and an elapsed-time threshold. One role with standing authority and ordered deputies, reachable out of band, held by someone other than the repair lead, and a stated point at which the failover is declared regardless of the current repair estimate.
2. Fence the primary positively before any write is accepted elsewhere. Unreachable is not stopped; a fence is an action with a readback, enforced from outside the failing host and surviving its reboot, because the hazard is that “both systems think they are the primary, which will lead to confusion and ultimately data loss”.
3. Start in dependency order and gate every tier before the next begins. The gate asks whether the tier can answer what the next tier will ask, it carries a deadline, and a failed gate escalates to the declaration authority rather than being decided at the keyboard.
4. Accept the recovery from a client position and from the business, not from the dashboard. Probe the full path from outside the estate, then have the data owner accept a transaction and an invariant recorded before the incident, and write down the actual recovery point.
5. Name the point of no return and keep one timestamped record. After the recovery site’s first accepted write, returning is a failback; a scribe logs the declaration, the recovery point, the fence evidence, each gate, that write, and every external contact.

Cross-course references

• Kubernetes for Production Sysadmins — Part LXVII (etcd Quorum) is the concrete mechanism behind this lesson’s claim that a node cannot fence itself: a minority unable to see a majority must refuse to serve, which is the protection external fencing substitutes for where no quorum exists.
• Kubernetes for Production Sysadmins — Part XCVIII (Disaster Recovery) and Part XCIX (Complete Cluster Loss) turn the dependency-ordered start and the tier gates described here into a specific sequence for a cluster, where the control plane must answer before any workload that depends on it is admitted.
• Observability for Production Sysadmins — Part LXIII (Synthetic Monitoring) is the steady-state form of the client-position check required here before a failover is accepted, and Part LXII (Business Metrics) supplies the signals that separate a service that has started from one the business is receiving value from.

Quiz