Failback: the second outage

Everything the recovery site wrote while it was carrying production, which the previous lesson spent its length cataloguing, is the reason this one is difficult. Failover moved the service. It did not move the primary, which is still sitting there holding the exact state it held at the instant it stopped being asked questions, and from that instant the two sides have been writing independently. Failback is the operation that reconciles them, and it is not the failover run backwards. It is the half of disaster recovery almost nobody rehearses, and the second incident of an outage tends to happen here — days or weeks after the first one was declared over.

flowchart LR
    D["Recorded divergence point"] --> I["Inventory writes on both sites"] --> A{"Authoritative state per dataset"}
    A -->|"Recovery site"| Reverse["Reverse replication or full transfer"]
    A -->|"Primary site"| Preserve["Preserve primary and rebuild derived state"]
    A -->|"Both wrote"| Merge["Business-led semantic merge"]
    Reverse --> Validate["Validate data, dependencies and business transaction"]
    Preserve --> Validate
    Merge --> Validate
    Validate --> FenceDR["Fence recovery-site writers"] --> Switch["Move traffic and write authority"] --> Retain["Retain rollback copy and observe"]

There is no safe arrow directly from divergence to traffic switch. The authoritative-side decision and the validation gate are what keep failback from becoming the second data-loss event.

The primary holds an old state and the recovery site holds the new one

At the moment of failover the two sides shared a history. Whatever mechanism kept the recovery site current — streaming replication, snapshot shipping, periodic restore, a repository both could read — ended at a point both copies contained. That is the divergence point, and it is the last thing the two sides agree about. Afterwards the recovery site accepted writes and the primary did not, so the primary is not corrupt or wrong; it is a coherent picture of a world that has moved on.

That is the easy version. The harder one is that the primary frequently writes something too. A failover declared while the primary was still partially alive leaves it serving clients that never noticed the change of address, and a primary that comes back on its own hours later — a hypervisor restarts a guest under its high-availability policy — resumes serving whoever can still reach it. In both cases the copies are not old and new but independently new, and reconciling them is a merge rather than a catch-up.

Application data is only the most visible axis of divergence. The primary also holds the configuration, package state, firewall rules, certificates, service account credentials and schema of the moment it failed, each of which has moved at the recovery site — some deliberately, some under pressure during the incident, in changes nobody has yet written down.

So the first artefact of a failback is not a runbook step. It is an inventory of datasets and estate state in which each row records what has written to it since the divergence point, on which side, and how that is known. Rows answered with “we assume nothing did” are the rows the second outage comes from.

Choosing the authoritative side is a decision, not a discovery

For each row of that inventory there are exactly three possible answers, and picking among them is a judgement call belonging to the business rather than to whoever is holding the console.

The recovery site is authoritative for most rows, which is the case people imagine when they say failback: its copy is newer, the primary’s copy is discarded, and the work is transfer plus verification. The primary is authoritative for a smaller set — datasets the recovery site never carried, ones whose recovery-site copy was knowingly degraded to save time, derived stores that will be rebuilt anyway. The third answer, that both sides wrote and neither can be thrown away, is the expensive one, and its cost is not proportional to the volume of data.

Merging is expensive because the obstacles are semantic. Identifiers allocated independently on both sides collide, so two records carry the same key and mean different things. The same logical record edited on both sides has no natural winner, and the timestamps that look like a tiebreak came from clocks nobody was comparing. A deletion on one side is indistinguishable from a record that never existed on the other, so a naive union quietly resurrects things. And the side effects have already left the building: an invoice was emailed, a payment was captured, a settlement file was shipped to a partner. Reconciling a database un-sends none of that, so part of the merge is not a data operation at all but apologies, credits and manual corrections in somebody else’s system.

“Last write wins” is therefore a decision with consequences, not the absence of one, and whichever rule is chosen the discard deserves the sign-off that restoring over live production would get. Make these choices in the failback plan, written while the estate is healthy, rather than on a bridge call at two in the morning.

Transferring the delta when there may be no shared history

Once each row has an authoritative side, moving the delta takes one of three shapes, and which one you get was decided months earlier by a retention policy.

The cheapest shape is to resume replication in the reverse direction, possible only while a provable common ancestor still exists on both sides. The OpenZFS documentation states the constraint plainly for incremental streams: the incremental source must be an earlier snapshot in the destination’s history. The same idea appears at the platform layer — the Proxmox VE storage replication documentation notes that replication uses snapshots to minimise traffic, so new data is sent only incrementally after the initial full sync, and that the replication direction switches automatically when a guest is migrated to the replication target node. Direction there follows the guest, which is exactly the property a failback wants.

COMMON=tank/appdata@failover-2026-08-14T09-12Z
FINAL=tank/appdata@failback-freeze

# describe what the incremental stream would contain, without sending it
zfs send -nvP -i "$COMMON" "$FINAL"

The second shape repairs the diverged copy in place, and PostgreSQL supplies the canonical example. pg_rewind synchronises a data directory with another that forked from it by examining the timeline histories of both clusters to find where they diverged, then copying only the changed data blocks along with new relation files, configuration files, WAL segments and pg_xact files — which the documentation notes is significantly faster than the alternatives when the database is large and only a small fraction of blocks differ. It is equally explicit about the price of admission: the target must have either wal_log_hints enabled or data checksums enabled at initdb time, full_page_writes must be on, and the target must be shut down cleanly first.

TARGET_PGDATA=/var/lib/postgresql/18/main
SOURCE_CONN='host=dr-db-01 port=5432 user=rewind dbname=postgres'

# the old primary must already be shut down cleanly
pg_rewind --target-pgdata="$TARGET_PGDATA" \
  --source-server="$SOURCE_CONN" \
  --dry-run --progress

The third shape is to discard the primary’s copy entirely and seed it in full from the recovery site. This always works, costs the size of the data rather than the size of the change, and is what remains when the first two are unavailable — which happens silently: if the primary’s retention expires the last common snapshot while the outage runs, or its pg_wal stops reaching back to the divergence point, the cheap options vanish without anyone being told.

The second cutover has a window nobody budgeted

The transfer ends at a cutover, and the cutover is an outage. Its shape is the same as the first one: freeze writes at the recovery site, move the final increment, verify that the two sides now agree, switch the traffic, validate. Every dependency from the failover lessons returns unchanged — resolver caches and their time-to-live, certificates that must now be valid at the primary too, connection pools holding their endpoints, and third-party allowlists finally updated to accept the recovery site that must now accept the primary as well, without dropping the entry just proven useful.

Two things make this cutover different and they pull in opposite directions. It is planned, which is a genuine advantage: it can be scheduled, staged and practised. But it is also the second cutover, which produces false confidence, because the team believes it did this a month ago. It did not. It performed the opposite operation, with the data flowing the other way, different preconditions to satisfy beforehand and a different set of systems to inform.

The window has its own arithmetic. The delta grows for as long as the decision is deferred, so a failback postponed by four weeks moves four weeks of writes, and the standard migration technique applies: send increments repeatedly while the recovery site is still serving, so only the last small one falls inside the freeze. What must not be carried over are the plan’s numbers, engineered for an unplanned failover from a stated architecture. This cutover has its own achievable loss window, set by how far behind the primary is when writes stop, and its own duration, set by the final increment plus validation.

Rebuild rather than resume

Two conditions make booting the primary as it was left the wrong move, however convenient.

The first is compromise. If the outage was caused by an intrusion, or merely coincided with one, the primary is not a machine that stopped. It is a machine an adversary had access to before it stopped and then had entirely unsupervised access to for the duration of the incident, while every pair of eyes in the organisation was pointed at the recovery site. Its state is evidence of what the adversary chose to leave, and nothing else. Discarding its application data in favour of the recovery site’s copy resolves the divergence and does nothing about the host, so the decision here is about the machine, not the dataset.

The second is staleness sharp enough to be dangerous on its own. A primary dark for five weeks holds credentials that have since been rotated, certificates that may have expired, firewall rules predating the incident’s changes, package state that missed a security update cycle, and configuration that has diverged from what the recovery site runs. Starting it and pointing traffic at it is an implicit rollback of every change made since the failover — including the emergency ones, made under pressure, that nobody documented and that therefore cannot be reapplied from a list.

The usable rule is short: resume the primary only if the differences between it and the current estate can be enumerated, and rebuild it whenever they cannot. The rebuild is not exotic work — it is the reconstruction path this course already describes, run against a site that happens to be racked already — and a rebuilt primary is a fresh, current, empty site, which turns the failback into an ordinary migration.

Exhaustion, reduced urgency and the plan nobody wrote

Look at the conditions under which a failback is actually executed. The incident has been declared over, so the bridge call has dissolved and incident command has stood down. The executive attention that authorised bold action last week has moved elsewhere. The people who ran the failover are tired, on leave, or catching up on the work that stopped. And nothing is currently broken: the recovery site is serving customers. Every organisational property that made the failover work is absent, by construction, at the moment the harder operation begins.

The countermeasures are unglamorous. Failback is its own change, with its own written plan produced before the disaster rather than during it, its own scheduled window, its own named approver and its own go/no-go criteria agreed in advance. Its rollback is to remain at the recovery site — an outcome that must be genuinely acceptable, or the go/no-go is theatre. Its validation validates the primary rather than the plan, against what the recovery site has been proving daily: the service works, backups run there again, and the estate can still fail over.

Naming the actual reason for failing back sets how much risk the second cutover is worth. If the recovery site is meeting its service levels the pressure is rarely availability; it is cost, capacity, licence terms, data residency, or a policy about where production lives. Those are real reasons and negotiable ones, which leaves an option that is often right and almost never on the table: declare the recovery site the primary, build the next recovery site elsewhere, and convert an outage into a documentation change.

Production discipline

1. Write the failback plan before the failover, as its own document with its own approver. It is a different operation with different preconditions, and the one thing you can rely on when it is needed is that nobody will be in a state to design it from scratch.
2. Inventory every dataset, record its authoritative side, and have the discard signed off by name. Three answers are possible per row — recovery site, primary, or both wrote and must be merged — and the merge case costs apologies and manual corrections in other people’s systems, not disk.
3. Protect the state that anchors the divergence point. An incremental send requires the source to be an earlier snapshot in the destination’s history, and pg_rewind requires wal_log_hints or data checksums, full_page_writes on, and a cleanly shut-down target; without them the only remaining shape is a full seed.
4. Treat the second cutover as an outage and compute its own window, loss window and duration. The disaster recovery plan’s numbers were engineered for the unplanned failover; this one is bounded by the final increment and the validation, and the delta grows for every week the decision is deferred.
5. Rebuild the primary whenever the differences cannot be enumerated. Compromise makes its state evidence of the adversary’s choices rather than yours, and weeks of staleness make resuming it an implicit rollback of every undocumented change made during the incident.

Cross-course references

• PostgreSQL for Production Sysadmins — Part XV (High Availability, Failover and Disaster Recovery) works through the promoted standby and the demoted old primary, which are exactly the two divergent copies this lesson reconciles, and Part XIV (Replication, Slots and Read Replicas) builds the streaming relationship whose direction a failback must reverse; both supply the database-level mechanics the pg_rewind discussion here assumes.
• Proxmox VE for Production Operators — Part XIV (Disaster Recovery) covers bringing guests up at a second site, and Part XXI (Migration to Proxmox) supplies the staged-increment migration technique borrowed here for the second cutover; together they are the platform-level expression of this lesson’s claim that a failback is a migration rather than a return.
• Linux for Production Sysadmins — Part XLIX (Restore) and Part L (Disaster Recovery) build the host-level procedures that “rebuild rather than resume” falls back on, which is what makes that decision affordable: rebuilding a primary is only reasonable where reconstruction is already practised.

Quiz