Active/passive and active/active for disaster recovery

Cold, warm and hot described how much of a recovery estate has already been built, and two of those three temperatures imply a second site that is already holding data. The moment a second site holds data, a question appears that the temperature vocabulary cannot answer, and it decides almost everything downstream: which site is allowed to accept a write. Answer one and you have an active/passive estate whose recovery is a discrete, rehearsable cutover. Answer both and you have an active/active estate that has traded the cutover for a reconciliation problem which never stops running.

Which site is allowed to say yes

Strip away the diagrams and a two-site topology is a rule about write authority. Active/passive says that at any instant exactly one site accepts writes for a given dataset, and everything else in the design exists to maintain that invariant: the standby applies a stream and serves nothing, or serves reads only; the traffic layer points at one place; the promotion sequence is guarded so that it cannot run twice. Recovery is then a named event, with a trigger, a decision-maker, an ordered sequence, an end state and a verification step — exactly the shape of a thing you can write down, rehearse and time. It is also disruptive and infrequent, and therefore practised rarely.

Active/active says that both sites accept writes concurrently for the same logical dataset. There is no cutover, because there is no authority to transfer; losing a site is a withdrawal of capacity rather than a change of role, and that is genuinely the attraction. What has actually happened, though, is that an invariant previously maintained by construction now has to be maintained by continuous agreement between two systems that can stop being able to talk to each other. The problem did not disappear with the cutover; it moved from an incident-time procedure into a steady-state property that runs every second of every day and fails in ways that look like ordinary operation until they do not.

Three arrangements get sold under the same phrase, and they are not equivalent. The first partitions write authority by key: tenants, regions or shards are each owned by exactly one site, and no record ever has two writers. Both sites are busy, so the estate is active/active at the service level, while every individual dataset remains active/passive — and this is the most common honest answer, because it keeps the single-authority invariant while using both buildings. The second uses synchronous agreement across sites, where a write is not acknowledged until enough members agree to record it. The invariant survives, and the price is paid on every write in latency that no product can optimise below the round-trip time between the sites. The third is asynchronous multi-master: both sites acknowledge locally and exchange changes afterwards. It is the only one of the three that delivers local write latency at both sites, and it is the only one that creates conflicts, because two writers can modify the same record before either has heard about the other.

Which of the three you have determines what the rest of this lesson does to you. A team that believes it runs the first while actually running the third has a conflict policy it has never read.

Split brain, and the arbiter that is supposed to prevent it

A network partition is the case where the sites cannot reach each other but can still reach clients. Each side observes the same thing — silence from the other — and silence is indistinguishable from death. If both sides resolve that ambiguity in their own favour, both keep serving, both accept writes, and the estate holds two histories that cannot in general be reconciled, because each contains committed business facts the other never saw.

Note that this is not exclusive to active/active. An active/passive pair with automatic failover has the same hazard: the standby concludes the primary is gone, promotes itself, and the primary — perfectly healthy, merely unreachable from the standby — keeps accepting writes from clients on its own side of the partition. The invariant was single authority; the failure mode is that nothing in the promotion path was able to enforce it. Split brain is therefore a property of any topology where a second component can decide to become authoritative, and the only defence is that the decision is not taken locally.

Two mechanisms convert a local observation into an authorised one, and they answer different questions. Quorum answers “am I allowed to proceed?” by requiring agreement from a majority of a defined membership. Ceph’s monitor cluster is the familiar illustration: the monitors maintain the cluster maps through a consensus protocol and require a majority to be available for the cluster to be usable, which is why monitor counts are chosen with the majority arithmetic in mind. The etcd documentation makes the same arithmetic explicit for its own membership, recommending odd cluster sizes because an even member adds a vote without improving how many failures the cluster tolerates. Fencing answers a different question — “is the other one definitely stopped?” — by taking an action against the loser rather than a vote among the winners. The PostgreSQL documentation describes STONITH, shoot the other node in the head, as the approach for making certain that a failed-over-from server is not still running and writing.

The two are complementary rather than alternative. Quorum stops a minority from proceeding, but a node in the minority that is wedged, paused or partitioned from the arbiter may not learn that it lost in time to stop what it is already doing. Fencing closes that gap by removing the loser’s ability to act, which is why estates that take this seriously run both: a vote to decide, and an action to enforce the decision.

What each site accepted and had not yet replicated

Topology decides where the loss window comes from, and the arithmetic is different in the two cases.

In an active/passive pair with asynchronous replication, the data at risk is whatever the primary acknowledged to clients and had not yet shipped when it disappeared. That quantity is the replication lag at the instant of loss. It is not a schedule, and treating it as one is the most common error in this area: nightly backups have a fixed worst case set by the interval, while a replication stream has a lag that moves continuously with write rate, network conditions and how fast the standby can apply what it receives. The honest statement is therefore distributional — the measured lag over some observed period, with its tail called out, because the tail is what a batch job at 02:00 produces and the tail is what you will be standing in when the site fails. Under synchronous replication the acknowledged-but-unshipped set is empty by construction, which is what that mode is bought for; the costs are the per-write round trip and the fact that losing the standby stalls the primary unless the mode is allowed to degrade, and a mode that degrades silently has reinstated the window without telling anyone.

In active/active the same arithmetic runs at both ends, and the consequence is qualitatively worse. Losing site B loses everything B had accepted and not yet replicated to A. Those are not the tail of one stream that A was already following; they are a distinct set of committed facts that only B ever knew — orders taken, payments authorised, records amended — belonging to whichever clients happened to be routed to B. A cannot enumerate them, because the only record of their existence was at B. The conversation with the business changes shape accordingly. It is no longer “we lost the last stretch of one history”; it is “an unknown subset of customers had work accepted and lost, and we cannot tell you which ones without reconstructing it from somewhere outside the database”.

Conflicts are the same problem when both sites survive. If the topology allows two writers for one record, something has to decide which version stands. Last-write-wins by timestamp decides it by discarding one of them, and clock behaviour across the two sites decides which — so resolved and correct are different words, and the difference is data a customer entered. Merge strategies that keep both versions exist and work, but they require the data model to have been designed for merging rather than retrofitted. Choosing multi-master and leaving the conflict policy at its default is choosing a data-loss policy without reading it.

Both sites apply the same bad migration

Active/active is an availability architecture. It answers the loss of a site, the loss of capacity, a regional network event, a maintenance window that would otherwise be an outage. Those are real and worth money.

What it cannot answer is any fault that arrives through the write path, and the write path is exactly what it replicates. A migration that drops the wrong column, an accidental mass delete, an application defect writing wrong values, an operator whose credentials work at both ends, ransomware encrypting through a mounted share — every one of these is a legitimate write as far as replication is concerned, and every one arrives at the other site at the speed the design was tuned for. The stronger the consistency guarantee, the faster the mistake propagates: synchronous agreement delivers it to both sites before the transaction is even acknowledged.

This is the earlier-state test from the opening lesson of this part, applied to a topology instead of a product. Neither site holds a state from before the mistake. Both hold the current one, which is the mistake. Recovery from a logical fault requires a copy that is deliberately behind — a point-in-time recovery target, a snapshot chain, a repository under object-lock retention for a defined window — and that copy is a separate line item that no topology includes. Running two live sites does not reduce the need for it. It raises it, because a second site widens the blast radius of one bad statement and adds a second control plane, a second credential set and a second place from which such a statement can be issued.

Choosing which cost you would rather carry

Both topologies cost something; they differ in when the bill arrives. Active/passive concentrates its cost into a rare event: a cutover with a duration, a decision-maker, a rehearsal budget and a real probability of going wrong, plus the failback afterwards — which is a second cutover complicated by the fact that the old primary’s data diverged while it was down, so it usually has to be rebuilt from the current authority rather than simply restarted. Active/active spreads its cost across every day: a permanent consistency obligation, latency or conflicts depending on which arrangement you chose, a doubled change surface, and an arbiter whose correctness has to be maintained by people who will not get feedback if they break it.

Re-admitting a site has its own trap in both topologies. A site that was down holds stale state, and a traffic layer that restores it on a health check which only asks whether processes are listening will serve old data to real users. Admission has to depend on a currency check — the site is caught up to within a stated bound — not on liveness.

None of this argues for one topology. It argues that the choice is a statement about which failure you are buying down and at what recurring price, and that the statement should be written where the people running the estate can read it.

Production discipline

1. Name the write authority for every dataset, in writing. For each dataset record which site may accept writes, what changes that answer, and which of the three arrangements — partitioned keys, synchronous agreement, asynchronous multi-master — is actually in force. A team that cannot answer this per dataset does not yet know whether it has conflicts.
2. Rehearse the partition, and confirm the fence. Drill the case where both sites are up and cannot see each other, not the case where one is shut down cleanly. Require the fencing action to return a confirmed result, and treat an arbiter that has never run under a real partition as an untested hypothesis in the risk register.
3. Place the tiebreaker in a third failure domain and prove it. Check power, network and hosting independently, then remove each site in turn and confirm the survivor still reaches a majority. A witness that shares a fate with one site converts a site loss into a total outage.
4. State the loss window as a measured lag distribution. Publish the observed replication lag with its tail and the conditions that produce the tail, not a headline figure, and say plainly that in an active/active pair the loss is whatever the vanished site accepted and only it recorded.
5. Fund an earlier state independently of the topology. Keep recovery points that are deliberately behind both sites, reachable through a path the production write path cannot reach, and verify them on their own cadence. Availability architecture and recovery capability are separate budget lines because they answer separate failures.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part IX (Monitor Quorum) develops the majority arithmetic this lesson depends on in a system where you can watch it operate, which is the concrete counterpart to the counting argument above about why two symmetric sites cannot arbitrate themselves without a third failure domain.
• PostgreSQL for Production Sysadmins — Part XIV (Replication, Slots and Read Replicas) is where the replication lag that this lesson calls the real loss window becomes something you can measure rather than assume, and its material on slots explains what happens to the stream when the other end is absent for a long time.
• PostgreSQL for Production Sysadmins — Part XV (High Availability, Failover and Disaster Recovery) covers promotion, fencing and the failback problem at the level of one database, and is the practical answer to this lesson’s claim that an old primary usually has to be rebuilt from the current authority rather than restarted.

Quiz