Operating in DR, and the data you create there

The failover procedure ended when traffic arrived and the services answered, which is where the previous lesson stopped. What begins at that moment has no procedure attached to it: an open-ended period during which the whole estate runs on infrastructure designed, sized and rehearsed as a destination rather than as a residence. The incident stops moving, attention drains toward the postmortem, and the estate quietly starts doing the one thing nobody planned for, which is creating data.

The recovery estate was validated as a destination, not as a residence

How long the estate stays at the recovery site is almost never a decision the team that failed over gets to make. Restoration of the primary waits on power, on hardware lead times, on a cloud provider’s own recovery, on an insurer, or — after a ransomware event — on a forensic hold that forbids touching the primary at all. NIST SP 800-34 separates reconstitution from recovery for exactly this reason: returning to the original site is a distinct phase with its own preconditions, and those preconditions frequently sit outside the organisation. The honest planning assumption is not “a few hours” but “until further notice”.

Everything ever proved about the recovery site was proved under a different question. The rehearsals asked whether the team could get there, whether the data arrived and whether the services started. Very few asked whether the site could sustain a month of production: absorb its own growth, survive its own component failures, take patches, be changed safely, and protect what it produces. The AWS guidance frames backup-and-restore, pilot light, warm standby and multi-site as a trade-off in what is already running before the event; none describes how the chosen site behaves as a home.

Two clocks are running. The incident clock is short and has an audience; the DR-duration clock starts when it stops, runs for as long as the primary is gone, and has nobody assigned to it. Every failure in this lesson lands on the second one, by which time the recovery site holds the only current copy of the running state and its own single points of failure have been promoted into the estate’s.

Data created at the recovery site is production data with no backup

During the failover, data moved one way — out of backups, replicas and snapshots into the recovery site. The instant services answer, the direction reverses. Orders, tickets, uploaded documents, new accounts, audit records, write-ahead log, queue contents, certificates issued because the originals were unreachable, secrets rotated because the old ones were exposed, and the incident’s own evidence: all of it is now written at the recovery site and nowhere else.

The protection did not travel with it. Backup coverage is not a property of data; it is a configuration attached to a location, and every part of that configuration stayed behind. The schedule was a timer on a primary host; the write credentials were issued to primary identities; the repository endpoint may only accept connections from primary networks. The recovery-site machines were never backup sources at all — they were backup destinations, a different role with different plumbing.

The consequence is arithmetic on a stated architecture, not a property of any product. If exactly one copy of the post-failover data exists, the loss window for that data is the elapsed time since failover, and it grows at one second per second, without bound, until a second copy exists somewhere the recovery site did not create. No tool changes that number; only a backup does.

This is the single most common way an incident produces permanent loss, and its shape is consistent. The original disaster usually destroys nothing irreplaceable, because replacing it is what the backups were for. The permanent loss arrives with a second, much smaller event days later: a volume that fills, an instance an autoscaler terminates, a mistaken deletion by a tired operator, a hypervisor host that fails at the recovery site. Every one is ordinary and survivable anywhere else in the estate. None is survivable against data that has one copy.

So the rule is a sequencing rule, and it is deliberately uncomfortable: backups at the recovery site are the first thing established once services are up, before anything else is optimised. Before performance tuning, before re-enabling deferred batch work, before standing the lower tiers back up, before the timeline. Everything on that list can be done later at its ordinary cost; the backup is the only item whose absence is converted into permanent loss by events outside your control.

“Established” carries four requirements, and dropping any leaves the window open. The target must be writable from the recovery site using credentials that exist there. It must not sit inside the recovery site’s own failure domain, since a copy that dies with the original is not a second copy. A first backup must be taken immediately rather than scheduled for the following night. And one file must be restored out of it, because this course’s entire argument is that an unverified backup is a claim rather than a capability.

Degraded capacity is a set of decisions, not a condition

A warm site is normally sized for a survivable subset rather than the whole estate, which is the trade-off the AWS guidance describes as the point of the pattern. Undersizing is a purchased position, not a fault. What matters is that the shortfall is not a condition to be endured: it presents as a queue of decisions, each a change failback will have to reverse.

The decisions are recognisable in advance. Which tiers run and which stay down. Which batch and reporting work is suspended, and whether its inputs are queued or discarded. How many replicas each component keeps, and therefore how much redundancy remains inside the site. Which timeouts and concurrency limits are lowered to keep the site inside its capacity. None of these should be invented under pressure by whoever is awake; they belong in the service tier register, decided while the business can argue about them.

Two failure modes bracket this. Deferring the decisions leaves a site running the whole estate on a fraction of the hardware, and the failover gets judged a failure for what was really a sizing choice. Taking them and not recording them is worse: a timer disabled at 03:00 becomes invisible configuration, found six weeks later by someone wondering why a report stopped arriving.

Backup is the one item that must never be shed. In capacity terms it is among the cheapest things the site does, and the only one whose absence cannot be repaid afterwards. A compounding effect argues the same way: reduced redundancy raises the probability that the recovery site suffers its own component failure, and that elevated probability applies at exactly the moment the data has one copy. The two risks multiply rather than add.

Monitoring at the recovery site is a third thing that may not exist

The observability platform is an estate of its own, and it fails over — or fails to — like any other. Three conditions get confused and need different remedies. In the first, the telemetry pipeline was in the primary and is simply gone. In the second, a platform exists at the recovery site but its scrape targets, log sources and dashboards still name primary hosts, so it is healthy and blind. In the third, it collects correctly but every threshold was calibrated against the primary’s capacity, so a deliberately smaller site emits a stream of alerts that are technically true and operationally useless.

The third matters because of what people do about it. A firehose of capacity alerts gets silenced, silencing is coarse, and the silence outlives its reason; from then on the estate runs abnormally with its alerting off, which is the worst combination available. Silence deliberately instead: narrow the scope to the signals expected to be wrong while degraded, attach an expiry to every silence, and re-evaluate at a stated time.

One alert must survive all of this. Restore-point age at the recovery site — the elapsed time since the last verified backup taken there — is the only signal that reports the condition this lesson is about, and on a newly stood-up site it usually does not exist yet. Its most important branch is the one that fires when nothing has ever run.

DR_HOST=dr-app-01
MARKER=/srv/dr-backup/app/last-verified

if ! LAST=$(ssh "$DR_HOST" stat -c %Y "$MARKER" 2>/dev/null); then
  echo "CRITICAL: the recovery site has produced no verified backup at all"
  exit 2
fi

AGE_HOURS=$(( ( $(date +%s) - LAST ) / 3600 ))
echo "restore point age at the recovery site: ${AGE_HOURS}h"

The minimum viable set alongside it is short: is each tier-1 service answering a request that exercises its dependencies, is any volume filling, is the backup here running and verified.

Change control while the estate is abnormal

Two opposite instincts appear here and both are wrong. The first freezes all change because the estate is fragile, which blocks the fixes a degraded site needs and pushes the work into undocumented workarounds. The second suspends change control because an incident is in progress, producing a fortnight of unrecorded modifications and turning failback into archaeology.

Change control should not stop; it should change shape. Approval gets faster and the quorum smaller, because the alternative is an estate nobody may repair. Recording gets stricter, because in normal operation the configuration repository is a reasonable record of the estate and during DR it is not. Every change gets a ledger line as it is made: what changed, where, who, when, why, and — the field that does the work later — which of three categories it falls into: reverse at failback, carry back to the primary, or keep permanently.

DR_HOST=dr-app-01
LEDGER=/srv/dr/ledger/dr-changes.tsv
STAMP=$(date -u +%Y-%m-%dT%H:%M:%SZ)

printf '%s\t%s\t%s\t%s\t%s\n' \
  "$STAMP" "$USER" "$DR_HOST" 'disabled nightly-reindex timer' 'reverse-at-failback' \
  >> "$LEDGER"

The middle category is the one teams forget. A schema migration applied here, a bug fix, a rotated credential and a firewall rule added for a partner must all reach the primary before it serves traffic again; failing back onto a system that never received them reproduces the outage in a new form.

Automation deserves its own caution. The configuration repository still describes the primary’s intended state, so a convergence run aimed at the recovery site faithfully reasserts it — putting capacity settings back, re-enabling services the team deliberately shed, or undoing part of the failover itself. The Ansible documentation describes check mode as a way to run without making changes while reporting what would have changed, paired with diff mode for the detail; that is the mode to use against a recovery estate until the repository has been taught which site is production. Read the report as a list of everything the automation believes is wrong, then decide which of those beliefs holds today.

Drift, and why failback gets more expensive every day

Drift accumulates in four places at once. Data diverges because the site keeps producing it while the primary holds only what existed at the moment of failure. Configuration diverges through the degradation decisions and the operational fixes above. Infrastructure diverges as hosts are added and capacity is adjusted. Knowledge diverges fastest of all, because decisions live in the heads of whoever was on shift.

The primary, whenever it returns, is a snapshot of the instant it failed, stale by the entire DR period. That is why the next lesson can fairly call failback a second outage: a full migration in the harder direction, carrying back state that exists in one place. The backup established in the first hours is what makes that state movable at all, so its absence is felt twice — as risk during the DR period, and as an unrepeatable migration at the end.

Drift grows faster than linearly in the duration, because each change enlarges the reverse migration and raises the chance two changes conflict. Keep the reverse path warm from day one: maintain the ledger, back up the recovery site’s data, and begin replicating toward the primary as soon as it can accept a copy. Decide early which end state is intended, too, because promoting the recovery site to be the new primary removes failback entirely — provided it is chosen deliberately and the site is brought up to primary standards for capacity, redundancy, monitoring and backup, rather than arrived at by exhaustion.

Production discipline

1. Establish a backup at the recovery site before anything else is optimised. Once services answer, the next task is a backup written with recovery-site credentials to a target outside that site’s failure domain, taken immediately rather than scheduled, and proved by a restore.
2. Treat every post-failover write as data with one copy until proven otherwise. Enumerate what the site produces — application data, write-ahead log, queues, issued certificates, rotated secrets, incident evidence — and confirm each is inside the new backup.
3. Decide degraded-capacity trade-offs from the service tier register, and record each as it is taken. Suspended jobs, reduced replica counts and lowered limits are changes with a reversal owed at failback, and backup is never a candidate for shedding.
4. Keep change control running in a faster, stricter form. One ledger line per change as it is made, classified as reverse at failback, carry back, or keep permanently; and run automation in check and diff mode first, because the repository still describes the primary.
5. Stand up restore-point-age monitoring for the recovery site, and silence alerts with an expiry. The signal reporting an unbacked-up recovery site does not exist until you create it, and a blanket silence applied to stop capacity noise hides it for the whole DR period.

Cross-course references

• Observability for Production Sysadmins — Part XCII (Disaster Recovery) covers failing over the telemetry platform itself, which is the precondition for anything in this lesson being visible, and Part CX (Observability During Major Incidents) addresses the alert-quality problem met here in its degraded form, where thresholds calibrated for the primary make a correctly sized recovery site look permanently unhealthy.
• Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) describes the mechanism that will silently reassert the primary’s intended state over a recovery estate and undo the degradation decisions recorded here, and Part XLVIII (Maintenance Windows and Rollback) supplies the change discipline this lesson’s ledger adapts for an abnormal estate.
• Terraform for Production Sysadmins — Part XVII (Drift Detection and Reconciliation) is the infrastructure-level counterpart of the configuration drift described here, and Part XXVIII (Disaster Recovery and Resilience) covers rebuilding a recovery estate declaratively, which makes the reverse migration in the next lesson tractable rather than manual.

Quiz