critical riskdata loss risk~180 min

Execute a DR failback

1 · Prerequisites

Confirm every item is in place before any state change.

The failover record: the timestamp at which the recovery site began accepting writes, what was cut over, what was deliberately left behind, and who authorised it. Without that instant, nothing below can be decided — only guessed.
Write access at both sites and the authority to stop writes at one of them, held by a named person for the whole window rather than requested during it.
A verified backup of the recovery site, taken to the same standard as the primary ever had: schedule, verification and retention. The recovery site has been production since the failover, and this runbook will not move a byte until that is true.
The primary's condition established in writing by somebody who looked: intact, damaged, or compromised. "Unknown" is a fourth answer and it is handled as compromised.
Infrastructure definitions, configuration and identity material for the primary held in version control, so that rebuilding it is an option rather than an aspiration.
Restore tooling and key material usable independently at both sites, proven by opening a repository from each side before the window opens.
Control of DNS, load balancers and certificates for the service name, the current TTL, and the ability to change it well ahead of the cutover rather than during it.
A maintenance window agreed with the business, with the abort path written into the agreement: remaining in DR is one of the outcomes this window is allowed to produce.
A scratch host or namespace where the rebuilt or resumed primary can be brought up isolated, with its application stopped, before it is allowed anywhere near live traffic.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the recovery site is healthy enough to keep serving through the whole window. Failback is elective. If the recovery site is degraded, you are not choosing between two sites, you are running a second incident with a deadline, and the window should be postponed rather than compressed.
· Establish which side is authoritative for what, and write the answer down before anything moves. The primary is authoritative for nothing committed after the failover instant; the recovery site is authoritative for everything after it. Both sentences are trivial until somebody has to act on them at 02:00 without the timestamp.
· Enumerate the data created at the recovery site during the DR period, and store the enumeration somewhere that is neither site. Row counts, object keys, file paths, ticket ranges, invoice numbers. This list is cheap now and impossible to reconstruct once the two sides have been merged.
· Take a backup of the recovery site and verify it by restoring something from it. This is production data with no second home; the primary predates it by definition. A failback that begins before this check is a migration of the only copy.
· Confirm the primary's condition, and treat "we are not sure" as compromised. A host that was reachable by an attacker, or that has been powered off long enough that nobody can account for it, is rebuilt rather than resumed. This is the pre-check that decides how long the window has to be.
· Confirm the primary's application, scheduler and replication agents are stopped and masked before it is powered on. A primary that boots into its old configuration and accepts one write becomes a second authoritative side, and the estate then has two of them and no way to merge.
· Read the current DNS TTL for the service name and record when it was last changed. The TTL in force is the minimum length of the cutover tail, and it is the number most often discovered after the window has already started.
· Confirm both sites can open the backup repositories and unlock the key material independently of each other. A failback that assumes the primary will be able to reach a repository it has not touched for a week is an assumption, and this is the last cheap moment to convert it into a fact.
· Record the start time, the failover instant, the agreed window, and the named person who may call the abort. The abort is a decision, not an admission, and naming its owner in advance is what makes it available later.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm and record the failover instant before anything else happens. Every later decision — which delta moves, which records can conflict, what business validation looks like — is derived from that one timestamp and from the last transaction the primary committed. Establish both, write them in the incident record, and state them aloud to the window participants.
2Preserve the primary as you found it, before you power anything on. An image, a snapshot or a copy of the surviving disks. If the primary turns out to be compromised, that copy is evidence; if the reconciliation goes wrong, it is the only remaining witness to what the primary held at the moment it failed.
3Enumerate the data the recovery site created, and keep the enumeration off both sites. Counts and identifiers, not impressions. The enumeration is the instrument you will use twice: once to size the delta, and once after the cutover to prove the DR-era data arrived.
4Back up the recovery site and verify that backup by restoring from it. The recovery site holds hours or days of production that exists nowhere else. Protecting it is not a formality before the real work; for the next several steps it is the only thing standing between a routine change and a data-loss incident.
5Select the pre-failback recovery point explicitly, by repository, identifier and timestamp. This is the rollback anchor for the whole window, and it must be a recovery point you have listed and can name, never "the latest" and never a job that reported success. State what is in it and what is not.
6Select the recovery point that seeds the primary, deliberately, and never take the newest that exists. If the primary failed because of corruption or a compromise, the newest recovery point is the one most likely to carry it forward. The one you want is the newest whose contents predate the event, and the difference between those two is the whole reason this step is written down.
7Decision point — resume the primary, or rebuild it. The criteria are tabulated in the body. Resuming is faster and inherits everything, including whatever caused the failover; rebuilding costs hours and produces a primary whose state you can account for. The choice is authorised by a named person and recorded with the condition that settled it.
8Rebuild from infrastructure definitions rather than from the surviving disks whenever the condition is compromised, unknown, or stale enough that nobody can describe what changed. Rebuilding is not the pessimistic option here; it is the option that produces a primary somebody can vouch for, and it can run in parallel with the recovery site still serving.
9Bring the primary up isolated, with its application stopped, its timers masked and its replication agents disabled. It is a restore target at this stage and nothing else. It must not resolve the service name, must not accept client connections, and must not be reachable from anything that could send it a write.
10Transfer the bulk of the delta while the recovery site is still serving. The window only has to cover what changes after this point, and the difference between transferring a week and transferring twenty minutes is the difference between an outage and a pause.
11Run every transfer in dry-run form first and read what it proposes to delete. A mirroring transfer from the recovery site to a primary that has been reconstructed will propose deletions, and most of them are correct. Reading the list is how you find the ones that are not, while they are still a proposal.
12Reconcile the divergence as a data question, with the data owner, before the window opens. Sequences and auto-increment counters that advanced at the recovery site, identifiers issued on both sides, records changed in both places if the primary was ever writable again. Infrastructure cannot answer any of these; only the person who owns the records can.
13Lower the DNS TTL ahead of the window and confirm the old value has expired before you rely on the new one. The TTL in force when you change the record is the tail you inherit. Lowering it during the window changes nothing for clients that already hold the old answer.
14Rehearse the cutover on the isolated primary and measure it. Start the application, run the read-only checks, stop it again. The measured duration is what you commit to the business; an estimate is what you apologise for afterwards.
15Open the window by stopping writes at the recovery site, deliberately and with a recorded timestamp. Stopping writes is what makes the final delta finite. Until it happens the two sides are chasing each other, and the reconciliation you validated will not hold.
16Transfer the final delta, then re-run the enumeration on both sides and compare the numbers, not the impressions. Equal counts and matching identifiers are the pass condition. A difference is not a rounding error; it is either a record still in flight or a record that has been lost, and those are distinguished by looking rather than by waiting.
17Move the traffic, and leave the recovery site running read-only rather than shutting it down. A recovery site that is stopped and wiped at the moment of cutover removes the rollback that the next twenty minutes may need. It costs a little capacity and it buys the whole abort path.
18Validate the business at the primary, using records created during the DR period and chosen by the data owner. Not "the application starts". A named order, ticket or invoice that exists only because the recovery site served it, found at the primary afterwards, by somebody who is not you.
19Re-establish protection at the primary before declaring the failback complete: backup schedules, monitoring, replication direction and retention. The primary spent the DR period unprotected and unmonitored by definition. A failback that returns the traffic without returning the protection has moved production to the less safe of the two sites.
20Decommission nothing for the agreed hold period. The recovery site data, the pre-failback recovery point and the preserved primary image all stay until the business has confirmed a full cycle at the primary — a month-end, a reporting run, whatever the estate's longest natural period is.
21Record both timestamps, both recovery points, both decision points, the reconciliation counts, the measured cutover duration and the sign-off. The reconciliation counts are the artefact somebody will ask for in six months, and they are the one nobody writes down.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The failover instant and the write-stop instant are both in the record, to the second, and the DR period is the interval between them rather than a description like "the weekend".
✓A backup of the recovery site taken before any transfer began exists, was verified by an actual restore, and is named by repository, identifier and timestamp.
✓Both recovery points — the pre-failback anchor and the seed for the primary — are named explicitly, and the record states why the seed was not simply the newest available.
✓The enumeration of DR-period data was taken twice, before the transfer and after the cutover, and the two sets match on count and on identifiers. Differences, if any, are itemised and explained individually.
✓Sequences, counters and identifier ranges at the primary are at or beyond the highest value issued at the recovery site, confirmed by query rather than by assumption.
✓The dry-run output of every mirroring transfer was read and retained, and the deletions it proposed were accounted for before the real run.
✓The primary served no client traffic until the cutover: the service name did not resolve to it, and its application and timers were confirmed stopped while it was a restore target.
✓The measured cutover duration is recorded next to the duration promised to the business, and any gap between them has an explanation.
✓A named person from the business has confirmed, against records they chose that were created during the DR period, that those records are present and correct at the primary.
✓Backup schedules, monitoring, alerting and replication direction at the primary are confirmed running and producing evidence — a completed backup and a fired test alert — not merely enabled.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Rollback is a return to the recovery site, and it is cheap only while the recovery site is still intact and still holds the authoritative copy. Nothing there is deleted, wiped or repurposed until the hold period ends.
↶Point the service name back at the recovery site and allow writes there again, then record the second write-stop and the second resume so the DR period is a documented interval rather than a smear.
↶Any records created at the primary after the cutover are now the delta, in the opposite direction. Enumerate them before returning writes to the recovery site, because the same reconciliation problem now exists mirrored, and it is much smaller if you catch it in minutes.
↶Restore the DNS TTL to its normal value once the position is stable, and record that it was changed and restored. A TTL lowered for a window and left low is a standing cost nobody remembers agreeing to.
↶Re-enable and unmask every timer, scheduler and replication agent that was stopped at either site, from the list written down when they were stopped. A masked backup schedule is invisible until the night it protects nothing.
↶Remove the monitoring silences created for the window. A silence that outlives the change is the commonest reflex mutation left behind by a cutover, and it hides the failure of the thing that was just changed.
↶Revert speculative changes made during the window: firewall rules opened to let the two sites talk, temporary credentials, hosts-file entries, connection-string edits. Record them even where they were harmless, because the next responder cannot tell which ones were.
↶Leave the preserved image of the primary in place and untouched. If the failback was abandoned because of something found on the primary, that image is the evidence, and rebuilding over it destroys the finding.

6 · Escalation

When the runbook isn't enough, contact:

· The recovery site cannot be backed up, or the backup cannot be verified: stop and escalate to the incident owner before any transfer. Proceeding moves the only copy of the DR-era data, and no schedule pressure justifies that.
· The reconciliation shows records changed on both sides during the DR period: escalate to the data owner and the application owner together. This is a split-brain merge, it has no generic answer, and choosing a winner by timestamp is a decision somebody with domain knowledge must make.
· The primary shows any sign of compromise — unexplained accounts, altered scheduled tasks, modified binaries, missing logs: escalate to security before rebuilding, because a rebuild overwrites the evidence, and hold the position in DR while they decide.
· The final delta will not converge inside the agreed window: escalate to the change authority with the measured remaining work and recommend abort. Extending a cutover window by improvisation is how a planned change becomes the second outage.
· Sequences or identifier ranges cannot be reconciled without a risk of collision: escalate to the application owner. A duplicate primary key discovered a week later is far more expensive than an aborted window.
· The business validation fails at the primary — DR-era records are missing or wrong: invoke the rollback immediately and escalate to the incident owner. Do not investigate with production traffic at the primary; investigate from the recovery site once the traffic is back.

A failover is an emergency, and everybody treats it like one. A failback is a change, and the danger is that it gets treated like a formality. By the time this procedure runs, the recovery site has been production for hours or days. It holds orders, tickets, rows and log lines that exist nowhere else. The primary holds a frozen copy of the world as it was at the instant it failed. Going back is therefore a data reconciliation with a cutover attached, not a cutover with some copying attached.

Which side is authoritative, and who wrote it down

Two sides hold real data and neither holds all of it. The primary is authoritative for nothing committed after the failover instant; the recovery site is authoritative for everything after it. Both statements are obvious, and the timestamp that makes them actionable is the thing most often missing, because during the failover nobody had a spare minute to record it.

RBDR_FAILOVER_AT='2026-08-21 04:17:00+00'
RBDR_DR_HOST=rbdr-dr.example.net
RBDR_PRIMARY_HOST=rbdr-primary.example.net
printf 'recovery site authoritative from: %s\n' "$RBDR_FAILOVER_AT"

Data class	Authoritative side	How you tell
Records created during the DR period	recovery site	created after the failover instant
Records last changed before the failure	either; they agree	identical on both sides
Records changed on both sides	neither — escalate	modified after the instant on the primary too
Configuration changed to enable DR	recovery site, temporarily	it exists only to serve the failover
Secrets rotated during the incident	recovery site	the primary holds the superseded copy

The third row is the one that ends windows. It can only exist if the primary was writable again at some point after the failover, which is exactly why the primary stays isolated until the reconciliation is settled.

Enumerate what the recovery site created

The enumeration is boring to produce and impossible to reconstruct once the two sides have been merged. Take counts and identifiers, not impressions, and keep the file somewhere that is neither site.

RBDR_DR_DATA=/srv/rbdr-app/data
find "$RBDR_DR_DATA" -type f -newermt "$RBDR_FAILOVER_AT" -printf '%TF %TT %p\n' \
  | sort | tee /var/tmp/rbdr-dr-created.txt | wc -l
psql -h "$RBDR_DR_HOST" -d rbdr_orders -Atc \
  "SELECT count(*), max(id) FROM orders WHERE created_at >= '$RBDR_FAILOVER_AT'"

Protect the recovery site before a single byte moves

Configuration changethe pre-failback backup, which is also the rollback anchor— No transcript in this course covers a two-site failback, so no output is shown. Command shape from the restic documentation.

$ restic -r "$RBDR_REPO" --password-file "$RBDR_PASS" backup "$RBDR_DR_DATA" --tag rbdr-pre-failback

RBDR_REPO=sftp:rbdr-recovery@vault.example.net:/srv/rbdr-repo
RBDR_PASS=/mnt/rbdr-escrow/rbdr-recovery.pass
restic -r "$RBDR_REPO" --password-file "$RBDR_PASS" snapshots --tag rbdr-pre-failback
restic -r "$RBDR_REPO" --password-file "$RBDR_PASS" restore latest --target /srv/rbdr-verify

Recovery point selection: two of them, neither one “the latest”

The delta is a data problem before it is an infrastructure one

Moving files is the easy half. The half that ends windows is arithmetic: sequences that advanced at the recovery site, identifier ranges issued on both sides, and records touched in two places. Run every mirroring transfer as a dry run first and read what it proposes to remove.

Data-loss riskpreview the delta, including the deletions— No captured transcript covers this transfer; output omitted deliberately. Flags from the rsync(1) manual page.

$ rsync -a --delete --dry-run --itemize-changes "$RBDR_DR_DATA/" "rbdr-ops@$RBDR_PRIMARY_HOST:/srv/rbdr-app/data/"

rsync -a --delete --dry-run --itemize-changes \
  "$RBDR_DR_DATA/" "rbdr-ops@$RBDR_PRIMARY_HOST:/srv/rbdr-app/data/" \
  > /var/tmp/rbdr-delta-preview.txt
grep '^\*deleting' /var/tmp/rbdr-delta-preview.txt | head -50
psql -h "$RBDR_DR_HOST" -d rbdr_orders -Atc "SELECT last_value FROM orders_id_seq"

Decision point — resume the primary, or rebuild it

Condition	Resume the primary	Rebuild from definitions
Condition confirmed intact, downtime short	Chosen	Unnecessary cost
Any indication of compromise, or condition unknown	Refused	Chosen
Days of unapplied patches and drift	Risky	Chosen
Cause of the original failure not yet understood	Refused	Chosen
Infrastructure definitions absent or unproven	Chosen by default	Becomes a project
Recovery site can serve for another week	Available	Available, run in parallel

Resuming inherits everything, including whatever caused the failover. Rebuilding costs hours and produces a primary somebody can vouch for — and it runs happily in parallel while the recovery site keeps serving, which is why the fastest option is rarely the one worth choosing.

The second cutover window

The window only has to cover what changes after the bulk transfer, so move everything you can while the recovery site is still serving. Lower the TTL ahead of time; the value in force when you change the record is the tail you inherit.

RBDR_FQDN=app.example.net
dig +noall +answer "$RBDR_FQDN"

Then the sequence that makes the delta finite: stop writes at the recovery site and record the instant, transfer the final delta, re-run the enumeration on both sides, compare counts and identifiers, and only then move the traffic. Leave the recovery site running read-only. Shutting it down at the moment of cutover removes the rollback the next twenty minutes may need.

Abort criteria

Stop and hold the position when any of these is true:

The recovery site cannot be backed up, or its backup cannot be verified.
The enumeration disagrees across the two sides and the difference cannot be explained record by record.
Sequences or identifier ranges cannot be reconciled without collision risk.
Records were changed on both sides during the DR period.
The final delta will not converge inside the agreed window.
The primary shows any sign of compromise once it is up and isolated.

Staying in DR is a legitimate outcome of this window, not a failure of it. The recovery site is serving. The estate is running. The only thing an abort costs is the plan, and the only thing continuing costs is the data. Record the abort, restore the DR-side protection, and schedule the window again.

Business validation: prove the DR-era data came home

“The application starts at the primary” is a statement about the application. The failback is finished when somebody who is not you finds records that exist only because the recovery site served them.

RBDR_ORDER_REF=$(head -1 /var/tmp/rbdr-dr-witness.txt)
psql -h "$RBDR_PRIMARY_HOST" -d rbdr_orders -Atc \
  "SELECT id, created_at FROM orders WHERE reference = '$RBDR_ORDER_REF'"
psql -h "$RBDR_PRIMARY_HOST" -d rbdr_orders -Atc \
  "SELECT count(*) FROM orders WHERE created_at >= '$RBDR_FAILOVER_AT'"

The data owner picks the sample, not you, and the count must match the enumeration taken before the transfer. Then re-establish what the primary lost while it was down: backup schedules, monitoring, alerting and replication direction, each confirmed by evidence — a completed backup, a fired test alert — rather than by being enabled.

What to record

The failover instant and the write-stop instant. Both recovery points, by repository, identifier and timestamp, with the reason the seed was not the newest. Both decision points and the condition that settled each. The enumeration, twice, and every difference between the two runs. The measured cutover duration beside the promised one. The sign-off, by name. And the hold period: what stays undeleted at the recovery site, and until when.

Cross-course references

Linux for Production Sysadmins, Part L (Disaster Recovery) — the failover half of this pair, and the site-level mechanics this runbook assumes have already been exercised once.
PostgreSQL for Production Sysadmins, Part XV (High Availability, Failover and Disaster Recovery) — why a promoted standby cannot simply be demoted back, and what reseeding the old primary actually involves.
Kubernetes for Production Sysadmins, Part XCVIII (Disaster Recovery) — the same authoritative-side question where desired state is reconciled continuously and the cluster will happily assert the stale version.
Terraform for Production Sysadmins, Part XXVIII (Disaster Recovery and Resilience) — rebuilding the primary from definitions rather than resuming it, and the state-file question that decides whether that is possible.