Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-failback~55 min

The failover was the easy half and the return lost two days of orders

Reported symptoms

  • ●A power failure at the primary site at 22:14 on 2026-07-29 triggers a failover that runs exactly to plan: the standby at rbdr-site-b is promoted at 22:31, DNS moves, and svc-orders is taking orders again seventeen minutes after the lights went out
  • ●The estate runs at rbdr-site-b for two days and nothing goes wrong there: orders are taken, invoices are issued, dispatch notes are printed, and customers are billed
  • ●On 2026-08-01 the primary is available again and the failback is executed in twenty-four minutes, which everyone reads as a sign that the exercise went well
  • ●At 14:20 that afternoon the finance team asks why the order sequence has gone backwards, and why two customers who were invoiced on 31 July have no order on file
  • ●Every order created between the failover and the failback — 1,692 of them, identifiers 48214 through 49905 — is absent from the live database, while orders created before 22:14 on 29 July are all present and correct
  • ●New orders taken since the failback are being issued identifiers that were already used during the two days at the recovery site, so the collisions are still accumulating while the incident is being investigated
  • ●Nobody on the call can say what time the recovery site began accepting writes, and the change record for the failover names an approver and a date but no instant

Evidence

  • · At rbdr-site-a the orders relation ends cleanly at 2026-07-29 22:13:52 UTC with identifier 48213 and there is no partial or corrupt row after it, which is the signature of a database that stopped being asked questions rather than one that was damaged
  • · The recovery site archived its own write-ahead log to rbdr-wal-b throughout the DR period, and that archive is on timeline 2 and runs from the promotion through 2026-08-01 09:38 UTC, containing the commits for all 1,692 missing orders
  • · The timeline history file 00000002.history in rbdr-wal-b records the fork point at which the two sides parted, and it is the only artefact anywhere in the estate that fixes that instant; no register, runbook or change record contains it
  • · The live data directory at rbdr-site-b is now a base backup taken from rbdr-site-a on 2026-08-01 and is on timeline 1, so the recovery site no longer holds the DR-era rows either
  • · The rebuild-standby script log for 2026-08-01 09:41 shows it took a fresh base backup from rbdr-site-a, overwrote the data directory at rbdr-site-b and exited 0, and it was the same script and the same direction the estate had used since the sites were built
  • · Section 8 of the disaster recovery plan, titled Failover, contains twenty-four numbered steps with named owners and stated verification for each; section 9, titled Failback, reads in full "When the primary site is available, reverse the steps in section 8"
  • · Nowhere in section 9, or in any other document, is there a step that names an authoritative side per dataset, sizes or transfers a delta, or validates the returned service against a record created during the DR period
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first destroyed two days of orders; the second is the reason nothing between the decision and the outcome asked a question. **The immediate cause.** The failback was executed by making the primary authoritative and rebuilding the recovery site from it. The primary was started, the estate's long-standing rebuild-standby script was run in the direction it had always run — `rbdr-site-a` to `rbdr-site-b` — and DNS was moved back. That script takes a base backup from the source and overwrites the destination's data directory, which is correct behaviour for the operation it was written for and catastrophic for this one. The primary held the state of 2026-07-29 22:13:52 and nothing after it, because from that instant it was not asked any questions. The recovery site held that state plus two days of production. Failback moved the older copy over the newer one. No step in the sequence transferred the delta in the other direction, because no step in the sequence was about data at all. **The control that should have caught it.** The disaster recovery plan documents a failover in twenty-four numbered steps, each with an owner and a stated verification, and documents the failback in one sentence that says to reverse them. Reversal is not a defined operation here. Failover moved a service onto a site holding a known-good copy; failback has to reconcile two copies that have both moved on, and a procedure that does not contain the word data cannot be reversed into one that does. The same gap appears as a missing record. The estate has no register of which site was authoritative when. The failover change record names a person and a date; it does not name the instant writes began at the recovery site. Without that instant nobody can size a delta, so nobody can notice that one was never moved, and the operator who ran the failback in twenty-four minutes had no artefact that would have told him he was twenty-four minutes short. **The alternative hypothesis.** The obvious competing explanation is that the recovery site never received the writes at all — that the application at `rbdr-site-b` was pointed at something stale, or queued the orders and never drained them, in which case the two days were lost at the failover rather than at the failback and the investigation belongs to the application team. The recovery site's own archived write-ahead log eliminates it. The archive in `rbdr-wal-b` is on timeline 2, its segments run from the promotion through 09:38 on 2026-08-01, and they contain the commits for identifiers 48214 to 49905. The writes reached the recovery site and were committed there. They were overwritten, not never written — which also means they are recoverable.

Remediation

Freeze before you fix. The only surviving copy of the DR-era data is the recovery site's pre-fork base backup plus its archived write-ahead log in `rbdr-wal-b`, and both are inside the reach of an ordinary retention policy. Put a hold on them before anything else happens, and stop the archive from being rotated, expired or re-pointed while the incident runs. Establish the two instants from the database rather than from memory. The fork point comes from `00000002.history` in the recovery site's archive; the last commit on timeline 2 comes from the newest archived segment on that timeline. Those two timestamps bound the DR period, and every later decision is derived from them. Recover the DR-era data to an isolated host and never over the live database. Restore the recovery site's last base backup taken before the fork, set the recovery target timeline explicitly to 2 rather than relying on a default, and replay the archive to the last commit. That produces a readable copy of the estate as the recovery site left it, alongside the live one rather than instead of it. Then enumerate and reconcile, which is a data owner's job and not an infrastructure one. Extract the DR-era rows from the isolated copy by identifier, and hand the list to the person who owns the records. Two questions belong to them: which side wins where a record exists on both, and what to do about the identifiers issued twice, because the sequence at the primary restarted at 48214 and orders taken since the failback are colliding with orders taken during the DR period. Reconciliation also reaches outside the database — invoices were sent, payments were captured, dispatch notes were printed — and none of that is undone by a row. Set the sequence beyond the highest identifier issued anywhere before returning the service to normal operation, or the collisions continue to accumulate while the merge is being planned. Only then fix the two defects: write the failback procedure the plan does not have, and start the register that records which site is authoritative from which instant.

Verification

The DR-era identifiers recovered from the isolated copy are compared against the live database by count and by identifier, and every difference is itemised and explained individually rather than accepted as a rounding error. No identifier appears twice in the live database, and the sequence is at or beyond the highest value issued at either site, confirmed by query rather than by assumption. A named person from the business, not from the infrastructure team, confirms against records they chose themselves — a specific order, invoice and dispatch note created during the DR period — that those records are present and correct in the live service. The retention hold on `rbdr-wal-b` and on the pre-fork base backup is released only after the business has confirmed a full reporting cycle, and the release is a decision somebody records rather than an expiry nobody sees. The rewritten section 9 names an authoritative side per dataset, a delta transfer with a stated direction, a reconciliation step and a validation performed against DR-era records, and a reader who has never seen the estate can point at the step that would have stopped this failback. The authoritative-site register contains a row for this incident with the failover instant taken from the timeline history file, and a rehearsal confirms that a future operator can find that instant without reading a database.

Prevention

**Write the failback procedure before the failover, as its own document with its own approver.** "Reverse the failover steps" is not a procedure. Failover moves a service onto a site holding a known-good copy; failback reconciles two copies that have both been written to, and the second operation is the harder one. **Keep a register of which site is authoritative from which instant.** One row per transition, recorded when it happens, holding the timestamp writes began at the new site. Without it no delta can be sized, and an unsized delta is one nobody notices was skipped. **Make any tool that overwrites a data directory refuse when the destination holds history the source does not.** The rebuild-standby script did exactly what it was written to do; the defect is that nothing between the operator and the overwrite compared the two sides first. **Validate a failback against records created during the DR period, chosen by somebody else.** "The service starts and the dashboards are green" was true here at 10:05 and stayed true for four hours. A named order that exists only because the recovery site served it is the test that fails immediately. **Treat the recovery site as production while it is production, and back it up to the same standard.** For two days it held data with no second home. The archive that saved this incident existed by configuration inheritance rather than by decision, which is not a control anyone should rely on twice. **Rehearse the failback, not only the failover.** The half of disaster recovery that is never practised is the half that is executed after the bridge call has dissolved, by people who are tired, with nobody watching.

Reported symptoms

At 22:14 on 2026-07-29 the primary site lost its upstream and the generator did not pick up the load. The failover ran to plan: the standby at rbdr-site-b was promoted at 22:31, DNS moved, and svc-orders was taking orders seventeen minutes later. Nobody had a bad word for it.

The estate ran at rbdr-site-b for two days — orders taken, invoices issued, dispatch notes printed, customers billed. On 2026-08-01 the primary was back and the failback was executed between 09:41 and 10:05. Twenty-four minutes, which everybody read as a sign the exercise had gone well.

At 14:20 finance asked why the order sequence had gone backwards, and why two customers invoiced on 31 July had no order on file. Every order created between the failover and the failback — 1,692 of them, identifiers 48214 to 49905 — is missing from the live database. Orders from before 22:14 on 29 July are all present. New orders are being issued identifiers already used during the two days at the recovery site, so the damage is still growing.

Evidence provided

Read-only / Safewhere the primary's copy stops
$ psql -At -d rbdr_orders -c 'SELECT max(id), max(created_at) FROM orders'

The primary ends at identifier 48213, created 2026-07-29 22:13:52 UTC. No partial row after it and no corruption: the clean stop of a database that was no longer being asked questions.

The recovery site kept its own archive. rbdr-site-b archived its write-ahead log to rbdr-wal-b throughout, on timeline 2.

RBDR_ARCHIVE=/srv/rbdr-wal-b
RBDR_TIMELINE=00000002
cat "$RBDR_ARCHIVE/${RBDR_TIMELINE}.history"
find "$RBDR_ARCHIVE" -maxdepth 1 -name "${RBDR_TIMELINE}*" -printf '%f\n' \
  | sort | tail -1

The history file records the fork point at which the two sides parted, and it is the only artefact in the estate that fixes that instant — no register, runbook or change record contains it. The segments on that timeline run from the promotion through 09:38 on 2026-08-01 and carry the commits for all 1,692 missing orders.

What the recovery site holds now. Its live data directory is a base backup taken from rbdr-site-a on 2026-08-01, on timeline 1. The DR-era rows are not there either.

The script that did it. The rebuild-standby log for 09:41 shows a base backup taken from rbdr-site-a, the data directory at rbdr-site-b overwritten, exit 0. Same script, same direction, as every day since the sites were built.

The plan. Section 8, Failover: twenty-four numbered steps, each with an owner and a stated verification. Section 9, Failback, in full: “When the primary site is available, reverse the steps in section 8.”

Work the evidence before reading on

  1. The primary’s data stops three minutes before the promotion; the recovery site’s continues two days past it. Which of those was anybody required to look at during the failback?
  2. Section 9 says to reverse section 8. Which of those twenty-four steps, reversed, moves a byte of order data northwards?
  3. The archive on timeline 2 survived. What does its existence rule out?
  4. Nobody can state the instant writes began at the recovery site. Name three decisions during the failback that needed that number.

Root cause

The primary was made authoritative and the delta was never moved

Failback consisted of starting the primary, running the estate’s rebuild-standby script in the direction it had always run — rbdr-site-a to rbdr-site-b — and moving DNS back. That script takes a base backup from the source and overwrites the destination’s data directory: correct behaviour for the operation it was written for, catastrophic for this one.

The primary held the state of 22:13:52 and nothing after, because from that instant it was asked nothing. The recovery site held that state plus two days of production. The failback moved the older copy over the newer one. No step transferred the delta the other way, because no step in the sequence was about data at all.

A plan with a failover procedure and no failback procedure

Twenty-four steps on one side, one sentence on the other. “Reverse the steps” is not a defined operation: failover moved a service onto a site holding a known-good copy, while failback reconciles two copies that have both moved on, and a procedure that never mentions data cannot be reversed into one that does.

The gap has a second face. The estate keeps no register of which site was authoritative from which instant; the failover change record names a person and a date, not the moment writes began at rbdr-site-b. Without that moment nobody can size a delta, so nobody can notice one was never moved — and the operator who finished in twenty-four minutes had no artefact to tell him he was twenty-four minutes short.

Resolution

Freeze before you fix. The only surviving copy of the DR-era data is the recovery site’s pre-fork base backup plus rbdr-wal-b, and both sit inside the reach of an ordinary retention policy. Hold them, and stop the archive being rotated or re-pointed while the incident runs.

Take the two instants from the database rather than from memory: the fork point from 00000002.history, the last commit from the newest segment on timeline 2. Then recover to an isolated host, alongside the live database rather than instead of it — never over it.

RBDR_STAGE=/srv/rbdr-stage/pgdata
RBDR_ARCHIVE=/srv/rbdr-wal-b
tar -xf /srv/rbdr-base-b/rbdr-base-20260729.tar -C "$RBDR_STAGE"
cat >> "$RBDR_STAGE/postgresql.auto.conf" <<EOF
restore_command = 'cp $RBDR_ARCHIVE/%f %p'
recovery_target_timeline = '2'
EOF
touch "$RBDR_STAGE/recovery.signal"

Set the target timeline explicitly rather than relying on a default. Then enumerate the DR-era rows and hand the list to whoever owns the records.

RBDR_STAGE_PORT=5433
psql -p "$RBDR_STAGE_PORT" -At -d rbdr_orders -c \
  "SELECT id, created_at FROM orders WHERE created_at >= '2026-07-29 22:31:00+00'"

Two questions are theirs: which side wins where a record exists on both, and what to do about identifiers issued twice — the primary’s sequence restarted at 48214, so every order since the failback collides with one taken during the DR period. Reconciliation reaches outside the database too: invoices were sent, payments captured, dispatch notes printed, and no row undoes any of that. Move the sequence beyond the highest identifier issued anywhere before normal operation resumes.

Verification

The recovered DR-era identifiers are compared with the live database by count and by identifier, with every difference itemised rather than absorbed. No identifier appears twice, and the sequence sits beyond the highest value issued at either site, confirmed by query rather than by assumption.

A named person from the business — not from infrastructure — confirms against records they chose themselves that a specific order, invoice and dispatch note created during the DR period are present and correct.

The rewritten section 9 names an authoritative side per dataset, a delta transfer with a stated direction, a reconciliation step and a validation against DR-era records; the register carries a row for this incident whose failover instant came from the timeline history file.

Prevention

Write the failback procedure before the failover, with its own approver. “Reverse the failover steps” is not a procedure, and the second operation is the harder of the two.

Keep a register of which site is authoritative from which instant. Without it no delta can be sized, and an unsized delta is one nobody notices was skipped.

Make any tool that overwrites a data directory refuse when the destination holds history the source does not. The script did what it was written to do; nothing between the operator and the overwrite compared the two sides.

Validate a failback against DR-era records chosen by somebody else. Green dashboards were true at 10:05 and stayed true for four hours; a named order that exists only because the recovery site served it fails immediately.

Treat the recovery site as production while it is production. For two days it held data with no second home, and the archive that saved this incident existed by configuration inheritance rather than by decision.

Rehearse the failback, not only the failover. The unrehearsed half is the one executed after the bridge call has dissolved, by people who are tired, with nobody watching.