Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-business-validation~50 min

Every service is running and no order can be placed

Reported symptoms

  • ●The primary site for the order platform rbdr-shop is lost at 18:41 when the storage array fails, DR recovery runs at the secondary site, and recovery is declared complete at 21:07 on the grounds that every service is running
  • ●At 21:19 the business reports that no customer can complete a purchase, and the failure rate is one hundred per cent rather than intermittent — not a slow checkout, not some baskets, none of them
  • ●Browsing works, search works, sign-in works, existing orders display correctly with their totals, and the customer-facing site shows no error until the final confirm step
  • ●Every unit is active, every port answers, every /healthz returns 200, both database clusters accept connections and answer SELECT 1, and every monitoring dashboard is green
  • ●The incident bridge concludes within minutes that one of the recovered services must be misconfigured — a wrong connection string, a stale environment variable, a feature flag that did not come back — and starts diffing unit files
  • ●The recovery completion record states, for each system, that it was restored from the latest good backup, and records no recovery point for any of them
  • ●Checkout logs show the request reaching the pricing step and returning zero rows rather than an error, so nothing in the log looks like a failure until the step after it

Evidence

  • · The recovery point achieved by each system, read from the system itself rather than from the request: rbdr-orders was recovered by point-in-time recovery to 2026-08-27 18:39:04 and rbdr-catalog was restored from its nightly logical dump taken at 23:05 the previous evening, a divergence of nineteen hours and thirty-four minutes
  • · A scripted end-to-end checkout fails at step four of six, price the basket: the pricing resolver reads active_price_list_id = 4127 from rbdr-orders and the highest price list present in rbdr-catalog is 4126, so the lookup returns no rows and the transaction is abandoned before payment
  • · A business invariant that held at zero in the nightly reconciliation report and does not hold after recovery: 312 distinct product identifiers referenced by order lines in rbdr-orders have no corresponding row in rbdr-catalog, and every one of them was created after 23:05
  • · Each service is healthy in isolation: rbdr-catalog returns the correct price and tax class for product rbdr-sku-88014, created 2026-08-21, matching the figure in the pre-incident record, and rbdr-orders accepts, persists and reads back a synthetic basket against its own schema
  • · The configuration of all three recovered units is byte-identical to the configuration-management output captured before the incident, so the diff the bridge started producing has no lines in it
  • · The same checkout code path succeeds for a basket whose lines were all created before 23:05 and fails for any basket containing a line created after it, with identical configuration, identical credentials and identical binaries on both attempts
  • · The DR validation runbook has five rungs — unit active, port answering, health endpoint 200, database accepts connections, dashboards green — and every one of them tests a single component on its own
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first stopped the business; the second is the reason it was discovered by a customer rather than by the recovery team. **The immediate cause: two systems in one transaction, recovered to two different instants.** `rbdr-orders` has a base backup and an archived write-ahead log, so it was recovered forward to 18:39:04, seconds before the array failed. `rbdr-catalog` has the same archive available, but its entry in the recovery plan says to restore the nightly logical dump, so it came back at 23:05 the previous evening. Both restores were correct against their own instructions and both produced a healthy cluster. The order path spans both. Products, price lists and tax classes live in the catalogue; baskets, orders and the pointer to the currently published price list live in the orders database. Nineteen hours of catalogue changes exist in the orders database as identifiers and nowhere else, and because the two datasets sit in separate clusters no foreign key was ever in a position to refuse them. The pointer is what makes the failure total rather than partial: every checkout resolves `active_price_list_id`, that value is 4127, and 4127 does not exist in the recovered catalogue. A basket that contains only old products still cannot be priced. **The control that should have caught it: a validation ladder made entirely of isolation checks.** Five rungs, each asking one component whether it is alive. A defect that lives in the relationship between two components cannot fail any of them, and adding a sixth rung of the same kind would not have helped. The ladder was not too short; it was the wrong shape, because it never crossed a boundary and never compared anything between two systems. It also recorded no recovery point per system, so the divergence was not merely unnoticed — it was not represented anywhere in the completion record. **The alternative hypothesis, and what eliminates it.** The bridge believed a recovered service was misconfigured. Three pieces of evidence rule that out. The configuration of all three units is byte-identical to the pre-incident configuration-management output. Each service answers correctly in isolation: the catalogue returns the right price and tax class for a product created before the divergence, and the orders database accepts and reads back a synthetic basket. And the same code path, with the same configuration and the same credentials, succeeds for a basket whose lines predate 23:05 and fails for one that does not. The discriminator is a timestamp, and a configuration error does not have a timestamp boundary.

Remediation

Do not make the invariant hold by moving the orders database backwards. The orders database holds nineteen hours of committed business that customers were told had succeeded; the catalogue is the system that is behind, and it has the archive needed to catch up. Recover `rbdr-catalog` again, this time from its base backup rolled forward through the archived write-ahead log to a target at or after the point `rbdr-orders` actually reached, and take that point from the recovered orders cluster's own log rather than from the target that was requested of it. Then run the referential invariant and the end-to-end checkout before anything is returned to customers. Declare the outage still open while this runs. A service that accepts a basket and cannot price it is not degraded, and describing it as restored costs the business more than the extra hour. Afterwards fix the second defect, which is the one that will otherwise recur with a different pair of systems: rewrite the ladder so that its final rung is a transaction that crosses every boundary in the order path, and add the cross-dataset invariants as executable checks that run before the recovery can be declared complete.

Verification

The recovery point achieved by every system in the order path is read from the system itself, written into the completion record, and shown to be the same instant, or explicitly accepted as different with the reason stated. The referential invariant returns zero: no product identifier referenced by an order line is missing from the catalogue, and `active_price_list_id` resolves to a row that exists. A scripted end-to-end checkout completes all six steps against the recovered estate, using a basket that contains at least one product created inside the divergence window, and the resulting order total matches the figure recorded for a known pre-incident order. The rewritten ladder is shown to fail. Recover two dependent systems to deliberately different points in a drill and confirm that the last rung rejects the recovery. A validation step that has never been observed to fail is not known to be a control.

Prevention

**Recover systems that share a transaction to one point, and name that group before the incident.** The unit of recovery is the set of systems an identifier can travel between, not the individual database. Write the group down in the recovery plan, and give every member the same recovery mechanism so that one of them cannot silently be nineteen hours behind. **Record the recovery point each system actually reached, read from the system.** The requested target and the achieved point are different numbers, and only the achieved one can be compared against another system. **End the validation ladder at a business transaction, not at process health.** A ladder of per-component checks cannot detect a defect that lives between components, however many rungs it has. **Codify cross-dataset invariants as executable checks and run them at recovery time.** The invariant in this incident already existed in a nightly report; what was missing was anything that ran it when it mattered. **Rehearse the divergence.** Deliberately recover two dependent systems to different points during a drill and require the ladder to reject the result. **Treat a total, immediate failure of one business action as evidence about a shared dependency**, not about the service reporting it. One hundred per cent is a different shape from intermittent, and it points at something singular that every request touches.

Reported symptoms

The storage array under the primary site failed at 18:41. DR recovery ran at the secondary site and was declared complete at 21:07: every unit active, every port answering, every /healthz returning 200, both databases accepting connections, every dashboard green.

At 21:19 the business called. No customer can complete a purchase — not slow, not some baskets, none of them, since the moment the site reopened.

Browsing, search and sign-in work, and existing orders display with the right totals. Nothing errors until the final confirm step, and even the checkout log looks unremarkable: the pricing step returns zero rows, which is not a failure until the step after it. The bridge settles on a misconfigured service and starts diffing unit files. That theory survives six minutes.

Evidence provided

The recovery point each system actually reached. rbdr-orders was recovered forward through archived write-ahead log to 18:39:04. rbdr-catalog was restored from its nightly logical dump, taken at 23:05 the previous evening. Nineteen hours and thirty-four minutes apart.

Note where the first number comes from. The point a recovery reaches is not the point it was asked for, and only the cluster can say which it got:

Service impact possiblethe requested target, and the point actually reached
$ pg_ctl -D /work/base start
  2026-08-28 13:35:12.708 UTC [631] LOG:  starting point-in-time recovery to 2026-08-28 13:34:40.077562+00
2026-08-28 13:35:12.708 UTC [631] LOG:  redo starts at 0/3000028
2026-08-28 13:35:12.713 UTC [631] LOG:  restored log file "000000010000000000000004" from archive
2026-08-28 13:35:12.713 UTC [631] LOG:  completed backup recovery with redo LSN 0/3000028 and end LSN 0/3000120
2026-08-28 13:35:12.713 UTC [631] LOG:  consistent recovery state reached at 0/3000120
2026-08-28 13:35:12.713 UTC [625] LOG:  database system is ready to accept read-only connections
2026-08-28 13:35:12.726 UTC [631] LOG:  restored log file "000000010000000000000005" from archive
2026-08-28 13:35:12.735 UTC [631] LOG:  recovery stopping before commit of transaction 836, time 2026-08-28 13:34:42.096745+00
2026-08-28 13:35:12.735 UTC [631] LOG:  redo done at 0/52EBC90 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:35:12.735 UTC [631] LOG:  last completed transaction was at log time 2026-08-28 13:34:38.041366+00

Three timestamps: one requested, one the replay stopped before, one actually committed. The completion record for rbdr-shop contains none of them. It says “restored from latest good backup”, which is not a recovery point.

The transaction, and the step it dies at. A scripted checkout runs six steps. It fails at four, price the basket.

Read-only / Safethe pointer, and the highest value that exists to point at
$ psql "$RBDR_ORDERS" -Atc "select value from rbdr_config where key = 'active_price_list_id'"
psql "$RBDR_CATALOG" -Atc "select max(price_list_id) from rbdr_price_list"
4127
4126

Illustrative output

The invariant. Every product identifier on an order line should exist in the catalogue. The nightly reconciliation report recorded that count as zero at 23:05. After recovery it is 312, all of them created after the catalogue’s recovery point.

RBDR_OUT=/srv/rbdr-validate
RBDR_ORDERS="postgresql://rbdr_ro@rbdr-orders-dr:5432/rbdr_orders"
RBDR_CATALOG="postgresql://rbdr_ro@rbdr-catalog-dr:5432/rbdr_catalog"
psql "$RBDR_ORDERS" -Atc "select distinct product_id from rbdr_order_line" \
  | sort -u > "$RBDR_OUT/orders-products.txt"
psql "$RBDR_CATALOG" -Atc "select product_id from rbdr_product" \
  | sort -u > "$RBDR_OUT/catalog-products.txt"
comm -23 "$RBDR_OUT/orders-products.txt" "$RBDR_OUT/catalog-products.txt" | wc -l

Each service in isolation. The catalogue returns the correct price and tax class for rbdr-sku-88014, created 2026-08-21, matching the pre-incident record. The orders database accepts, persists and reads back a synthetic basket. The configuration diff the bridge started has no lines in it.

Work the evidence before reading on

  1. The failure is total from the first request. What shape of cause produces that, and what shape does it rule out?
  2. Two identifiers, one minted before 23:05 and one after, take the same code path with the same configuration. One works. What is the variable?
  3. Five rungs passed. What do they have in common, and what would a sixth of the same kind have added?
  4. Which of the two systems is wrong, and how do you decide without asking which one is easier to change?

Root cause

Two systems in one transaction, recovered to two different instants

rbdr-orders had a base backup and an archived write-ahead log, so it rolled forward to seconds before the array failed. rbdr-catalog had the same archive, but its entry in the recovery plan said to restore the nightly dump, so it came back at 23:05. Both restores were correct against their own instructions, and both produced a healthy cluster.

The order path spans both. Nineteen hours of catalogue changes therefore exist in the orders database as identifiers and nowhere else, and because the two datasets sit in separate clusters no foreign key was ever in a position to refuse them.

The pointer is why the failure is total rather than partial: every checkout resolves active_price_list_id, reads 4127, and 4127 does not exist. A basket of nothing but old products still cannot be priced.

A validation ladder built entirely of isolation checks

The course has the counter-example on record. A PostgreSQL data directory copied while the workload ran started cleanly, completed crash recovery and returned every row — none of which established that it was a consistent image of any instant:

Service impact possiblea database that starts and answers, proving nothing about its contents
$ pg_ctl -D /work/naive-copy start
  2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

The same capture shows an acceptance decision made on the data instead — a row count and a business checksum, both compared against figures recorded before the incident:

  rows recovered  : 50000   (expected 50000)
  sum(amount)     : 825025000   (expected 825025000)
  RECOVERED - row count and business checksum both match the pre-DELETE state

The rbdr-shop ladder had no rung of that kind at all.

Resolution

The invariant can be made to hold from either side, and only one side is acceptable. rbdr-orders holds nineteen hours of committed business customers were told had succeeded; the catalogue is the system that is behind, and it has the archive it needs to catch up.

Recover rbdr-catalog again from its base backup, rolled forward to a target at or after the point rbdr-orders actually reached — read from the orders cluster’s own log, not from the target requested of it.

RBDR_DATA=/var/lib/rbdr-catalog-dr
RBDR_TARGET="2026-08-27 18:39:04.512883+00"
{
  echo "restore_command = 'cp /srv/rbdr-wal-catalog/%f %p'"
  echo "recovery_target_time = '$RBDR_TARGET'"
  echo "recovery_target_action = 'pause'"
} >> "$RBDR_DATA/postgresql.auto.conf"
touch "$RBDR_DATA/recovery.signal"

recovery_target_action = pause is deliberate: it holds the cluster at the target so the invariant and the checkout run before anything is promoted.

Keep the outage open while this runs. A service that accepts a basket and cannot price it is not degraded, and calling it restored costs more than the extra hour.

Verification

Every system in the order path reports its achieved recovery point, read from the system itself and written into the completion record, and the points match — or are accepted as different, with the reason stated by whoever owns the data.

The referential invariant returns zero unresolved identifiers, and active_price_list_id resolves to a row that exists.

A scripted checkout completes all six steps, using a basket containing at least one product created inside the divergence window, and the order total matches the figure recorded for a known pre-incident order.

The rewritten ladder is shown to fail: recover two dependent systems to different points in a drill and confirm the last rung rejects the result. A check never observed failing is not yet known to be a control.

Prevention

Recover systems that share a transaction to one point, and name that group before the incident. The unit of recovery is the set of systems an identifier can travel between, not the individual database. Give every member the same recovery mechanism, so one cannot quietly be nineteen hours behind.

Record the recovery point each system actually reached. The requested target and the achieved point are different numbers, and only the achieved one can be compared against another system.

End the ladder at a business transaction that crosses every boundary, and run the cross-dataset invariants at recovery time. Per-component checks cannot see a defect that lives between components, however many you add. The invariant here already existed in a nightly report; what was missing was anything that executed it when it mattered.

Read a total, immediate failure of one business action as evidence about a shared dependency, not about the service reporting it. Total is a different shape from intermittent, and it points at something singular that every request touches.