Business-level validation

Restoring onto clean infrastructure closes the last gap the platform can close by itself: nothing in the recovered system is borrowed from the machine that failed, so what runs is what the backup actually held. It leaves the question that no infrastructure check reaches. A service can be structurally perfect and hold the wrong data — the right tables, the right size, the right number of rows — while every signal an operator can see reads green. The remaining gap between a running system and a recovered service is a claim about the data itself, and closing it depends entirely on whether somebody wrote that claim down before the incident.

An invariant is a claim recorded while the data was still correct

An invariant, for this purpose, is a property of the business data recorded at a moment the data was known to be right, which must still hold once the recovery finishes. Both halves carry weight: it is a property of the data rather than of the system, so no amount of process health satisfies it, and it predates the incident, so the recovered system had no hand in producing it.

Four shapes cover most of what real services can offer.

A reconciliation total is a sum the business already computes for its own reasons — the value of open orders, a ledger balance, the total invoiced in a period. Its strength is that a second, independent system usually holds the same figure, so the reference need not live in your estate at all.

A checksum over a business column is the engineered version: a sum, or a hash over an ordered projection of a key column, computed at backup time so that a later recovery can be compared against it. It exists for no purpose but validation, which means it can be made sensitive to exactly the wrongness you care about.

A count of open records — unshipped orders, unresolved tickets, active subscriptions — is weaker, with one large advantage: the people who run the business look at it daily and will notice a discrepancy of the size a bad recovery produces.

A known transaction is the cheapest and often the most convincing. Order 4482911 was placed at 09:14, has three line items, and a customer holds the confirmation email. If the row is absent or its total differs, the recovery is rejected on evidence that lives outside every system you operate.

What all four have in common is the property that makes this lesson necessary: none of them can be produced during the incident. At the moment you need a reference, the only copy of the correct data is the one you are trying to rebuild. Any figure computed then comes from a system whose correctness is the open question.

Measured: 50000 is easy to satisfy, 825025000 is not

The point-in-time recovery captured for this course makes the difference concrete. A base backup was taken while the orders table held 45000 rows, business continued afterwards, and two figures were recorded from the live database at that point — while the data was still correct, and by something other than the backup tool.

$ pg_basebackup -D /work/base -X stream -c fast

  >>> exit code: 0
rows contained in the base backup: 45000

--- business continues after the backup: 5,000 more orders arrive ---
rows now                      : 50000
checksum of the business data : sum(amount)=825025000
recovery target time          : 2026-08-28 13:34:40.077562+00

Then an unqualified DELETE ran, the capture recorded the table at zero rows, the base backup was recovered forward through five archived write-ahead log segments, and the acceptance decision was made on two comparisons rather than on the fact that a server had started.

  rows recovered  : 50000   (expected 50000)
  sum(amount)     : 825025000   (expected 825025000)
  RECOVERED - row count and business checksum both match the pre-DELETE state

The two lines look symmetric and are not. 50000 is a scalar over cardinality: it says how many rows exist and nothing whatever about what is in them. 825025000 is a scalar over content, and it changes if any row’s amount changes, if the wrong rows are present, or if rows from a different period were substituted for the ones that should be there.

Consider what each one catches. A recovery that stops thirty seconds early is short several hundred orders, and both figures reject it — the weak invariant is adequate for gross wrongness. Now consider a workload that inserts and deletes, which is every real order system with cancellations: a recovery that lands at the wrong instant can hold the same number of rows while the membership differs, five hundred late orders standing in for five hundred earlier ones. The count passes. The sum does not, and it also rejects the table repaired from a stale export and the right table restored from the wrong day.

A sum is not a cryptographic hash, and the limit is worth stating rather than overselling. Two compensating errors cancel, and any column outside the sum is invisible to it — a shipping address corrupted by a bad migration passes a monetary reconciliation unmarked. The answer is not to abandon the sum but to record several figures for the price of one scan: a count, a sum, the maximum key and a per-day breakdown constrain the table far more tightly than any of them alone. An invariant is chosen against a class of wrongness you have decided to detect, and naming that class is part of choosing.

The most important property of 825025000 is not arithmetic at all. It is that the number was written down before the DELETE, somewhere the recovery could not influence.

Who is allowed to say the data is correct

The engineer performs the recovery. The data owner accepts it. That division is not politeness or governance theatre; there are three reasons the engineer cannot hold the decision, and each is sufficient on its own.

The engineer has no standard for correct. The number 825025000 carries no meaning for the person who typed the recovery command. It carries a great deal for the person who reconciles the ledger every month and knows what the total should be moving at, what it did on the day in question, and which of the figures on the screen would be strange. Correctness is a semantic property of the business’s data, and semantics live with the people who use it.

The person who did the work cannot certify the work. By the time a recovery reaches validation, the engineer has usually been awake for hours and wants it to be finished, and every ambiguity looks slightly more acceptable than it did at the start. Independent verification of an operator’s own result is a basic control everywhere else that handles consequences.

Acceptance is a business risk decision. Declaring the data good means resuming invoicing, shipping and reporting on it. If it is wrong, the cost is wrong invoices, mis-stated balances, a regulatory filing built on bad numbers, and a second incident weeks later when somebody notices. Those consequences land on the business, so the business holds the decision.

Three consequences follow. Name the owner and a deputy per data set in the restore runbook, beside the invariant, because an acceptance waiting on an unreachable person makes recovery time a function of somebody’s phone. Agree the criterion in advance as a pass or fail with an explicit tolerance — “the monetary total matches exactly” is a different commitment from “the open order count is within five”, and both are the owner’s call. And give them a way to look for themselves: a quarantined read-only instance and a query they can run beats an engineer reading a number aloud over a bridge call.

The transaction that exercises everything a query never touches

An invariant inspects data at rest. A service is a path, and between a customer and a stored row sit a DNS name, a certificate chain, a load balancer, a connection pooler, authentication and role grants, a sequence handing out primary keys, a queue carrying the confirmation, an object store holding the attachment, a search index, and whatever third parties the transaction calls. None of that is visible to select sum(amount).

The end-to-end transaction is one genuine unit of work pushed through the whole path: place the order, take payment, watch it appear in the customer’s list, confirm the notification arrives. Every layer participates, so it is the only check that catches the failures living between components — a certificate that expired while the recovery site sat idle, a sequence whose next value produces a duplicate key on the first insert, a role the restored application cannot authenticate as, a queue whose credentials were never restored, a DNS name still resolving to the site that failed.

Four design constraints make it usable. It must write, because a read-only smoke test proves reachability and nothing about the write path where most recovery defects live. It must be recognisable to the owner, because the purpose is to let them say the service works, not to let you say a script returned zero. It must be bounded — a designated test account, an order flagged as a rehearsal, a documented clean-up — or the exercise injects fictional business data into the data you just spent hours restoring, and the next reconciliation fails for a reason nobody traces. And its external dependencies must be arranged in advance, because a real path reaches real third parties: a real card charge, a real email to a real customer, a real webhook into a partner’s production system. Provision the sandbox and its credentials as part of the recovery design, or you own a check nobody dares run during an incident, and a check nobody dares run is not a control.

Run it the way a customer would, from outside, over the same names, rather than from a shell on the recovered host. The difference between those two is exactly the set of dependencies you are trying to exercise.

Recording the invariant is a backup design requirement

Everything above collapses into one requirement: for each data set with a recovery objective, the job that produces the backup also produces the invariant.

It must be the same job because a figure computed by a separate, later task describes a different instant, so it disagrees with a faithful recovery by however much the business moved in between. Teams meeting that during a rehearsal usually label the mismatch known noise, and the comparison then decides nothing.

The value is stored outside the failure domain it describes, ideally on write-once storage of the kind the object storage and immutability parts of this course cover, because an invariant that ransomware can rewrite alongside the data is decoration. It has to be cheap enough to compute every time — a full aggregate scan of a multi-terabyte table at every backup is a real cost, and the honest options are an indexed column, a per-partition figure, or a sampled invariant the owner has agreed to. And it must be findable at three in the morning by somebody who does not know your storage layout, which means the restore runbook names the exact location.

set -euo pipefail

STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
EVIDENCE="/mnt/recovery-evidence/orders"
mkdir -p "$EVIDENCE"

pg_basebackup -D /work/base -X stream -c fast

psql -d orders -At -F'|' \
  -c 'select count(*), sum(amount), max(id), now() from orders' \
  > "$EVIDENCE/orders-$STAMP.inv"

The file in $EVIDENCE is what a recovery will be judged against months later: give it the retention and immutability you give the backup, and record which owner agreed to it.

Production discipline

1. Name the owner before you need them. Each data set with a recovery objective gets a named accepting owner and a deputy, with contact details in the restore runbook, because the engineer who performed the recovery has no standard for correct and cannot certify their own work.
2. Emit the invariant from the backup job itself. The measured acceptance turned on sum(amount)=825025000 being recorded beside the base backup while the data was still right; a figure produced by a later job describes a different instant and produces mismatches that get dismissed as noise.
3. Prefer a content invariant to a cardinality one, and record both. The recovery was accepted on rows recovered : 50000 together with sum(amount)=825025000, because a count is satisfied by a table with the right size and the wrong contents — a screening check, not an acceptance one.
4. Write down the pass criterion and its tolerance in advance. Exact match or a stated margin is a business decision made calmly, not a judgement made at 03:00 by whoever is holding the terminal.
5. Rehearse the end-to-end transaction with its sandbox provisioned. It is the only check that reaches DNS, certificates, pooling, sequences, queues and third parties, and it is worthless if the first attempt to run it would place a real charge on a real customer.

Cross-course references

• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) covers the archiving and recovery-target mechanics that produced the measured result quoted here, and the relationship is a division of labour: that material decides where a recovery can land, while this lesson decides whether the place it landed is acceptable to the business.
• Observability for Production Sysadmins — Part LXII (Business Metrics) builds exactly the kind of figure this lesson depends on, recorded continuously and stored outside the database, which makes an existing business metric the cheapest candidate invariant an estate can offer — it already predates the incident and nobody has to invent it.
• Observability for Production Sysadmins — Part LXIII (Synthetic Monitoring) is the engineering discipline behind the end-to-end transaction: the same synthetic user journey that watches production continuously is the check to point at a recovered service, including the sandboxed third parties that make it safe to run during an incident.

Quiz