medium riskservice affecting~60 min

Validate a recovered application

1 · Prerequisites

Confirm every item is in place before any state change.

A recovery that has already been performed and has not yet been declared finished. This runbook validates; it does not restore.
The identifier and timestamp of the recovery point that was selected, and the name of whoever selected it. Validation is a comparison against a decision, so the decision has to exist in writing first.
An invariant recorded before the incident: a row count and a checksum over a business column, or the equivalent for whatever the application stores. In the measured PostgreSQL capture these were 50000 rows and sum(amount)=825025000.
A pre-incident inventory of the schema objects the application needs — tables, indexes, sequences, functions, extensions — so "the schema is present" is a comparison rather than an impression.
A client position: a host on the network the real users sit on, with the trust store the real clients use. Validating from inside the recovered host proves the loopback works.
A named data owner who is available now, and who agreed before the incident which records they would check.
Read access to the monitoring and notification path for the recovered system: the metrics endpoint, the rule set, and the alert receiver.
Authority to declare the outcome, or direct access to whoever holds it. The three outcomes below are decisions, not observations.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Write down the selected recovery point before you look at the recovered system. Identifier, timestamp with an explicit UTC offset, and who chose it. A recovery point recalled after seeing the result is not a recovery point; it is a rationalisation, and it will agree with whatever you found.
· Confirm the invariant was recorded before the incident, and confirm who supplied it. An invariant derived from the recovered data proves only that the recovered data equals itself. In the capture, the row count and the money-column checksum were written down while the business was still running normally.
· Confirm you are validating from a client position. Curl from inside the recovered host uses the loopback, the host's own resolver and the host's own trust store, none of which the users have. Every reachability and transaction check below is worth less from the host than from the network.
· Confirm the recovered system is reachable by the name clients use, not by an address you were given. Name resolution is the single most common thing that survives an incident unchanged and is therefore wrong.
· Confirm nothing else is still writing to the same data. If the original instance is up and holding the same storage or the same database, two writers are producing a state neither recovery point describes, and validation cannot mean anything.
· Confirm the monitoring for this system is in scope of the validation. Monitoring is configuration like any other; it can be restored, missed, or left pointing at a host that no longer exists. Assume it is broken until it has answered.
· Agree the abort criteria and the three possible outcomes with the incident owner before starting. Deciding what "degraded" means while looking at a failing check produces the answer that ends the call fastest.
· Note the start time. The elapsed validation time is part of the recovery time this service actually has, and it is the part everyone forgets to count.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Name the selected recovery point, and keep it visible for the whole procedure. Everything below compares the recovered system against that one line. Without it, check four has nothing to check against and quietly becomes optional.
2Check that the process is running, and treat it as the weakest evidence you will collect all day. It proves a binary parsed a configuration file and bound a socket. In the measured capture, a copy that was never a consistent image of any instant started cleanly, completed crash recovery, and answered queries.
3Check every dependency from the application's own position. Database, cache, queue, object store, identity provider, outbound egress. A dependency that is unreachable usually lets the application start and fails on its first real request, which is after you have declared it recovered.
4Check the schema and the objects, not the connection. Count tables, indexes, sequences, functions and extensions against the pre-incident inventory. A restore that produced an empty or partial schema still accepts connections and still returns rows — zero of them.
5Prove the recovery point actually restored is the one selected. Read it back out of the recovered system's own log or metadata rather than out of the restore command you typed. The two disagree more often than anyone expects, and the log is the one that is right.
6Check the business invariant recorded before the incident, with both halves. A row count alone survives a restore that lost the values; a checksum alone survives a restore that lost rows in equal and opposite directions. The capture compared rows recovered : 50000 and sum(amount) : 825025000, and needed both.
7Run one end-to-end transaction from a client position: write, read back, then read something that existed before the incident. The write proves the service accepts work. The read-back proves it persisted. The third call proves the restored data is behind the running service rather than an empty schema that happens to answer.
8Repeat the reachability and transaction checks from where the users are, if you have not already. TLS name matching, the trust chain, the route, the proxy and the firewall are all in the path from the client and none of them are in the path from the host.
9Confirm the monitoring for the recovered system works, rather than assuming it survived. The scrape target must be up and recently scraped, the alert rules must be loaded, and a test alert must reach a human who confirms receipt. An unmonitored recovered service is a second incident with a delayed start.
10Ask the data owner to confirm, against records they chose in advance. This is not a technical check and it is not yours to perform. It is the only step that converts "the numbers match" into "the business has its service back".
11Settle the decision explicitly: declare recovered, declare degraded, or roll back. Say which one, in writing, with the checks that supported it. A validation that ends without a named outcome defaults to "recovered" in everyone's memory.
12Publish the RPO statement the business is owed: which committed work is not in the recovered system, expressed as a window with two timestamps. Not "minimal", not "a small amount". The recovery point and the incident timestamp are both known numbers, and the difference between them belongs to the people who have to re-enter it.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The selected recovery point is written down with a timestamp and an owner, and it was written down before the recovered system was inspected.
✓Every dependency answered from the application's position, and the list of dependencies came from the architecture rather than from memory.
✓The schema object counts match the pre-incident inventory, object class by object class, and any difference has a written explanation.
✓The recovered system's own log or metadata names the recovery point, and it is the selected one. In the capture this was recovery stopping before commit of transaction 836, time 2026-08-28 13:34:42.096745+00.
✓Both halves of the invariant reconcile: the count and the checksum, each compared against the number recorded before the incident rather than against the recovered system itself.
✓One end-to-end transaction completed from a client position — write, read-back, and a read of a record that predates the incident — with the HTTP status codes recorded, not summarised.
✓The monitoring path answered: the target is up, the alert rules are loaded, and a named person confirms the test notification arrived.
✓The data owner has confirmed, by name and at a recorded time, against records they chose.
✓One of the three outcomes is recorded, with the checks that supported it, and the RPO statement is published with two timestamps and the work that falls between them.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Undo the reflex mutations first, because they are the ones nobody logged. Alert silences opened during the incident, monitor edits, thresholds relaxed "just for now", and any check pointed at a temporary hostname or port.
↶Remove the probe records the transaction check created. RBDR-PROBE-* rows are real writes to a production dataset, and the data owner will find them during the reconciliation you are about to ask for.
↶Close the test alert you injected, and confirm with the receiver that it cleared. A synthetic alert left firing trains the on-call to ignore the next real one.
↶Revoke any temporary credential, client certificate or firewall exception opened to reach the recovered system from the client position.
↶If the outcome is roll back: stop the recovered system before anything else, so it cannot take further writes that belong to no recovery point, then return to the recovery runbook with the evidence of which check failed.
↶Keep the failed recovery and its source recovery point until the replacement has been validated and has served long enough to trust. Deleting either converts a recoverable mistake into a permanent one.
↶Leave the validation record in place even when the outcome was roll back. The checks that failed are the input to the next attempt and the only honest input to the next recovery time estimate.

6 · Escalation

When the runbook isn't enough, contact:

· The recovered system's log names a recovery point that is not the one selected: escalate to whoever performed the recovery before any further checks. Everything below check four is now describing the wrong data, and repeating it costs time without adding information.
· The invariant does not reconcile and the difference has no explanation: escalate to the data owner and stop. A recovered system serving data nobody can account for is worse than an outage, because the outage is visible.
· The end-to-end transaction fails on the write but the read-back path is healthy: escalate to the application owner. This is the signature of a dependency the recovery did not restore, and it is not a data problem.
· The monitoring path cannot be made to work within the validation window: escalate to the incident owner and to monitoring. Declaring a service recovered while it is unwatched is a decision somebody senior should make explicitly.
· The data owner is unavailable and the incident owner wants the declaration anyway: escalate rather than declaring. Degraded is available as an outcome precisely so that this does not require a lie.
· Validation is exceeding the agreed recovery time objective while the checks are still failing: escalate with the measured remaining work. Whether to serve partial or stale data is a business decision.
· The recovered data shows signs of deliberate tampering rather than accidental loss: escalate to security before declaring anything. Continued writes to the recovered system destroy the evidence of what else the actor reached.

There is a gap between a process running and a service recovered, and almost every premature all-clear lives inside it. The seven checks below close the gap in order. Each one is necessary. Only two of them are sufficient.

The seven checks, and what each is worth

#	Check	What it proves	What it cannot see
1	The process is running	A binary parsed a configuration file and bound a socket	Whether it has any data behind it
2	The dependencies are reachable	The application can talk to what it needs	Whether what it reaches holds the right state
3	The schema and objects are present	The structure was restored	Whether the structure holds any rows
4	The recovery point restored is the one selected	The restore did what it was told	Whether the target itself was right
5	The business invariant still holds	The data matches a property recorded before the incident	Whether the service can act on it
6	An end-to-end transaction succeeds	The whole path works, right through to storage	Whether the business agrees
7	The data owner confirms	The declaration has an owner	Nothing — this is authority, not evidence

Checks one to four can all pass on a system holding entirely the wrong data. Checks five and six are the only pair that cannot: an invariant recorded before the incident cannot be derived from the recovered system, and a transaction that writes, reads back and then reads a pre-incident record exercises every layer at once. Check seven is not evidence at all. It is the signature.

Check 1 costs nothing and proves nothing

Read-only / Safea server that started, recovered, answered, and established nothing— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2). Byte-for-byte from PART A of docs/courses/backup-dr/execution-evidence/pg-application-consistency-and-pitr.txt, four contiguous lines with the capture's own indentation and its blank separator line.

$ start the copy and count what it will hand back

  2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

That copy was made by walking a live data directory with cp while the workload ran. It started, it completed crash recovery, and it returned every row anyone asked for. Nothing about it was a transaction-consistent image of any instant. A practice that usually appears to work, and has no defined failure signal when it does not, is more dangerous than one that fails loudly — and “the process is running” is the check that lets it through.

Checks 2 and 3, from a client position

RBDR_SVC=rbdr-orders.restore.example.internal
RBDR_CA=/etc/ssl/certs/rbdr-internal-ca.pem

getent hosts "$RBDR_SVC"
for dep in rbdr-db:5432 rbdr-cache:6379 rbdr-queue:5672; do
  RBDR_HOST=${dep%%:*}
  RBDR_PORT=${dep##*:}
  if timeout 5 bash -c "exec 3<>/dev/tcp/$RBDR_HOST/$RBDR_PORT"; then
    printf 'reachable %s\n' "$dep"
  else
    printf 'ABORT unreachable %s\n' "$dep"
  fi
done

Then the objects, counted rather than glanced at:

PSQL=(psql -h rbdr-db -p 5432 -U rbdr_app -d rbdr_orders -At -c)
"${PSQL[@]}" "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';"
"${PSQL[@]}" "SELECT count(*) FROM pg_indexes WHERE schemaname = 'public';"
"${PSQL[@]}" "SELECT count(*) FROM pg_sequences WHERE schemaname = 'public';"
"${PSQL[@]}" "SELECT count(*) FROM pg_extension;"

Check 4: the recovery point restored is the one selected

Read the recovery point back out of the system’s own record, not out of the command you typed. In the measured PostgreSQL recovery this is a single line.

Read-only / Safewhere replay actually stopped, in the server's own words— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2). Byte-for-byte from the '--- full recovery log ---' section of pg-application-consistency-and-pitr.txt: three contiguous lines with the capture's two-space indentation preserved.

$ read the stopping point out of the recovered cluster log

  2026-08-28 13:35:12.735 UTC [631] LOG:  recovery stopping before commit of transaction 836, time 2026-08-28 13:34:42.096745+00
2026-08-28 13:35:12.735 UTC [631] LOG:  redo done at 0/52EBC90 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:35:12.735 UTC [631] LOG:  last completed transaction was at log time 2026-08-28 13:34:38.041366+00

Two timestamps bracket the recovered state: replay refused transaction 836 at 13:34:42.096745, and the last commit it did apply was at 13:34:38.041366. That pair is the raw material for the RPO statement below. Elsewhere the equivalent line is a snapshot name, a backup set identifier, or a manifest timestamp — the principle does not change. If the system cannot tell you which recovery point it holds, check four has not been performed.

Checks 5 and 6: the only two that are sufficient

Read-only / Safeboth halves of the invariant, against numbers written down before the incident— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2). Byte-for-byte from PART C of pg-application-consistency-and-pitr.txt, with the capture's indentation. The parenthesised expectations are the capture's own, recorded before the DELETE ran.

$ count rows and sum amounts, then compare with the invariant recorded beforehand

  rows recovered  : 50000   (expected 50000)
sum(amount)     : 825025000   (expected 825025000)
RECOVERED - row count and business checksum both match the pre-DELETE state

Both halves are load-bearing. The count alone passes a restore that kept every row and lost the values; the sum alone passes a restore that lost rows in equal and opposite directions. Together they are a property of the business data that could not have been derived from the recovered database, which is exactly why they mean something. The capture’s own note is the sentence to carry: what was validated is “not that the server started, but that the data it now holds matches an independently recorded property of the business data from before the incident”.

Then one transaction, end to end, from the client position:

RBDR_BASE="https://$RBDR_SVC"
RBDR_REF="RBDR-PROBE-$(date -u +%Y%m%dT%H%M%SZ)"
RBDR_CODE=$(curl -sS --cacert "$RBDR_CA" -X POST "$RBDR_BASE/api/orders" \
  -H 'content-type: application/json' \
  -d "{\"ref\":\"$RBDR_REF\",\"amount\":1.00}" \
  -o /tmp/rbdr-probe.json -w '%{http_code}')
test "$RBDR_CODE" = "201" || { printf 'ABORT create returned HTTP %s\n' "$RBDR_CODE" >&2; exit 1; }
curl -sS --cacert "$RBDR_CA" "$RBDR_BASE/api/orders/$RBDR_REF" | grep -q "$RBDR_REF"
curl -sS --cacert "$RBDR_CA" "$RBDR_BASE/api/orders/ORDER-1001" | grep -q 'ORDER-1001'

The third call is the one that matters. Reading a record that existed before the incident proves the restored data sits behind the running service, rather than an empty schema that answers politely.

Confirm the monitoring, rather than assuming it

RBDR_PROM=http://rbdr-prometheus.example.internal:9090
RBDR_AM=http://rbdr-alertmanager.example.internal:9093

curl -sS "$RBDR_PROM/api/v1/query?query=up%7Bjob%3D%22rbdr-orders%22%7D" | grep -q '"value"'
curl -sS "$RBDR_PROM/api/v1/rules" | grep -q 'RbdrOrdersDown'
amtool --alertmanager.url="$RBDR_AM" alert add RbdrRecoveryProbe \
  service=rbdr-orders severity=warning \
  --annotation=summary='post-recovery notification test'

Three separate things, each of which can be broken independently: the target is being scraped, the rule that would fire is loaded, and the notification reaches a human who says so out loud. Monitoring is configuration, and configuration is restored, missed or left pointing at a host that no longer exists like anything else. A recovered service nobody is watching is the next incident with a head start.

Decision point: recovered, degraded, or roll back

What the checks show	Outcome	What you owe with it
All seven pass	Declare recovered	The RPO statement, and the elapsed recovery time
1-4 pass, the invariant reconciles, the transaction fails on a non-essential path	Declare degraded	The named limitation, the workaround, and who is fixing it
The invariant reconciles but the data owner will not confirm	Declare degraded, keep serving	The owner’s specific objection, in their words
Check 4 names the wrong recovery point	Roll back	Which recovery point was selected, and which one arrived
The invariant does not reconcile and the difference is unexplained	Roll back	The two numbers, side by side

Degraded is a real outcome and it exists so nobody has to choose between an honest answer and a working service. What it is not is a place to file a failed invariant.

Abort criteria

Stop validating, leave the recovered system in place, and escalate when any of these hold. Do not proceed to the later checks in the hope of better news.

Check four names a recovery point other than the one selected. Every check below it is now describing the wrong data.
The invariant does not reconcile and no explanation survives ten minutes of scrutiny.
Another writer is active against the same data. Nothing measured while two writers are running describes any recovery point.
The end-to-end transaction writes successfully but the pre-incident read returns nothing. The service is serving an empty dataset convincingly.
The monitoring path cannot be made to answer and the incident owner will not accept an unwatched service.
The loss looks deliberate. Further writes destroy the evidence, and security decides before validation continues.

The RPO statement you owe the business

Two timestamps and one sentence: the recovered system holds work committed up to 13:34:38.041366 UTC, and the incident began at 13:34:42.096745 UTC; everything committed between those two moments is not present and has to be re-entered. Say which records, how many, and who is doing the re-entry. “Minimal” is not a statement, and the difference between the recovery point and the incident is a number you already have.

What to record

The selected recovery point and who selected it. The recovery point the system reported, verbatim. The seven checks with their results and their timings. Both halves of the invariant, expected and actual. The HTTP status codes from the transaction, not a summary. The monitoring confirmation and the name of the human who received it. The data owner’s confirmation and its time. The outcome declared, and the RPO statement as published. Finally, the total elapsed time, because validation is part of the recovery time this service has, and it is almost never in the plan.

Cross-course references

PostgreSQL for Production Sysadmins, Part XIII (Backup, Archiving and Point-in-Time Recovery) — where the recovery point that check four reads back is chosen, and what the server log records about it.
Observability for Production Sysadmins, Part LXIII (Synthetic Monitoring) — the client-position probe as a standing capability rather than a one-off check performed by hand during an incident.
Linux for Production Sysadmins, Part XLIX (Restore) — the file-level restore mechanics that produce the system this runbook is asked to validate.