Calculate RPO and RTO for three real workloads

Objective

Most recovery objectives are quoted, not derived. Somebody wrote “RPO 15 minutes” on a catalogue page and nobody has asked what schedule would produce it.

In this lab you derive both numbers from three stated architectures: the worst-case loss window for each, an RPO statement that survives being read aloud in an incident review, an RTO as a nine-stage budget, the dominant stage — and then one stage measured with a real restic restore, so at least one line of the budget is observed rather than assumed.

Architecture

RPO is measured backwards from the incident to the last usable recovery point. RTO is measured forwards through stages that add up. They meet at one instant and share nothing else.

flowchart LR
    LP["last recovery point\nW1: 01:00 daily\nW2: last archived WAL segment\nW3: mirror, seconds ago"]
    LP -->|"loss window = RPO"| INC["incident"]
    INC --> D1["detection"] --> D2["decision"] --> D3["provisioning"]
    D3 --> D4["retrieval"] --> D5["transfer"] --> D6["restore"]
    D6 --> D7["replay"] --> D8["validation"] --> D9["cutover"]
    D9 -->|"RTO ends here"| OK["users working again"]

Requirements

• restic 0.18.0 or newer. The transcripts quoted below were captured on restic 0.18.0 and restic 0.19.1.
• About 2 GiB free under $HOME. Task 6 writes 512 MiB and restores it once.
• Every object you create is named rbdr-*, so Cleanup can be scoped and asserted.

Scenario

Three workloads, three architectures, one service catalogue that claims a single number for all of them.

• rbdr-w1-fileserver — a department share. One restic job starts at 01:00 and finishes about 01:26. The repository is copied to a second site at 04:00. No other copies exist.
• rbdr-w2-orders — a PostgreSQL orders database. Base backup at 01:00, plus WAL archiving with archive_timeout set to 60 seconds and each closed segment copied to object storage.
• rbdr-w3-media — a media share held continuously in step with a second building by a mirroring agent.

Tasks

Task 1 — Record the pre-lab state

Cleanup is compared against this file, so capture it before creating anything.

LAB="$HOME/rbdr-rpo-rto"
BEFORE=$(find "$HOME" -maxdepth 1 -name 'rbdr-*' | sort)
mkdir -p "$LAB"
{
  printf 'host: %s\n' "$(hostname)"
  printf 'date: %s\n' "$(date -Is)"
  printf 'restic: %s\n' "$(restic version)"
  printf 'free-kib: %s\n' "$(df -Pk "$HOME" | awk 'NR==2 {print $4}')"
  printf 'rbdr-before:\n%s\n' "$BEFORE"
} | tee "$LAB/00-prelab.txt"

Task 2 — W1: a nightly job, and two different answers

The job starts at 01:00. The earliest instant it covers is 01:00, so that is the recovery point — never 01:26, because a file changed at 01:10 may or may not have been read yet.

awk 'BEGIN {s=(24*3600)-1; printf "host loss, worst case: %d s = %dh %dm %ds\n", s, s/3600, (s%3600)/60, s%60}'
awk 'BEGIN {s=(27*3600)-1; printf "site loss, worst case: %d s = %dh %dm %ds\n", s, s/3600, (s%3600)/60, s%60}'

An incident at 00:59:59 leaves the 01:00 snapshot from the previous day: 86,399 seconds. But the offsite copy only runs at 04:00, so an incident at 03:59:59 that takes the site finds an offsite repository holding the snapshot from 01:00 the previous day: 97,199 seconds.

One workload, one schedule, two RPOs. The failure mode is part of the statement or the statement is not a statement.

Task 3 — W2: what archiving actually bounds

The open WAL segment is not in the archive. Anything committed into it is outside the recovery point. archive_timeout forces a segment switch, so the loss window is that interval plus however long the copy to object storage takes.

This is the worked reference, from the course’s PostgreSQL capture:

$ an unqualified DELETE, then pg_stat_archiver and a listing of the WAL archive

--- and then somebody runs an unqualified DELETE ---
rows after the mistake        : 0
pg_stat_archiver:
  archived=6 failed=0 last=000000010000000000000005
WAL segments in the archive   : 5
  000000010000000000000001
  000000010000000000000002
  000000010000000000000003
  000000010000000000000003.00000028.backup
  000000010000000000000004
  000000010000000000000005

The recovery target that run asked for was 2026-08-28 13:34:40.077562+00. Here is what it actually got:

$ the recovery log lines reporting where redo stopped

  2026-08-28 13:35:12.735 UTC [631] LOG:  recovery stopping before commit of transaction 836, time 2026-08-28 13:34:42.096745+00
2026-08-28 13:35:12.735 UTC [631] LOG:  redo done at 0/52EBC90 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:35:12.735 UTC [631] LOG:  last completed transaction was at log time 2026-08-28 13:34:38.041366+00

awk 'BEGIN {printf "target minus achieved: %.6f s\n", 13*3600+34*60+40.077562 - (13*3600+34*60+38.041366)}'

The achieved point was 2.036196 seconds earlier than the requested one, because recovery stops at the last commit at or before the target. Your RPO is the archive’s coverage, rounded down to a commit boundary — not the number you typed into recovery_target_time.

And there is a second failure mode. While archiving is failing, the loss window is unbounded: it is the age of the last segment that reached the archive, growing by one second per second.

Task 4 — W3: the mirror, honestly

LAB="$HOME/rbdr-rpo-rto"
cat > "$LAB/rpo-statements.txt" <<'EOF'
w1 the appliance provides an RPO of 15 minutes
EOF

That line is the failing case, and Task 5 builds the check that rejects it. First finish the reasoning: against a primary hardware loss the mirror is current, so the loss window is the propagation lag. Against a deletion the mirror is not a backup at all — it applies the deletion, and there is no earlier point to return to. That failure mode has no RPO expressible as a duration, and writing “seconds” for it is the lie this task exists to prevent.

Task 5 — Statements, and a check with teeth

LAB="$HOME/rbdr-rpo-rto"
cat > "$LAB/rbdr-validate.sh" <<'EOF'
#!/usr/bin/env bash
set -u
F="$1"
rc=0
if grep -Eiq '(this|the|our)[[:space:]]+(backup[[:space:]]+)?(solution|product|tool|platform|appliance)[[:space:]]+(has|provides|gives|offers|delivers)[[:space:]]+an?[[:space:]]+(RPO|RTO)' "$F"; then
  echo "FAIL attribution: an RPO or RTO is credited to a product"
  rc=1
fi
for field in failure_mode= mechanism= schedule= worst_case_loss=; do
  if ! grep -q "$field" "$F"; then
    echo "FAIL statement: no $field field"
    rc=1
  fi
done
if [ "$rc" -eq 0 ]; then echo "OK statements"; fi
exit "$rc"
EOF
chmod +x "$LAB/rbdr-validate.sh"
"$LAB/rbdr-validate.sh" "$LAB/rpo-statements.txt"; echo "exit: $?"

That prints the attribution failure and exit: 1. Now write the six real statements:

LAB="$HOME/rbdr-rpo-rto"
cat > "$LAB/rpo-statements.txt" <<'EOF'
w1 failure_mode=host-loss mechanism=nightly-restic-snapshot schedule=01:00-daily worst_case_loss=86399s
w1 failure_mode=site-loss mechanism=nightly-snapshot-then-offsite-copy schedule=01:00+04:00-daily worst_case_loss=97199s
w2 failure_mode=host-loss mechanism=base-backup-plus-archived-wal schedule=01:00-daily+archive_timeout-60s worst_case_loss=60s-plus-archive-copy-latency
w2 failure_mode=archiver-stalled mechanism=base-backup-plus-archived-wal schedule=unbounded-while-failing worst_case_loss=age-of-last-archived-segment
w3 failure_mode=primary-hardware-loss mechanism=continuous-mirror schedule=continuous worst_case_loss=mirror-propagation-lag
w3 failure_mode=logical-deletion mechanism=continuous-mirror schedule=continuous worst_case_loss=no-recovery-point-exists
EOF
"$LAB/rbdr-validate.sh" "$LAB/rpo-statements.txt"; echo "exit: $?"

Task 6 — The RTO budget, and its dominant stage

Budget rbdr-w2-orders, 240 GiB, recovered onto a replacement host.

LAB="$HOME/rbdr-rpo-rto"
cat > "$LAB/rto-budget.csv" <<'EOF'
stage,minutes,source
detection,6,alert-history
decision,15,estimated
provisioning,20,estimated
retrieval,12,estimated
transfer,35,estimated
restore,48,estimated
replay,22,estimated
validation,10,estimated
cutover,8,estimated
EOF
awk -F, 'NR>1 {t+=$2; if ($2+0>m+0) {m=$2; s=$1}}
  END {printf "total: %d min (%.2f h)\ndominant: %s at %d min (%.1f%%)\n", t, t/60, s, m, 100*m/t}' \
  "$LAB/rto-budget.csv" | tee "$LAB/rto-total.txt"

Total 176 minutes; restore dominates at 48. Eight of the nine stages are guesses.

Task 7 — Measure the stage that dominates

LAB="$HOME/rbdr-rpo-rto"
export RESTIC_REPOSITORY="$HOME/rbdr-repo"
export RESTIC_PASSWORD='rbdr-lab-disposable'
mkdir -p "$HOME/rbdr-w1-data"
dd if=/dev/urandom of="$HOME/rbdr-w1-data/rbdr-payload.bin" bs=1M count=512 status=none
restic init
restic backup "$HOME/rbdr-w1-data"
T0=$(date +%s.%N)
restic restore latest --target "$HOME/rbdr-w1-restored"
T1=$(date +%s.%N)
awk -v a="$T0" -v b="$T1" 'BEGIN {d=b-a; printf "restore_seconds=%.2f\nrestore_MiB_s=%.1f\n", d, 512/d}' \
  | tee "$LAB/measured-restore.txt"
RESTORED=$(find "$HOME/rbdr-w1-restored" -name 'rbdr-payload.bin' -print -quit)
cmp "$HOME/rbdr-w1-data/rbdr-payload.bin" "$RESTORED" && echo "restored bytes match"

Fold the measured rate into the budget and recompute:

LAB="$HOME/rbdr-rpo-rto"
MEASURED=$(awk -F= '/restore_MiB_s/ {printf "%.0f", (240*1024)/$2/60}' "$LAB/measured-restore.txt")
awk -F, -v m="$MEASURED" 'BEGIN {OFS=","} NR==1 {print; next}
  $1=="restore" {print $1, m, "measured-rate-this-host-only"; next} {print}' \
  "$LAB/rto-budget.csv" > "$LAB/rto-budget-v2.csv"
awk -F, 'NR>1 {t+=$2; if ($2+0>m+0) {m=$2; s=$1}}
  END {printf "total: %d min\ndominant: %s at %d min\n", t, s, m}' \
  "$LAB/rto-budget-v2.csv" | tee "$LAB/rto-total-v2.txt"

Compare the two dominant: lines. If the measured stage is no longer the largest, the dominant stage has moved — which is the whole point: you cannot know which stage dominates until at least one of them has been observed. The measured rate came from local storage with no network in the path, so it bounds the restore stage from below and says nothing about the others. The reason to measure at all is this:

$ a first backup, an unchanged second backup, and a restore of the same 400 MiB

--- the asymmetry that IS large, and is the point of the exercise ---
a second backup of 'big' with nothing changed:
backup  : .73s

first backup of 400 MiB : 1.49s
second backup, unchanged: .73s
restore of 400 MiB      : 1.07s

Validation

Each line names the command, the string to expect, and the exit code.

LAB="$HOME/rbdr-rpo-rto"
test -s "$LAB/00-prelab.txt"; echo "prelab exit: $?"
awk -F, 'NR>1 {t+=$2} END {print t}' "$LAB/rto-budget.csv"
grep -c . "$LAB/measured-restore.txt"
"$LAB/rbdr-validate.sh" "$LAB/rpo-statements.txt"; echo "statements exit: $?"
grep -c 86399 "$LAB/rpo-statements.txt"
printf 'w9 the appliance provides an RPO of 15 minutes\n' > "$LAB/rbdr-bad.txt"
"$LAB/rbdr-validate.sh" "$LAB/rbdr-bad.txt"; echo "bad exit: $?"

1. test -s .../00-prelab.txt prints prelab exit: 0.
2. The budget sum prints exactly 176; exit code 0. Any other number means a stage row was edited or lost.
3. grep -c . on measured-restore.txt prints exactly 2; exit code 0.
4. The statements check prints OK statements; exit code 0.
5. grep -c 86399 on the statements file prints 1; exit code 0. If it prints 0, Task 2’s arithmetic never reached the statement.
6. The failing case prints FAIL attribution: an RPO or RTO is credited to a product as its first line, four FAIL statement: lines after it, and bad exit: 1. A check that never rejects anything has not been shown to work.

Expected Outcome

Six RPO statements, each naming a failure mode, a mechanism and a schedule. Two of the six carry no duration at all, correctly.

An RTO budget of 176 estimated minutes with restore dominant at 48, and a revised budget in which that one line came from a stopwatch.

Record both of these from your own run:

• Actual restore time: the restore_seconds value in measured-restore.txt.
• Actual RPO observed: for the restic snapshot you took in Task 7, the interval between the last write to rbdr-payload.bin and the snapshot timestamp reported by restic snapshots. It is not zero.

Troubleshooting

Fatal: unable to open config file from restic init. RESTIC_REPOSITORY is unset or points at a path that does not exist. Both exports in Task 7 are required in the same shell.

restic restore runs but cmp reports a difference. It almost certainly did not: check the target path first. restic restore --target recreates the full absolute source path underneath the target, so the file lands at $HOME/rbdr-w1-restored$HOME/rbdr-w1-data/.

The budget sum prints something other than 176. A CSV row was edited, or a minutes field acquired a unit. The awk sum treats a non-numeric field as zero rather than failing.

restore_MiB_s is implausibly large. date +%s.%N returned the same value twice because the restore completed inside the timer’s resolution. Raise the count= on dd until the restore takes more than a second.

The statements check passes on a file you know is wrong. It only tests for four field names and one attribution pattern. It cannot check whether 97199s follows from the schedule; only Task 2’s arithmetic does that.

Cleanup

LAB="$HOME/rbdr-rpo-rto"
sed -n '/^rbdr-before:/,$p' "$LAB/00-prelab.txt" | tail -n +2 | sed '/^$/d' > /tmp/rbdr-before.txt
rm -rf "$HOME/rbdr-w1-data" "$HOME/rbdr-w1-restored" "$HOME/rbdr-repo" "$LAB"
unset RESTIC_REPOSITORY RESTIC_PASSWORD
find "$HOME" -maxdepth 1 -name 'rbdr-*' | sort | sed '/^$/d' > /tmp/rbdr-after.txt
diff /tmp/rbdr-before.txt /tmp/rbdr-after.txt && echo "CLEAN: rbdr- inventory matches Task 1"
rm -f /tmp/rbdr-before.txt /tmp/rbdr-after.txt

diff exiting 0 with CLEAN printed is the proof. A non-empty diff names the object still present.

Production notes

• Publish RPO per workload and per failure mode. W1 alone needs two rows; a single catalogue number hides the worse one.
• Alert on the archiver, not the database. W2’s bounded RPO holds only while segments keep reaching the archive; the moment they stop, the loss window grows by one second per second with no other symptom.
• Re-measure the restore stage whenever the dataset, the storage or the target host changes. A backup job’s reported duration does not predict it, and the gap widens as the data grows and the deltas do not.
• Budget detection and decision honestly. Together they were 21 of the 176 minutes here, and they are the stages nobody rehearses.
• A recovery point that exists only on the protected host is not a recovery point for the failure that takes the host.

What You Learned

• An RPO belongs to a failure mode, not to a workload. W1 had two, W2 had two, W3 had one duration and one honest “there is no point to return to”.
• The recovery point is the earliest instant the job covers, and for PITR it rounds down to the last commit at or before your target — 2.04 seconds earlier in the captured run.
• A bounded RPO is a claim about a mechanism still working. A stalled archiver turns 60 seconds into whatever the outage lasts.
• RTO is a sum of stages you can name, and until you have measured one you have a wish, not a budget.
• No product supplies either number. A schedule and a topology do.