Recovery SLOs: measuring the promise you made · Backup & DR

A dependency graph produces an order, a critical path and a list of bootstrap steps, and every one of them is a prediction. So is the loss window derived at the start of this part, so is the restore duration estimated from throughput, and so is the tier a service was assigned. Part II has produced a complete set of commitments and not one observation. An RPO and an RTO that are never measured are aspirations with units attached, and the first measurement is then taken by an incident, with no baseline to compare it against.

Three quantities, and the backup dashboard reports none of them

The first is observed data loss. For each restore test and each real recovery, record the interval between the recovery point actually achieved and the moment the data was needed, and where the business counts in units of its own — orders, messages, sensor readings — record the loss in those units too. It is a per-event figure and it should stay one. Averaging it destroys the information that matters, because nobody experiences the mean loss; they experience the incident they got, and the tail is the interesting part.

The second is measured restore duration, end to end. The clock starts where the business believes it starts, which is the fault or the decision to recover, not the moment somebody types the restore command, and it stops when the service is serving correct results that somebody has verified. Record the total and the stage breakdown together: a total alone says whether the commitment was met and nothing about where the hours went, and a miss whose stage nobody can name cannot be improved on purpose.

The third is the proportion of restore tests that succeeded on the first attempt. Of the tests attempted in a window, how many produced a verified correct result without a second run, an undocumented manual step, or a call to the one person who knows the trick? This is the closest thing the discipline has to a leading indicator, because it degrades before a real recovery does. A restore that succeeded on the fourth attempt after two hours of improvisation is not a pass with an asterisk; it is a failed test that eventually produced data, and scoring it as a pass removes the only signal the test generated.

All three share the property that explains why they are so rarely reported: each can be produced only by attempting a recovery, which costs machine time, people and a real target to restore onto.

Measured: the metric that cannot move

What estates report instead is backup job success percentage, for good reasons: the scheduler already produces it, there is a value every day rather than every quarter, and it aggregates across hundreds of systems into one figure a manager can read. In every operational sense except the one that matters, it is an excellent metric.

What it observes is one property of one process: that a program started at a scheduled time and exited with a status the wrapper treated as good. Everything that determines whether a recovery is possible happens either inside data the process never re-reads, or after the process has already exited. The restic capture makes that gap concrete. A repository holding two snapshots of a 60 MiB source tree had ten bytes overwritten in the middle of its largest data pack — 17374653 bytes before the damage and 17374653 bytes after it, with mtime the only filesystem-visible change. Then the check that most schedules run:

Read-only / Saferestic check, on a repository that has already lost a file— EXECUTION VALIDATED on restic 0.19.1, from the repository-integrity and restore capture, Step 7, after ten bytes were overwritten in the largest data pack.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0

A monitoring system scraping that job records a success. So does the next night’s, for as long as nobody reads the data. The same repository, checked by the command that reads and re-hashes every pack, answers differently:

Service impact possiblerestic check --read-data, same repository, same minute— EXECUTION VALIDATED on restic 0.19.1, from Step 7 of the same capture, executed immediately after the command above.

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
>>> exit code: 1

And the quantity this lesson asks you to report — what a recovery actually produced — agrees with the second command and not with the first:

Data-loss riskthe restore, and an independent check of what came back— EXECUTION VALIDATED on restic 0.19.1, from Step 8 of the same capture. The source tree was three files totalling 60.000 MiB, checksummed before the backup was taken.

$ restic restore 3fe43af4 --target /work/restore2

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore2
ignoring error for /work/prod/db/data.bin: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed
Summary: Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) in 0:00
Fatal: There were 1 errors
>>> exit code: 1

--- verifying whatever was restored, file by file ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

Compute the series over the month containing that damage. Backup job success: 100%. Check success, from the cheap check: 100%. Observed recovery capability: two of three files, 59.401 MiB of 60.000 MiB, and one file returned with a plausible name, a plausible size and the wrong contents. The first two numbers are not wrong. They answer a question nobody asked, and no arithmetic on them produces the third.

What a success ratio observes, and why a check metric needs a check label

A ratio inherits all its meaning from the event it counts. Backup job success is successful runs divided by scheduled runs, where “successful” is a predicate the wrapper evaluates over a process exit status at the instant the process ends.

That predicate is genuinely sensitive to a real class of failure: the tool not starting, credentials rejected, the destination unreachable, the source unreadable, the target out of space. Catching those is why the metric exists. It is structurally blind to everything else, and the blindness is not an implementation defect. Bytes that decay after the writing process exited are outside its observation window by construction, and so is a key that can no longer be produced, a retention rule that removed a parent an incremental chain needs, the absence of hardware to restore onto, and a pack whose ciphertext no longer verifies.

The label problem follows directly. A sample is a metric name, a set of labels and a value, and the labels are the only place the predicate’s meaning can live. restic check and restic check --read-data are different predicates over the same repository. On the healthy repository in Step 5 of the capture both reported no errors were found with exit code 0; after the damage in Step 7 they diverged, 0 against 1, with nothing else changed. A series called backup_check_success carrying no label to distinguish them is therefore uninterpretable: the value 1 could mean either claim, and only one of them is about the data.

The cost asymmetry makes the label mandatory rather than tidy. Re-hashing every pack costs roughly what a restore costs because it performs the same I/O, so it runs on a long cadence while the structural check runs nightly. An unlabelled mixture is dominated by the cheap samples, and a rare expensive failure is diluted into invisibility by exactly the samples that could not have detected it. Export which check produced the sample, and beside it the age of the last completed data verification — a green structural check whose read-data companion ran six months ago is a statement about six months ago.

Four series that make the three quantities observable

The three quantities are events, and events are awkward to alert on directly. The usual instrumentation is four series carrying timestamps and outcomes, one sample per protected system, exported through a text file a job rewrites atomically.

Recovery point timestamp, per protected system. The timestamp of the newest data a recovery could actually produce — read from the destination, never inferred from the source’s schedule. The newest snapshot as the repository itself lists it; the last log segment the archive confirms it holds; the last object the bucket acknowledges. Its age is the live recovery point, and this series is what turns the RPO from a design statement into an observed value.

Restore test timestamp, per protected system. When the last restore test finished. Its age is restore test age, and a system untested inside the review interval is unproven — a reportable state with a name, not a blank cell. Report the count of unproven systems beside the success ratio, because a ratio computed over three tested systems out of ninety is a statement about three systems.

Restore test outcome, with the verification named. A boolean is not enough, because “the restore command exited 0” and “the restored data matched an independently recorded property of the source” are different claims. The capture shows why: the restore exited 1 and the checksum comparison reported ./db/data.bin: FAILED. Carry a label for what was restored and one for how it was verified.

Restore duration, per system and per stage. Measured, not estimated, and broken into the stages from the previous lessons so the total is a sum you can attack rather than a number you can only regret.

TEXTFILE_DIR=/var/lib/metrics/textfile
SYSTEM=orders-db
LAST_TEST_EPOCH=1755741600
LAST_READDATA_EPOCH=1755741600
LAST_STRUCTURE_EPOCH=1756346400
OUT="$TEXTFILE_DIR/recovery_${SYSTEM}.prom"
{
  printf 'restore_test_timestamp_seconds{system="%s"} %s\n' "$SYSTEM" "$LAST_TEST_EPOCH"
  printf 'restore_test_outcome{system="%s",attempt="first",verified_by="checksum"} %s\n' "$SYSTEM" 0
  printf 'repo_check_timestamp_seconds{system="%s",check="read-data"} %s\n' "$SYSTEM" "$LAST_READDATA_EPOCH"
  printf 'repo_check_timestamp_seconds{system="%s",check="structure"} %s\n' "$SYSTEM" "$LAST_STRUCTURE_EPOCH"
} > "$OUT.tmp"
mv -- "$OUT.tmp" "$OUT"

The resulting file says what no job log can, including which of the two checks each sample came from:

restore_test_timestamp_seconds{system="orders-db"} 1755741600
restore_test_outcome{system="orders-db",attempt="first",verified_by="checksum"} 0
repo_check_timestamp_seconds{system="orders-db",check="read-data"} 1755741600
repo_check_timestamp_seconds{system="orders-db",check="structure"} 1756346400

Two disciplines keep this useful. Export timestamps rather than pre-computed ages, so a stale exporter produces an age that visibly grows instead of a frozen value that looks current. And keep the label set bounded: a recovery metric that becomes a cardinality incident gets dropped, and you are back to job success.

Anti-pattern: the restore test that cannot fail

Two arrangements produce a first-attempt success ratio of 100% that means nothing at all.

The same warm sample, every time. A 2 GB subset restored to a host already provisioned, already holding the encryption key, already on the repository’s network. It exercises the restore command and none of the prerequisites the previous lesson’s graph identified, and it passes until the day the key, the target or the network is what is missing.

“Snapshots are enough.” The anti-pattern here counts a rollback on live storage as a restore test. It proves the storage layer can revert and proves nothing about the copy held elsewhere — the copy that is the only one left in the failure model the objective was written against. A test that never touches the artefact a real recovery would read is not a restore test with limitations; it is a different test with a misleading name.

A target, an error budget, and what a divergence must trigger

A measurement without a target is a number, and a target without a budget is an argument waiting to happen. Both are stated per system and per failure mode.

For the orders database described earlier in this part — a base backup plus continuous archiving, stated against loss of the primary’s storage — a workable pair reads: recovery point age stays under fifteen minutes for 99% of measurement intervals over a rolling thirty days. The 1% is the error budget, and it is 7.2 hours of exceedance per window. Those figures are not properties of the archiving mechanism; they are a commitment the business made, expressed in a form the four series above can falsify.

Restore testing takes the same shape. If twenty restore tests are scheduled per quarter across the estate and the target is 95% first-attempt success, the budget is one failure per quarter — an uncomfortably small number, which forces an honest conversation about whether 95% was a considered figure or a round one. Restore duration is usually stated as a per-event bound instead, with the budget expressed as how many rehearsals may exceed the committed RTO before the commitment is treated as unmet.

The budget earns its place by giving the number a burn rate. A hard threshold flips on the day of the breach; a budget is consumed visibly, so half of it gone by the second week is a signal while there is still time to act. It also forces the question people avoid: what happens at zero? A budget with no defined consequence is decoration, and the consequences that work cost somebody something — a freeze on that system’s backup configuration until a test passes, a mandatory rehearsal, or the tier reduced on paper so the register stops claiming a protection level nobody can demonstrate.

When the measured number diverges from the committed one, there are exactly three legitimate responses. Change the architecture until the measurement meets the commitment. Change the commitment to match what is measurable and have whoever owns the consequence accept the new figure in writing. Or record the divergence as accepted risk, with a named owner and a review date. Choosing between them is not a technical decision, which is why it escalates to the owner of the commitment rather than around the backup team.

There is a fourth response, and it happens by default: the committed number stays on the slide, the measured number stays in a spreadsheet, and nobody puts them in the same sentence. That estate has not failed to meet its objectives. It has failed to notice, which is worse, because the divergence is then discovered at the same moment as the outage and by the same people.

Production discipline

Report the three quantities a recovery produces, not the one a job produces. Observed data loss per event, end-to-end restore duration with its stage breakdown, and first-attempt restore success — with a test that needed a second attempt recorded as a failure.
Label every check metric with the check that produced it and the age of the last full verification. restic check reported no errors were found with exit code 0 on a repository where restic check --read-data exited 1 and named pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f; an unlabelled series cannot tell those two claims apart.
Read the recovery point from the destination, and export timestamps rather than ages. The newest snapshot the repository lists, the last segment the archive confirms, the last object the bucket acknowledges — as a timestamp, so a stalled exporter shows an age that grows rather than a value that looks fresh.
Count a restore test as passed only against an independent property of the source. The measured restore returned 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and the checksum comparison reported ./db/data.bin: FAILED; a test that checks only that files appeared would have recorded a pass.
Give every target an error budget and a defined consequence at zero, then escalate divergence to the owner of the commitment. A 99% target over thirty days is 7.2 hours of exceedance; when it is spent, either the architecture changes, the commitment changes in writing, or the gap is recorded as accepted risk with a name and a date against it.

Cross-course references

Observability for Production Sysadmins — Part XXII (SLO-Based Alerting) supplies the machinery this lesson assumes: targets over a rolling window, error budgets and burn rates, and alerts that fire on budget consumption rather than on a single sample. The four recovery series above are the indicators that material operates on, and without them a recovery SLO has no signal to compute against.
Git, CI/CD & GitOps for Infrastructure Engineers — Part LXIII (CI/CD Observability) is where a scheduled restore test becomes an instrumented pipeline whose outcome, duration and stage timings are emitted as data rather than read from a log — the delivery mechanism for the restore test age, outcome and per-stage duration series described here.
PostgreSQL for Production Sysadmins — Part XVI (Observability, Logging and Alerting) covers the database-side counters that feed the recovery point timestamp for a cluster, making the archive’s own view of what it holds the source for the live RPO series instead of the backup job’s exit status.