Restore-point age, not job success

The previous lesson enumerated the signals a backup estate already emits and asked what each one is a statement about. Almost all of them turned out to be statements about a runner: a timer fired, a script started, a process exited with a number. This lesson makes one substitution to the primary signal — report the age of the newest recovery point instead of the outcome of the last job — and that change removes a class of silent failure, because the new number cannot be produced by anything except the artefact you would restore from.

A job result describes the runner; an age describes the exposure

The two measurements look interchangeable. Both go bad in the same direction, both are collected nightly, both end up on the same dashboard. They differ in where they come from, and that is the whole difference.

A job result is produced by the process doing the work. It is a claim the runner makes about itself, carried through whatever instrumentation the runner happens to have: an exit status, a log line, a textfile, a webhook. Every stage of that path is a component that can be wrong, and none of the stages touch the stored data.

A recovery point age is produced by asking the destination what it holds. It is the current time minus the timestamp of the newest snapshot covering a given system, in a given copy. The only way to make that number smaller is to write a snapshot the repository accepted and indexed — which is also, not coincidentally, the only way to make a restore of recent data possible. No exit status appears anywhere in the derivation.

Restated in the terms this course has used since Part II: job success answers “did a process run”, while recovery point age answers “how much of this system’s recent history exists nowhere I can recover it from”. The second is the quantity the business agreed on when it wrote an RPO, so the substitution replaces a proxy with the measured quantity itself.

The failure modes that follow differ in kind. Under job success, an estate fails by staying green while the exposure grows without bound, and nobody notices because there is no number to notice. Under recovery point age that mode does not exist: when backups stop, the number climbs at exactly one second per second, and the only remaining question is where the line should be drawn. A job result is also per job, while an age is per system and per copy — and a system covered by no schedule emits no success signal at all, which is precisely the case the last section of this lesson is about.

Reading the age off the repository, per system rather than per job

Obtaining the number is a listing operation, not a log parse. The repository already knows when it last accepted data, because it wrote an entry to record it.

$ restic snapshots

ID        Time                 Host          Tags        Paths       Size
-------------------------------------------------------------------------------
3fe43af4  2026-08-28 13:27:02  8211a08b55c3  daily       /work/prod  60.000 MiB
3e349a12  2026-08-28 13:27:03  8211a08b55c3  daily       /work/prod  60.000 MiB
-------------------------------------------------------------------------------
Timestamps shown in local time
2 snapshots

Look at what the columns are keyed by. Host and Paths identify what was protected, and Tags carries whatever label the schedule attached — here, daily. There is no column for the name of the job, the wrapper script or the runner, because the repository does not know any of those things. The listing is addressed the way a restore is addressed: by the system whose data it is, and by the path the data came from.

Two rows one second apart illustrate the second rule. The newest snapshot here is 3e349a12 at 2026-08-28 13:27:03, not the row printed first. Take the maximum timestamp within a group, never the first or last row of the output; print order is a property of the tool, and an alert must not depend on it.

The footer states Timestamps shown in local time. The same capture’s restore banner recorded that snapshot’s time as 2026-08-28 13:27:02.65376235 +0000 UTC, and the two agree only because the container’s local zone happened to be UTC. An age is a subtraction between two clocks and both operands must be in the same frame, so parse the machine-readable output documented under scripting rather than the human table:

REPO=/work/repo
HOST=8211a08b55c3
newest=$(restic -r "$REPO" snapshots --json \
  | jq -r --arg h "$HOST" '[.[] | select(.hostname == $h) | .time] | max')
age=$(( $(date +%s) - $(date -d "$newest" +%s) ))
printf 'recovery_point_age_seconds{system="%s"} %s\n' "$HOST" "$age"

The grouping that matters in production is per system and per copy, not per job. A host whose local repository accepted a snapshot twenty minutes ago and whose offsite repository last accepted one forty hours ago has two ages, not one. Reporting only the minimum answers “is something recent somewhere”, which is not the question either copy exists to answer: the offsite copy exists for the failure that takes the local one with it, and its own age is the only number that describes that exposure. Emit one series per system and copy, and let each carry the threshold that copy is held to.

A job-oriented view cannot express that cleanly, because the mapping from jobs to systems is neither one-to-one nor stable. One schedule often covers several paths on several hosts, one host is often covered by several schedules, and jobs are renamed, split and merged while the systems they protect stay the same.

The wrapper whose exit status is the exit status of its last command

A backup job in production is rarely a bare command. It is a script: take a lock, quiesce something, run the backup, release the lock, apply retention, write a log line, notify a channel. The scheduler above it sees exactly one number from that script, and the shell decides which number that is.

The same reasoning covers the case where the runner is not merely wrong but absent. A disabled timer, a decommissioned scheduler host or a job removed during a migration emits no failure — it emits nothing. Most alerting written against job results has the shape “page when the last result is a failure”, and no result is not a failure. An age has no equivalent hiding place, because the clock keeps running whether or not anything is scheduled.

The threshold is the RPO, so the metric encodes the objective

Choosing an alert threshold on job success is an exercise in invention. “Page when the seven-day success rate drops below 95%” — where does 95 come from? It is derivable from nothing the business agreed to, and it quietly encodes a loss window nobody chose, because the rule fires only after several consecutive misses have accumulated.

An age has no such freedom. The threshold is the RPO, restated in the units the metric is already in. A system with a four-hour RPO alerts when its recovery point age exceeds four hours. A system with a one-day RPO alerts at one day. The threshold is not tuned; it is transcribed.

That is the property worth pausing on. Because the number in the rule is the objective, reading the alert rules tells you what each system was promised, in the units the promise was made in, and a system whose threshold nobody can state is a system whose objective was never agreed. The service tiers from Part II become the threshold table directly, with no second decision that could drift. Neither figure is a property a product supplies: the RPO comes from the tiering decision, the age comes from the repository, and a tool contributes only the listing.

Expressing the join explicitly keeps the objective visible in the rule itself, where the RPO is emitted as its own series from the inventory:

- alert: RecoveryPointOlderThanRPO
  expr: |
    (time() - recovery_point_timestamp_seconds)
      > on (system, copy) rpo_seconds
  for: 10m
  labels:
    severity: page
  annotations:
    summary: 'recovery point for {{ $labels.system }} ({{ $labels.copy }}) is older than its RPO'

One arithmetic consequence is where teams are tempted to cheat. The age is a sawtooth: it climbs from the instant a snapshot completes and drops when the next one completes, so its peak — immediately before the next completion — is the schedule interval plus the run duration. The schedule meets the objective only when interval plus duration stays under the RPO, and if that arithmetic does not close the alert fires correctly.

The systems that have no age at all

The most valuable thing a recovery point age does is fail to exist.

A system nobody backs up produces no snapshot, therefore no newest snapshot, therefore no age. Under a job success rate that same system also produces no failing job: it contributes nothing to the numerator and nothing to the denominator, because the rate is computed over jobs that exist. A success rate of 100% is entirely compatible with a production system that has never had a recovery point, and the arithmetic is worse than neutral, because deleting a chronically failing job raises the reported rate.

Fixing this requires computing the metric against the inventory rather than against the repository alone. Enumerating repositories answers “what do we have”, and no amount of that can surface “what is missing”. The join runs the other way: for every system in the inventory built in the previous lesson, and for every copy it is supposed to have, look for the newest snapshot; a row with no match is not an empty panel but an alert. Emitting the RPO series from the inventory makes the second rule a one-liner, because the objective exists for systems whose recovery point does not:

- alert: NoRecoveryPointAtAll
  expr: |
    rpo_seconds
      unless on (system, copy) recovery_point_timestamp_seconds
  for: 30m
  labels:
    severity: page

Between the two extremes sits the case an age surfaces earliest. A system whose backups have stopped keeps its series, and the series does something visually unmistakable: it goes flat at the last successful value and then ramps at exactly one second per second, forever. That ramp is the most recognisable shape on a backup dashboard, and it appears long before the threshold is crossed. A review that reads slope rather than current value catches a stalled system within one interval of it stalling.

One honest limit closes the section. An age states that a snapshot exists with that timestamp; it does not state that the snapshot reads back. On the repository this course keeps returning to, restic check reported no errors with exit code 0 after ten bytes were overwritten in a data pack, and the next restore returned 6 of 7 files — while the recovery point age stayed perfectly healthy, because the snapshot really was there. Age bounds the exposure; verification and restore testing establish that the point is usable, and the remaining lessons in this part build that second signal.

Production discipline

1. Read the age out of the repository, never out of the runner. The measured listing printed 2026-08-28 13:27:02 and 2026-08-28 13:27:03 for host 8211a08b55c3 on /work/prod; the newest of those, grouped by host and path, is the number. Take the maximum in the group, not the first row, and parse the machine-readable form because the table prints local time.
2. Emit one series per system and per copy, keyed the way a restore is addressed. A host with a twenty-minute local age and a forty-hour offsite age has two exposures, and the minimum describes neither. Host and Paths are what the repository records and what the restore will use; job names are not a stable key.
3. Set the threshold to the RPO and change the schedule, not the threshold. Four hours for a four-hour objective, one day for a one-day objective, taken straight from the service tier. When interval plus run duration exceeds the RPO the alert is correct, and raising the threshold re-negotiates a commitment in a file nobody outside the team reads.
4. Compute the metric from the inventory, so a missing series is itself an alert. A system with no schedule produces no failing job and no age, a 100% success rate cannot represent it, and deleting a broken job improves that rate. Emit the RPO for every system that should hold a copy, and alert on objectives with no recovery point beside them.
5. Keep the age next to a verification signal, because it does not prove readability. On the same restic 0.19.1 capture, restic check reported no errors were found with exit code 0 after a data pack was corrupted, and the restore that followed reported Restored 6 / 7 files/dirs (59.401 MiB / 60.000 MiB) and exited 1. The age was green for all of it.

Cross-course references

• Observability for Production Sysadmins — Part XCIX (Missing Metrics) treats the general form of this lesson’s most important case: a series that does not exist cannot cross a threshold, so absence has to be alerted on deliberately rather than watched for on a dashboard. That is exactly the mechanism behind the unless rule above, and the reason a recovery point age must be computed from the inventory rather than from the repositories.
• Linux for Production Sysadmins — Part XXXV (Shell Scripting for Sysadmins) covers exit status, pipeline status and set -o pipefail as language features. This lesson is the operational consequence: a backup wrapper reports the status of its last command, so a mis-written script is biased towards green, and an age read from the artefact is immune to every variant of the mistake.
• PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) supplies the case where the recovery point is not the backup at all: with continuous archiving the newest recoverable point is the last archived WAL segment, so the age must be read from the archive rather than from the last base backup. The per-system, per-copy grouping argued here is what keeps those two ages from collapsing into one meaningless number.

Quiz