Alerting on recovery capability

Restore point age replaced job success in the previous lesson as the number worth publishing, and a number on a dashboard is read during office hours by someone who chose to look. An alert is different. It arrives unbidden, usually at three in the morning, to a person who was asleep four seconds earlier and who has to decide within a minute whether this is worth waking anyone else for. The design constraint is brutal and clarifying: an alert about recovery capability has to name a gap, name a consequence, and name an action, or it is better not sent at all.

The three ages and the three different repairs

The whole of recovery assurance compresses into three numbers per protected system. They look similar — each is a duration since a timestamp — and they are not interchangeable, because each one decays for a different reason and each one is repaired by different people doing different work.

The newest recovery point is the time since the most recent snapshot that exists and is addressable in the repository. It answers how much data is at risk right now. When it goes stale, something in the capture path has stopped: an agent, a timer, a credential, a full destination. The damage accumulates while the age grows, which makes this the only one of the three whose cost is still increasing at the moment the alert fires.

The newest completed data verification is the time since a check that actually read the stored bytes returned success. It answers whether the copy is still intact. When it goes stale, nothing is currently breaking — but the window in which corruption could have arrived undetected is now as wide as the age itself. The honest statement about a repository last verified 190 days ago is that it was intact 190 days ago.

The newest proven restore is the time since data was restored from that repository and compared against a known-good reference. It answers whether the whole path works: keys reachable, target available, procedure correct, people able to execute it. When it goes stale, everything outside the backup tool is unmeasured, and that is where recoveries usually fail.

A mature estate almost always presents the same shape: age one is minutes, age two is months, age three is never. That shape is not a monitoring failure. It is an accurate report of where the evidence actually is, and the reason to put all three ages side by side is that the fresh number stops standing in for the missing ones.

Measured: three timestamps that do not substitute for each other

Each age comes from a different command, and the commands are not equivalent even when their exit codes agree. A restore that is verified against checksums is the strongest evidence available, and it updates age three.

$ restic restore 3fe43af4 --target /work/restore

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

Seven files, 60.000 MiB, three checksums matching, two exit codes of zero. That event is worth a timestamp, and the timestamp is worth alerting on when it gets old. A verification that reads every pack touches the stored bytes the way a restore does and still proves less, because it never touches the target infrastructure, the written procedure or the people. It updates age two, not age three.

$ restic check --read-data

using temporary cache in /tmp/restic-check-cache-1847567736
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
read all data
[0:00] 100.00%  7 / 7 packs
no errors were found
>>> exit code: 0

The trap is which command’s timestamp you record. Ten bytes were then overwritten in the middle of the largest data pack — the file kept its name, its size of 17374653 bytes and its place in the directory listing — and the default check was run again.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found
>>> exit code: 0

Same words, same exit code, different repository. If the nightly schedule runs the cheap check and the pipeline records its success as the verification timestamp, then age two resets to zero every night for a repository that can no longer produce one of its files. The metric is fresh, the dashboard is green, and the number means nothing. An age is only as trustworthy as the command whose completion it records, which is why the emitter — not the alert rule — is where this design either works or quietly fails.

Severity comes from the consequence, not from the component

The reflex is to give every backup alert the same severity because they all come out of the same subsystem. That is severity by component, and it is why backup alerting has a reputation for being noise. Severity should describe what happens if the gap is not closed, and the three ages produce very different answers to that question even on the same host.

A stale recovery point on a service whose stated recovery point objective is one hour, in an architecture where snapshots are taken every fifteen minutes, is an active loss: every minute the alert goes unanswered is a minute of transactions that cannot be recovered. That pages someone. The same age on an archival share whose stated objective is one week, protected by a nightly job, is a ticket for the morning.

A stale verification age is almost never a page, because nothing is degrading faster because you are awake. It is a commitment that expires: the alert says the claim “our data is intact” is now unsupported, and the appropriate response is to schedule the expensive read rather than to run it at three in the morning. A stale proven-restore age is the same shape, with one exception — if a change landed in the recovery path since the last proven restore, then the last proof no longer describes the current system, and the severity should rise because the evidence has been invalidated rather than merely aged.

Written out, the rules stay short, and each one carries the action rather than leaving it to be reconstructed at three in the morning by someone reading a graph.

groups:
  - name: recovery-capability
    rules:
      - alert: RecoveryPointStale
        expr: time() - backup_snapshot_success_timestamp_seconds > 30 * 60
        for: 10m
        labels:
          severity: page
          owner_team: payments-platform
        annotations:
          summary: 'No new recovery point for the payments ledger in 30 minutes'
          action: 'Repair the capture path — the loss window is still growing'
      - alert: DataVerificationStale
        expr: time() - backup_verify_success_timestamp_seconds > 30 * 86400
        for: 6h
        labels:
          severity: ticket
          owner_team: payments-platform
        annotations:
          summary: 'No completed read-data verification in 30 days'
          action: 'Schedule a full verification — integrity is only known as of the last one'
      - alert: ProvenRestoreStale
        expr: time() - backup_restore_proof_timestamp_seconds > 90 * 86400
        for: 6h
        labels:
          severity: ticket
          owner_team: payments-platform
        annotations:
          summary: 'No verified restore in 90 days'
          action: 'Book a rehearsal — keys, target and procedure are unmeasured'

The thresholds above are not properties of restic. Each one is read off the stated architecture. Thirty minutes is two consecutive missed snapshots at a fifteen-minute cadence, and the ten-minute for puts the page at roughly forty minutes of staleness, still inside the one-hour recovery point objective the service has published. Thirty days is the monthly verification budget. Ninety days is the quarterly rehearsal slot. Change the cadence, the budget or the objective and every number moves with it; the same rules with different numbers would be correct for a different architecture and wrong for this one.

Routing: the page goes to whoever will be asked what happened

A recovery-capability alert routed to the backup team is routed to people who can see the gap and usually cannot close it. When the recovery point for a database has been stale for six hours, the capture path is broken by something in the database’s own world — a disk that filled, a credential rotated by its owning team, a schema migration that locked out the snapshot user. The backup team can escalate, and escalation at three in the morning costs an hour.

Route by the owner of the protected system. The label that makes this work is carried on the alert itself, so the routing tree never needs to know which service lives on which repository — it reads owner_team from the rule and delivers accordingly. That single decision has a second effect that matters more than the paging efficiency: the team that owns the service starts seeing the recovery consequences of its own changes, in its own channel, on the night the change lands rather than a quarter later in someone else’s report.

The backup team should still receive these alerts, as a copy rather than as the destination, because a pattern across many owners — five services whose verification age crossed thirty days in the same week — is a platform problem that no single owner can see.

The deadband problem, or the alert that has become a schedule

An alert that fires every night for the same known gap stops being an alert within about two weeks. The people receiving it learn — correctly, from evidence — that it requires no action tonight, and they generalise that lesson to the channel it arrives on, which is where the damage is done. The next genuinely novel recovery alert lands in a stream that has already been classified as ignorable.

The usual cause is a threshold set to the aspiration rather than to the architecture. If the rehearsal cadence funded by the business is quarterly and the proven-restore alert fires at thirty days, it will fire for sixty days out of every ninety, and it is reporting a budget decision rather than a fault. Either the threshold moves to match what the estate can actually sustain, or the cadence gets funded — but the alert is not the place to hold the argument.

For a gap that is genuinely accepted, suppress it explicitly rather than tolerating it: a suppression carrying a named owner and an expiry date, recorded where the next reviewer will find it. The difference between suppressing a gap and tolerating one is that a suppression comes back. An alert that is merely tolerated never returns to anyone’s attention, and the gap under it becomes permanent by default.

One more piece of hygiene keeps the deadband honest: hysteresis. An age that sits within minutes of its threshold will flap across it as jobs run slightly late, so give each rule a for duration long enough that a single late run does not page, and let the alert resolve only when a genuinely fresh timestamp arrives rather than when the age happens to dip back under the line.

Production discipline

1. Publish three ages per system, never one. Newest recovery point, newest completed data verification, newest proven restore. A fresh first number sitting beside a missing third is the normal state of an unrehearsed estate, and the report exists to make that visible rather than to hide it.
2. Record the age from the command that did the work. restic check and restic check --read-data both printed no errors were found with exit code 0 on repositories in different conditions; only the second read the 17374653-byte pack that the restore later failed on.
3. Set severity from what breaks, not from what reported. A stale recovery point on a service with a stated one-hour objective is an accumulating loss and pages; a stale verification age is an expired commitment and is a ticket, because nothing degrades faster while you sleep.
4. Route to the system’s owner and copy the backup team. The owner can fix the filled disk or the rotated credential tonight; the backup team can see the pattern across five owners that no single owner can.
5. Alert on the absence of a fresh result. An age expression over a missing series produces no alert at all, so every check needs a companion rule that fires when it stops reporting — the check that was silently disabled is the one that costs you the recovery.

Cross-course references

• Observability for Production Sysadmins — Part XX (Alert Quality) sets out why an alert that fires nightly without requiring action destroys the channel it arrives on, which is exactly the deadband failure described above: a proven-restore alert threshold set tighter than the funded rehearsal cadence trains a team to ignore the one signal that would have warned them the recovery path was broken.
• Observability for Production Sysadmins — Part XC (Meta-Monitoring) covers monitoring the monitoring path itself, and the meta-alert in this lesson is the recovery-assurance instance of that discipline: without a rule that fires on the absence of a fresh verification result, a disabled check and a passing check produce identical output, which is nothing.
• Kubernetes for Production Sysadmins — Part XCII (Alerting) works through routing alerts to workload owners rather than to the platform team, the same argument this lesson makes for recovery-capability alerts, and the reason a stale recovery point for a StatefulSet belongs in the owning team’s channel rather than in the backup team’s queue.

Quiz