Monitoring the backup estate: the signals that matter

Part IV established that a backup job exiting 0 proves a process ran and nothing further, which leaves an operational question wide open: if job success is not the signal, what is? An estate of two hundred systems cannot be assessed by reading logs, and it cannot be assessed by the dashboard the backup product ships, because that dashboard was built around the unit the product understands — the job — rather than the unit the business asks about, which is the system. Getting this right is mostly a matter of deciding what to emit, long before deciding what to display.

Seven classes of signal, and the sentence each one supports

A backup estate should emit seven kinds of number. They are not interchangeable, none of them substitutes for another, and an estate missing one has a blind spot that no quantity of the other six repairs.

Recovery point age, per protected system. How long ago the most recent usable recovery point for that system was created. This supports the sentence “the worst-case loss for orders-db at this moment is four hours and twelve minutes”, which is the only form in which an RPO position can be honestly stated — as a measurement against a target that came from an agreed architecture and schedule, never as a property a tool supplies.

Verification age, per repository and per level. When the repository was last checked, recorded separately for a check that read only the bookkeeping and a check that read the stored bytes. These are different claims at different prices, and collapsing them into one field called “last verified” is how an estate ends up believing the cheaper claim.

Restore-proof age, per protected system. When data for that system was last restored somewhere and validated against something independent of the backup tool. This is the scarcest signal in most estates and the only one that speaks to the third claim from Part I.

Capacity and growth, per repository. Free space, consumed space, and the first derivative. The number worth alerting on is the projected exhaustion date, not the percentage full: a repository at 70% growing 3% per day is a ten-day emergency, and one flat at 85% for a year is not an emergency at all.

Retention compliance, per system and per policy. Whether the recovery points the policy says should exist actually do — the count of dailies, weeklies and monthlies present, compared against the count the policy specifies. A policy applied against a mistyped tag thins history quietly, and every individual job in that history exited 0.

Destructive operations, wherever they can occur. Snapshot removals, prunes, retention-policy edits, object-lifecycle rule changes, and credential or permission changes on the backup account. These are the events an adversary produces and also the events a tired operator produces at 23:40, and the estate should be able to answer “what was removed, by which identity, when” without reconstructing it from storage bills.

Liveness of every check above. Each of the six preceding signals is produced by something — a timer, a script, an exporter, a scheduled query. That something can stop. This class exists because the other six are all statements about the estate, and this one is the only statement about whether those statements are still being made.

Recovery point age is a clock, not an event

The distinction that makes the difference is between an event signal and a clock signal. An event signal — “job failed”, “check errored”, “snapshot missing” — has to be produced by something running in order to exist. A clock signal is a stored timestamp that an expression turns into an age, and it grows on its own whether or not anything runs. Only one of those two keeps working when the producer disappears, and it is not the one most estates emit.

The implementation is deliberately dull. Every job writes the time of its own last success into a .prom file in the directory node_exporter is pointed at with --collector.textfile.directory, and the exporter re-reads every matching file in that directory on each scrape. Writing a temporary file and renaming it is the pattern the exporter’s own documentation gives for exactly this case: a scrape that lands mid-write otherwise reads a truncated file and the series flickers.

SYSTEM=orders-db
TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
OUT="$TEXTFILE_DIR/backup_$SYSTEM.prom"

if restic backup /srv/orders; then
  RC=0
else
  RC=$?
fi
NOW=$(date +%s)

{
  printf 'backup_last_run_timestamp_seconds{system="%s"} %s\n' "$SYSTEM" "$NOW"
  printf 'backup_last_exit_code{system="%s"} %s\n' "$SYSTEM" "$RC"
  if [ "$RC" -eq 0 ]; then
    printf 'backup_last_success_timestamp_seconds{system="%s"} %s\n' "$SYSTEM" "$NOW"
  fi
} > "$OUT.tmp"
mv "$OUT.tmp" "$OUT"

Note that the success timestamp is written only on success, and is never cleared. That is the point: after a failure it stops advancing, so the age derived from it starts growing, and the growth is what the alert watches.

time() - max by (system) (backup_last_success_timestamp_seconds)

That expression answers the question an incident asks, and answers it as well for a system failing for eleven days as for one that failed last night. An alert built on backup_last_exit_code == 1 answers neither, because a job that no longer runs never sets it.

A check that fails and a check that stops running

These are two different events with two different remedies, and monitoring that does not separate them will report the second as health.

A check that fails is the case every design anticipates. Something ran, a value crossed a threshold, an alert fired, a human looked. The loop is complete and the signal did its job.

A check that stops running produces no value at all. Nobody is alerted, because the alerting system has nothing to evaluate. The dashboard tile does not turn red; it turns grey, or it keeps displaying the last value it saw, and grey tiles on a wall of green tiles are read by tired people as green. This failure mode is not exotic. It is a masked timer, a host rebuilt from an image without its cron entry, a credential that expired so the exporter can no longer read the repository, a container that was never rescheduled after a node drain.

The only defence is to treat the staleness of the result as a first-class alertable condition, separately from the content of the result.

One repository, two verification ages

“Last verified” is not a fact until the level of verification is named, and the measurement that settles this was taken on a repository holding a file it could not produce. Ten bytes were overwritten in the middle of its largest data pack, a 17374653-byte file that kept its name, its size and its place in the directory listing.

$ restic check

using temporary cache in /tmp/restic-check-cache-2185980449
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00%  2 / 2 snapshots
no errors were found

>>> exit code: 0

Any monitoring that recorded this run recorded a true statement — the structure, the indexes and the blob metadata really were consistent — and published a false impression, because the field it wrote into was called “verified”. The same repository, minutes later, under a check that reads and re-hashes every pack:

$ restic check --read-data

[0:00] 100.00%  2 / 2 snapshots
read all data
pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f contains 2 errors: [blob 9a6d59cfa25fce43e433aff4b16bb04d240c3730b027ae677bcb15719e44a436: decrypting blob <data/9a6d59cf> from pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f failed: ciphertext verification failed unexpected pack id 39e18fa9bf2f4acad3899cbe025e82aad203d9cfa1ee30017b1667a8d1e9bdc7]
[0:00] 100.00%  7 / 7 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

>>> exit code: 1

So the estate needs two gauges, not one, and they need different thresholds because the second reads every pack back, which is the same I/O a full restore performs, and is rarely affordable nightly on a large repository. Something like backup_repo_structure_check_timestamp_seconds alerting past 48 hours and backup_repo_data_check_timestamp_seconds alerting past 30 days is a defensible pair; a single field that either check can update is not, because whichever check ran most recently wins and the expensive one is the one that ran least recently.

The aggregate that hides the one system that matters

Take an estate of one hundred systems where ninety-nine are protected and one is not, and compute a single “backup health” percentage. The tempting reading is that the figure shows 99%, close enough to full health that no threshold triggers. The real behaviour is worse.

The percentage is almost always computed as successes divided by jobs. A system nobody configured has no job, so it contributes nothing to the numerator and nothing to the denominator. The figure reads 100%. The unprotected system is invisible not because the ratio is coarse but because the ratio is computed over the wrong population — over the things that exist rather than the things that should.

Even when the denominator is fixed to the inventory, a mean is the wrong summary. Averaging compresses a distribution into its centre, and every question worth asking about a backup estate is a question about the tail. The one system that is fourteen days stale and the ninety-nine that are four hours stale produce a mean of about seven and a third hours, a figure that describes no system in the estate — every system is either at four hours or at fourteen days — and conceals the only one that matters. Nor are the numbers comparable before normalisation: six hours of staleness on a tier-1 payments database and six hours on a tier-3 reporting mirror are different events, and any aggregation performed before dividing each age by its own tier’s target has already destroyed the meaning.

The top-line number should therefore be a count, not a ratio: systems whose recovery point age exceeds their tier’s target: 1. A count of one is visually distinguishable from a count of zero at a glance from across a room. Ninety-nine per cent is not distinguishable from one hundred, and a full-screen dashboard of ninety-nine per cent figures trains everyone who looks at it to stop looking.

Every signal names the system, not the job

The label set decides which questions a signal can answer, and backup tooling tends to default to the wrong one. Products commonly emit metrics keyed by job identifier, schedule identifier, policy name, backup-server hostname or repository path, because those are the objects the product manages. The question asked during an incident is never about a job. It is “when could orders-db last have been recovered from, and who owns it”.

Keying on the job breaks in three concrete ways. A job renamed during a migration ends one series and starts another, so the alert watching the old name goes quiet rather than firing, which is the failure mode of the previous section arriving through a change ticket. A system protected by two mechanisms — a filesystem snapshot and a repository backup — needs its recovery point age taken as max by (system) across both, which cannot be expressed if the labels describe mechanisms. And a job covering forty filesystems reports one status for forty different answers.

So every backup signal carries system, tier and owner, and those three come from the estate inventory rather than from the backup tool. Cardinality then stays bounded by a number you already know — how many systems you have — which is the whole cardinality argument in one sentence. What must never enter a label is anything generated per run: snapshot identifiers, timestamps, file paths, and repository object names like the pack 2c3be6d1c75844d268248b7a2a90e42f5d325095bd381b31c25bcc3d0179951f above. That string belongs in a log line, where it is searchable and costs nothing; as a label value it multiplies every series it touches by the number of packs in the repository.

Ownership is the second half of the same rule. A signal that cannot resolve to a person resolves to nobody, and an alert routed to a team distribution list at 03:00 is an alert routed to nobody.

Production discipline

1. Emit a timestamp, not a status. A stored Unix time turns into an age through time() - metric and keeps growing when the producer dies; a boolean ok/failed field is only correct for as long as something is still writing it, and is indistinguishable from a healthy estate once nothing is.
2. Pair every value rule with a staleness rule. For each of the seven signal classes, alert both on the value being wrong and on the result being older than its own production interval, and terminate the chain at an external dead-man’s-switch whose silence is the event.
3. Record verification age per level, and never merge the levels. Keep the structural check and the reading check in separate gauges with separate thresholds; on restic 0.19.1 the two disagreed about the same repository in the same minute, exit 0 against exit 1.
4. Publish counts of systems out of objective, never a percentage of jobs. Compute the population from the estate inventory so a system with no backup job appears in the denominator, and normalise every age against its own tier’s target before any aggregation.
5. Label by protected system, tier and owner. Bound cardinality by the inventory, keep per-run identifiers such as snapshot IDs and pack names in logs rather than labels, and reject any signal that cannot be resolved to a named owner.

Cross-course references

• Observability for Production Sysadmins — Part XX (Alert Quality) and Part XC (Meta-Monitoring) supply the two halves this lesson depends on: XX gives the criteria that decide whether a stale recovery point age is worth waking someone for, and XC is the general treatment of the monitoring-that- monitors-monitoring problem, of which the “liveness of every check” signal class above is one concrete instance.
• Observability for Production Sysadmins — Part IV (Cardinality) and Part XIV (Aggregation) formalise the two rules stated here without proof: IV explains the cost model behind bounding labels to system, tier and owner and keeping pack identifiers out of them, and XIV explains exactly which information an average discards, which is why the estate health percentage cannot represent the one unprotected system.
• Linux for Production Sysadmins — Part XXXVI (Scheduled Operations) covers the layer where the second failure mode of this lesson originates: a check that stops running is nearly always a masked timer, a unit that failed to start after a rebuild, or a schedule that left with the host it was defined on, and diagnosing that requires reading the scheduler rather than the dashboard.

Quiz