Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

intermediatebdr-capacity~40 min

Repository filled mid-backup and the next three nights reported success

Reported symptoms

  • ●At 09:20 on 28 August a developer asks for a file deleted from an application export directory on the 26th, and the newest recovery point that can be offered is dated the 23rd.
  • ●The backup dashboard shows nine green tiles for each of the last four nights, including the night the repository filled, and nobody was paged.
  • ●The repository filesystem on rbdr-backup-01 reports zero bytes available and 100 per cent used.
  • ●The nightly run on 24 August ran for forty-one minutes and stopped partway through, and each of the three runs after it finished in under a minute and reported success.
  • ●The repository has grown every month for fourteen months, while the estate believes retention keeps at most fourteen daily, eight weekly and twelve monthly recovery points per client.
  • ●Within ten minutes of the restore failing, somebody proposes removing the oldest recovery points so that tonight's run has somewhere to write.

Evidence

  • · df on /srv/rbdr-repo reports a 6.0T filesystem with 0 available and 100 per cent used, on a host whose only workload is the repository.
  • · restic snapshots lists 3412 snapshots with the oldest dated fourteen months ago, against a configured policy of --keep-daily 14 --keep-weekly 8 --keep-monthly 12, which would retain at most 34 per host and 306 across the nine clients.
  • · The newest snapshot for every one of the nine clients is dated 23 August, so the newest recovery point age jumped from roughly one day to roughly five while every dashboard tile stayed green.
  • · restic stats --mode raw-data reports 5.4 TiB of blobs held in the repository, while the nine source paths total 470 GiB and the estate's own filesystem-used series shows those sources grew four per cent across the same fourteen months.
  • · The repository data directory holds pack files stamped the night of 24 August that no snapshot references, which is what restic documents will remain when a backup runs out of space partway through.
  • · The Ansible role that installs the restic forget --prune timer is applied to the inventory group rbdr_prune_runner, whose only member is rbdr-app-01; that host was decommissioned fourteen months ago and the nightly play has reported no hosts matched and exit code 0 ever since.
  • · The client wrapper runs restic backup piped into tee with no set -o pipefail, so the status the wrapper records and publishes is tee's and not restic's.
  • · The monitoring configuration contains three backup checks — per-client job exit status, timer last-run age and repository reachability — and no check that reads free space on the repository filesystem or the age of the newest recovery point.
Diagnosis and resolutionclick to reveal

Root cause

Two independent defects, one of which filled the repository and one of which hid it for four nights. The immediate cause is that the retention policy had never been applied. In restic, `forget` removes snapshots and `prune` reclaims the storage those snapshots were holding; the documentation is explicit that after `forget` "the data that was referenced by files in this snapshot is still stored in the repository" and that `prune` must be run to clean up unreferenced data. The estate scheduled `restic forget --prune` from a client rather than from the repository host, and that client — rbdr-app-01 — was decommissioned fourteen months ago when its workload moved to rbdr-app-07. The configuration-management role survived; its target did not. A play with no matching hosts is not an error, so it exited 0 every night for fourteen months and nothing anywhere recorded that no retention had been applied. The plausible alternative is that the protected data grew. The evidence eliminates it twice over. The nine source paths total 470 GiB and grew four per cent over the period, while the repository grew from roughly 1.1 TiB to 5.4 TiB of raw blob data — growth in the destination that the source cannot account for. More decisively, the snapshot count is 3412 against a policy ceiling of 306. Genuine source growth enlarges each recovery point; it cannot raise the number of recovery points above what the policy retains. A count above the ceiling can only mean the policy has not executed. The second defect is the control. Three checks watched the backup estate and all three watched the job: exit status per client, timer last-run age, and whether the repository answered. Not one read a property of the repository itself. Free space was zero and the newest recovery point aged past five days without either fact appearing anywhere a human would see it. The mechanism that made the tiles green is worth naming, because it is ordinary. The wrapper runs `restic backup "$SRC" | tee -a "$LOG"` with no `set -o pipefail`, and the exit status of a pipeline is the status of its last command. restic exits 1 when it cannot write to the destination; `tee` exits 0 because writing the log succeeded. The dashboard faithfully published a number that had never described the backup. That a job reported success does not establish that anything was written, and no wrapper can be trusted to report on itself.

Remediation

Stop the nightly timers on all nine clients first. Until writes stop, the repository keeps accumulating unreferenced packs and no reclamation can make net progress. Add capacity before removing anything. `prune` needs scratch space to repack and to rewrite the index, and a repository at zero free bytes may not be able to run it at all. Extend the logical volume, or attach a second volume and extend the filesystem onto it. restic documents `prune --max-repack-size 0` for the case where very little space is available, which reclaims whole unused pack files without repacking, and an explicitly unsafe last resort, `--unsafe-recover-no-free-space`, which the documentation says should not be used before the first has been tried. Only then apply retention, and read it before you run it. `restic forget --dry-run` prints exactly which snapshots the policy would remove and removes nothing. Read that list against what the estate believes it keeps; fourteen months of unapplied policy means the first real run removes thousands of recovery points at once and it is the only chance to notice a wrong flag. Run `forget` with the agreed policy, then `prune`. Reclamation is the prune step, not the forget step. The unreferenced packs from the aborted run on the 24th are removed by the same prune. Move the schedule to rbdr-backup-01, which owns the repository and cannot be decommissioned without the repository going with it, and change the inventory group so the play targets a host that exists. Then unmask the client timers and run a backup for each client by hand rather than waiting for the schedule. Fix the wrapper in the same change: `set -o pipefail` at the top, and publish restic''s status rather than the pipeline''s.

Verification

Free space on /srv/rbdr-repo is a positive number, and the headroom is stated as days at the measured daily growth rate rather than as a percentage. `restic snapshots` returns a count at or below the policy ceiling of 306, and the newest snapshot for each of the nine clients is dated tonight. Both figures are read from the repository, not from the scheduler. The pack files stamped the night of 24 August are gone, and `restic check` completes without error. Note what that establishes: `restic check` verifies structural consistency, not the stored bytes. Run `restic check --read-data-subset` against a sample of packs to confirm the data itself survived the fill. A file deleted on the 26th is restored from a recovery point newer than the 23rd, onto a scratch path and compared against a known copy. Until a restore has been performed the repair is asserted rather than demonstrated. Break each new alert deliberately: fill a test filesystem to trip the free-space threshold, and suppress one client''s backup for longer than its interval to trip the recovery-point-age threshold. An alert that has never fired is a configuration, not a control.

Prevention

Alert on two properties of the repository rather than on the outcome of the job: free space on the repository filesystem, thresholded in days of headroom at the measured growth rate, and the age of the newest recovery point per client, thresholded slightly above that client''s backup interval. Both are read from outside the backup software. Both would have fired on the morning of the 25th. Schedule retention on the host that owns the repository. A client can be rebuilt, renamed or decommissioned; the repository host cannot disappear without the repository disappearing with it. Make an empty host pattern a failure. A configuration-management run that matches no hosts exits 0 by default and is indistinguishable from success; add an assertion that the prune group is non-empty so the silence becomes an error. Add a line to the decommissioning checklist requiring every timer, cron entry and scheduled play on the host to be enumerated and explicitly reassigned before the host is removed. This defect was created by a correctly executed decommission that nobody checked for orphaned schedules. Put `set -o pipefail` in every backup wrapper in the estate, and audit the others tonight rather than assuming this one was unique. Record the repository''s size, used and free figures monthly with the date taken, so growth is a series somebody can extrapolate rather than a surprise discovered at zero.

Reported symptoms

At 09:20 on 28 August a developer asks for a file they deleted from /srv/app/exports on the 26th. The newest recovery point anyone can offer is dated the 23rd.

The dashboard shows nine green tiles for each of the last four nights, the night of the 24th included. Nobody was paged, no ticket exists, and the estate’s written record of this incident begins with the failed restore, five days after the repository stopped accepting data.

rbdr-backup-01 carries one filesystem, /srv/rbdr-repo, and it reports zero bytes available. The run on the night of the 24th lasted forty-one minutes and stopped partway through. The three runs after it each finished in under a minute and reported success.

Within ten minutes somebody proposes removing the oldest recovery points so tonight’s run has somewhere to write. Nobody has yet established why a repository under a fourteen-day retention policy grew for fourteen months.

Evidence provided

Read-only / Safethe destination, not the job
$ df -h /srv/rbdr-repo

A 6.0T filesystem, zero available, 100 per cent used, holding nothing but the repository.

Read-only / Safehow many recovery points actually exist
$ restic -r /srv/rbdr-repo snapshots

3,412 snapshots, the oldest fourteen months old. The configured policy is --keep-daily 14 --keep-weekly 8 --keep-monthly 12: at most 34 per host, 306 across the nine clients.

Read-only / Safenewest recovery point, per client
$ restic -r /srv/rbdr-repo snapshots --host rbdr-db-01

23 August, for all nine clients. Nothing since.

Read-only / Safewhat the repository is holding
$ restic -r /srv/rbdr-repo stats --mode raw-data

5.4 TiB of blobs. The nine source paths total 470 GiB, and the estate’s own filesystem-used series shows those sources grew four per cent over the same fourteen months.

The data/ directory holds pack files stamped the night of the 24th that no snapshot references. The Ansible role installing the restic forget --prune timer targets the inventory group rbdr_prune_runner, whose only member is rbdr-app-01 — decommissioned fourteen months ago when its workload moved to rbdr-app-07. The nightly play has reported no hosts matched, exit code 0, ever since.

The client wrapper is one line: restic backup "$SRC" | tee -a "$LOG". There is no set -o pipefail. Monitoring holds three backup checks — per-client job exit status, timer last-run age, repository reachability — and none of them reads free space or recovery point age.

Work the evidence before reading on

  1. 3,412 snapshots against a policy ceiling of 306. Which hypothesis does that single number eliminate, and why can it not be explained by growth?
  2. The sources grew four per cent. The repository grew roughly fourfold. If the protected data did not put those bytes there, what did?
  3. Three nights reported success and created nothing. Where did the success come from?
  4. What are the pack files from the 24th, and what is supposed to remove them?
  5. Which two numbers, had either been on a dashboard, would have turned this into a capacity ticket on a Tuesday afternoon?

Root cause

The retention policy had never been applied

forget and prune are two operations. forget removes snapshots; restic’s documentation states that afterwards “the data that was referenced by files in this snapshot is still stored in the repository”, and that prune must be run to clean up unreferenced data. Neither had ever run here.

The estate scheduled restic forget --prune from a client rather than from the repository host. That client was decommissioned fourteen months ago. The configuration-management role survived; its target did not, and a play matching no hosts is not an error. It exited 0 nightly for fourteen months.

The aborted run left the packs from the 24th behind. restic documents this exactly: “Should you run out of space during the middle of a backup, there will be some additional data in the repository, but the snapshot will never be created as it would only be written at the very (successful) end of the backup operation. Previous snapshots will still be there and will still work.” The older recovery points were never at risk. The new one was never created.

The control watched the job, not the repository

Three checks existed and all three watched the job. None read a property of the repository. Free space sat at zero and the newest recovery point aged past five days without either fact reaching a person.

The mechanism that kept the tiles green is ordinary. The exit status of a pipeline is the status of its last command unless pipefail is set. restic exits 1 when it cannot write to the destination; tee exits 0 because writing the log succeeded. The wrapper published tee’s number.

Resolution

Stop writes before anything else, then add space before removing anything. prune needs scratch room to repack and rewrite the index.

REPO=/srv/rbdr-repo
for h in rbdr-app-02 rbdr-app-07 rbdr-db-01; do
  ssh "$h" systemctl stop rbdr-backup.timer
done
lvextend -L +1T -r /dev/rbdr/repo
df -h "$REPO"

Then read the policy before executing it. Fourteen months of unapplied retention means the first real run removes thousands of snapshots at once, and the dry run is the only chance to catch a wrong flag.

REPO=/srv/rbdr-repo
restic -r "$REPO" forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --dry-run
restic -r "$REPO" forget --keep-daily 14 --keep-weekly 8 --keep-monthly 12
restic -r "$REPO" prune

Reclamation happens at the prune step. If space is still too tight for it to start, restic documents prune --max-repack-size 0, which reclaims whole unused pack files without repacking, before the explicitly unsafe --unsafe-recover-no-free-space.

Move the schedule onto rbdr-backup-01, which owns the repository, and repoint the inventory group at a host that exists. Add set -o pipefail to the wrapper and publish restic’s status. Then restart the client timers and run each client by hand rather than waiting for tonight.

Verification

Free space on /srv/rbdr-repo is positive and stated as days of headroom at the measured growth rate, not as a percentage.

restic snapshots returns a count at or below 306, and the newest snapshot for each of the nine clients is dated tonight — read from the repository, not from the scheduler.

The packs from the 24th are gone. restic check completes without error, which verifies structural consistency and not the stored bytes; run restic check --read-data-subset 5% against a sample to confirm the data itself survived the fill.

Restore the developer’s file from a recovery point newer than the 23rd onto a scratch path and compare it against a known copy. Until a restore has been performed, the repair is asserted rather than demonstrated.

Trip both new alerts on purpose and record the dates. An alert that has never fired is a configuration, not a control.

Prevention

Alert on free space and on newest recovery point age, both read from outside the backup software. Either would have fired on the morning of the 25th.

Schedule retention on the host that owns the repository. A client can be renamed, rebuilt or decommissioned; the repository host cannot vanish without taking the repository with it.

Make an empty host pattern a failure rather than a silence, add orphaned schedules to the decommissioning checklist, put set -o pipefail in every backup wrapper in the estate, and record the repository’s size, used and free figures monthly so growth becomes a series somebody can extrapolate.