Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

intermediatebdr-credential~40 min

Backup credential expired and the estate stopped being protected in silence

Reported symptoms

  • ●A routine restore request for a file deleted on app-prod-11 turns up a newest recovery point 43 days old, while the person who asked says the file was open on their screen last Thursday.
  • ●The backup dashboard has been green for that host every night of those 43 days, including the run that completed six hours before the restore request arrived.
  • ●Widening the query to the whole estate shows twelve of forty-one Linux hosts with recovery points clustered between 41 and 43 days old, and the remaining twenty-nine all under 26 hours. Nothing sits in between.
  • ●The twelve are not a rack, not a subnet, not a hypervisor, not an operating-system release and not a backup schedule; the first four grouping attempts return no pattern at all.
  • ●The notification channel has received the line "nightly backup finished" from all twelve hosts every night for six weeks, and nobody reading it had any reason to look further.
  • ●On each of the twelve, `systemctl` reports the backup service as inactive (dead) with a successful last result and the timer as active and waiting for the next elapse.
  • ●A responder proposes closing the incident as "these were decommissioned during the platform migration", because a migration did complete in the same week the recovery points stop.

Evidence

  • · Recovery point age computed per host from the repository listing rather than from job history: twelve hosts between 41 and 43 days, twenty-nine hosts under 26 hours, and an empty gap between the two clusters. A staggered failure would fill that gap; a shared dependency will not.
  • · The twelve hosts are exactly the set whose restic environment file names access key ID `rbdr-key-app`. The twenty-nine healthy hosts name one of the other two keys the estate issues. The correlation is complete in both directions.
  • · The object store request log records authentication failures from all twelve source addresses, one burst per host per night on the schedule, continuing up to and including last night. Rejected requests are not the same observation as no requests.
  • · The last request from any of the twelve that the destination answered successfully is dated 42 days ago, and the credential register records the expiry date of `rbdr-key-app` as the following morning.
  • · The wrapper `/usr/local/sbin/rbdr-backup.sh` sets `-e` on line two, pipes restic into `tee` without `pipefail`, and ends with a `logger` call that is its final command.
  • · The systemd service records `ExecMainStatus=0` for every invocation across the window. That zero is the status `logger` returned; restic's own status was discarded two commands earlier.
  • · The load balancer shows all twelve hosts in the active pool with current request counts, and their monitoring agents have reported without a gap throughout the six weeks.
  • · Configuration management applied successfully on all twelve within the past 24 hours, and no decommissioning change request exists for any of them.
Diagnosis and resolutionclick to reveal

Root cause

Two independent defects, one of which caused the outage and one of which hid it for six weeks. The immediate cause is an expired credential. The object-storage access key `rbdr-key-app` reached its recorded expiry date 42 days ago. From the next scheduled run onward, restic could not authenticate to the destination, the repository was never opened, and no snapshot was written. The failure is total and immediate rather than partial: an unopened repository produces nothing at all, which is why the ages cluster so tightly instead of drifting. The tight cluster and the exact overlap with one key are what identify a shared credential rather than twelve coincidental host problems. The second defect is the control that should have surfaced the first one within a day. `/usr/local/sbin/rbdr-backup.sh` runs restic, applies retention, and then writes a fixed notification line. The exit status of a script is the exit status of its last command, so the status systemd recorded every night was `logger`'s, not restic's. The `set -e` on line two looks like protection and provides none here, because the restic invocation is piped into `tee` and the status of a pipeline without `pipefail` is the status of its final element. Both mistakes bias the report in the same direction: toward success. The consequence is not that twelve backups failed. It is that twelve production hosts had no recovery point newer than six weeks, and every report the estate produced said otherwise. The restore request is how it surfaced; nothing in the monitoring would ever have surfaced it.

Remediation

Quantify the exposure before changing anything. Compute the recovery point age per host from the repository, write down which twelve hosts are affected and how far back each one reaches, and tell the service owners the number. The window is retrospective as well as forward: anything deleted or corrupted on those hosts in the last six weeks is recoverable only to a point 42 days old, and somebody needs to establish whether that has already happened. Issue a replacement credential for the destination, scoped to the same paths the old key was scoped to, and record its expiry date in the register on the day it is issued rather than afterwards. Distribute it through the mechanism that already owns secrets on these hosts; hand-editing an environment file on twelve machines produces a thirteenth incident. Prove the credential before trusting the schedule. Run one backup in the foreground on one host and read the exit status directly, then repeat on a second host before touching the other ten. Stagger the catch-up. Each of the twelve will send six weeks of accumulated change on its first successful run, and twelve of those at 22:00 is a bandwidth and capacity event at the destination. Spread them across several nights and watch repository growth. Do not delete existing recovery points to make room for the catch-up; that trades a known gap for a new one. Repair the wrapper in the same change. Capture restic's status in a variable, set `-o pipefail`, exit with the captured status, and make the notification carry that status rather than a fixed string. A wrapper that cannot report failure is not a smaller problem than the credential; it is the reason the credential mattered.

Verification

Recovery point age, read from the repository listing, is under the threshold for all forty-one hosts, and the twelve that were stale are named individually in the record rather than covered by an estate-wide average. A file restored from a new snapshot on one of the twelve returns the expected content, which demonstrates that the destination accepts writes and returns bytes under the new credential — a claim the snapshot listing alone does not make. The repaired wrapper reports failure when the backup fails. Prove this rather than reading the script: run it on a test host with a deliberately wrong access key and confirm it exits non-zero, that systemd records the non-zero status, and that the notification line carries the failure instead of the word "finished". The alert built on recovery point age fires when a host is held back past its threshold, demonstrated by leaving one test host unbacked past the threshold rather than by inspecting the rule. The credential register shows an expiry date for every key the backup estate uses, and the expiry alert fires against a key deliberately set to expire shortly.

Prevention

Alert on recovery point age per system and per copy, derived from the repository listing. It is the only signal in this incident that would have moved: the ages began rising the morning the key expired and rose continuously for six weeks, while every job-derived signal stayed green. A missing run and a failing run both raise an age; neither necessarily produces a failure event. Inventory credential expiry as recovery material. Every key, token and certificate the backup path depends on belongs in a register with its expiry date, an owner, and an alert that fires weeks ahead. An expiry date is the one outage in this category that is known in advance, and it still took six weeks to notice. Give backup wrappers a contract and test it: `set -euo pipefail`, capture the status of the command that matters, exit with that status, and include it in every notification. Test the wrapper by making the backup fail on purpose, which is the only way to learn what it reports when something goes wrong. Make shared dependencies visible in the monitoring. Twelve hosts failing together is legible the moment the dashboard can group by credential; it is a mystery when the only groupings available are rack, subnet and schedule. Rehearse restores on a rota that covers every credential group, not merely every service tier. A single restore from one of these twelve hosts, at any point in six weeks, would have ended the incident on its first day.

Reported symptoms

Someone asks for a file deleted from app-prod-11. The newest recovery point for that host is 43 days old. They say the file was open on their screen last Thursday.

The dashboard for that host is green and has been every night since, including the run that finished six hours ago. The notification channel holds six weeks of nightly backup finished.

Widened to the estate, twelve of forty-one Linux hosts have recovery points between 41 and 43 days old. The other twenty-nine are under 26 hours. Nothing sits between the two clusters.

The twelve are not a rack, a subnet, a hypervisor, an OS release or a schedule. Four grouping attempts return nothing.

A migration completed the same week the recovery points stop, and somebody proposes closing this as twelve hosts decommissioned and never cleaned up.

Evidence provided

Read-only / Safeages from the repository, not from job history
$ restic -r "$REPO" snapshots --json | jq -r 'group_by(.hostname)[] | "\(.[0].hostname) \([.[].time] | max)"' | sort

Twelve hosts return a timestamp 41 to 43 days old. Twenty-nine return one from last night. The gap between the clusters is empty.

Those twelve are exactly the hosts whose restic environment file names access key ID rbdr-key-app. The other twenty-nine name one of the estate’s two other keys. The correlation holds in both directions.

Read-only / Safewhich credential each host is using
$ ansible linux_prod -m shell -a 'grep -h AWS_ACCESS_KEY_ID /etc/rbdr-backup.env'

The destination’s request log shows nightly authentication failures from all twelve source addresses, one burst per host per night, continuing up to and including last night. The last request it answered successfully from any of them is 42 days old. The credential register gives the expiry date of rbdr-key-app as the following morning.

The wrapper those hosts run, recovered unchanged from app-prod-11:

#!/bin/bash
set -e

REPO="s3:https://objects.example.net/rbdr-estate-backup"
LOG=/var/log/rbdr-backup.log

restic -r "$REPO" backup /etc /srv /var/lib 2>&1 | tee -a "$LOG"
restic -r "$REPO" forget --keep-daily 14 --keep-weekly 8 --prune 2>&1 | tee -a "$LOG"

logger -t rbdr-backup "nightly backup finished on $(hostname -s)"

systemctl show rbdr-backup.service --property=ExecMainStatus returns 0 for every invocation in the window.

Meanwhile the load balancer has all twelve in the active pool with current request counts, their monitoring agents have reported without a gap for six weeks, configuration management applied on all twelve inside the last day, and no change request proposes decommissioning any of them.

Work the evidence before reading on

  1. Twelve ages within two days of each other, and an empty gap to the healthy twenty-nine. What kind of cause produces a cluster that tight, and what kind produces a spread?
  2. The proposal is that these hosts were decommissioned. Which single piece of evidence kills that outright — and why is it stronger than the load balancer entry?
  3. The service exited 0 every night for six weeks. Which command’s status is that, and what happened to restic’s?
  4. Line two is set -e. Why did it not stop the script when the backup failed?

Root cause

Dispose of the decommissioning theory first, because it explains the ages and nothing else. The destination is logging authentication failures from all twelve addresses, on schedule, up to last night — and a decommissioned host is silent, not turned away at the door. The twelve are also in the active load-balancer pool serving production traffic, with agents that never stopped reporting. These machines are alive, busy, and trying to back up every night.

The credential expired, so the repository was never opened

rbdr-key-app reached its recorded expiry 42 days ago. From the next scheduled run onward restic could not authenticate, the repository was never opened, and no snapshot was written.

That is an all-or-nothing failure, which is why the ages cluster instead of drifting. A slow disk, a growing dataset or a flaky network staggers hosts across days; a shared credential stops every host holding it on the same night. The tight cluster is the fingerprint of a shared dependency, and the exact overlap with one key names which dependency it was.

The wrapper reported on its notification, not on its backup

Resolution

Quantify the exposure before changing anything. Name the twelve hosts, record how far back each one reaches, and give the service owners the number. Then ask the retrospective question: did anything on those hosts need recovering in the last six weeks and quietly get written off?

Issue a replacement key, scoped as the old one was, and record its expiry in the register the day it is issued. Distribute it through whatever already owns secrets on these hosts rather than editing twelve environment files by hand.

Prove it on one host, in the foreground, reading the status yourself:

Configuration changeone host, one run, status read directly
$ sudo systemctl start rbdr-backup.service && systemctl show rbdr-backup.service --property=ExecMainStatus --property=Result

Then stagger the rest. Each host sends six weeks of accumulated change on its first successful run, and twelve of those at once is a capacity event at the destination. Spread them over several nights and watch repository growth. Do not delete old recovery points to free space for the catch-up: that trades a gap you have measured for one you have not.

Repair the wrapper in the same change, because the credential is only half the fault:

#!/bin/bash
set -euo pipefail

REPO="s3:https://objects.example.net/rbdr-estate-backup"
LOG=/var/log/rbdr-backup.log
HOST=$(hostname -s)

status=0
restic -r "$REPO" backup /etc /srv /var/lib >>"$LOG" 2>&1 || status=$?

if [ "$status" -eq 0 ]; then
  restic -r "$REPO" forget --keep-daily 14 --keep-weekly 8 --prune >>"$LOG" 2>&1 || status=$?
fi

logger -t rbdr-backup "nightly backup on $HOST exited $status"
exit "$status"

Verification

Recovery point age, read from the repository, is under threshold for all forty-one hosts, with the twelve named individually rather than hidden inside an estate average.

A file restored from a new snapshot on one of the twelve comes back with the expected content. The listing shows the destination accepted a write; only the restore shows it returns bytes.

The wrapper reports failure when the backup fails — demonstrated, not read:

Service impact possiblemake it fail on purpose
$ sudo env AWS_ACCESS_KEY_ID=rbdr-not-a-key /usr/local/sbin/rbdr-backup.sh; echo "wrapper exited $?"

It exits non-zero, systemd records that status, and the notification line carries it instead of the word finished.

The age alert fires against a host deliberately held past its threshold, and the expiry alert fires against a key deliberately set to expire shortly. Both are tested by tripping them, not by reading the rule.

Prevention

Alert on recovery point age, per system and per copy, derived from the repository listing. It is the only signal here that moved: the ages began rising the morning the key expired and never stopped. A failing run and a run that never happened both raise an age; neither reliably produces a failure event.

Put credential expiry in the recovery inventory. Every key, token and certificate the backup path depends on needs an owner, a recorded expiry and an alert weeks ahead. This is the one outage in the category that is knowable in advance, and it still ran unnoticed for six weeks.

Make shared dependencies visible. Twelve hosts failing together is obvious the moment a dashboard can group by credential, and unreadable when the only groupings are rack, subnet and schedule.

Rotate restore rehearsals across credential groups, not just service tiers. One restore from any of these twelve, on any day in six weeks, would have closed this on day one.