Investigate a failed backup job

1 · Prerequisites

Confirm every item is in place before any state change.

• The identity of the job that failed: which client, which schedule, which repository or destination, and which data set it protects. A failure notification that names only a job id sends the responder looking in the wrong place for the first ten minutes.
• Read access to the client host where the backup process actually ran, and to its logs. The scheduler that reported the failure is usually not the machine that has the error message.
• Read access to the repository, bucket or destination the job writes to, plus whatever key or passphrase opens it, taken from escrow rather than from the client being investigated.
• The agreed recovery point objective for this data set, written down as a number of hours or minutes. Without it the age of the newest recovery point is a curiosity rather than a decision.
• The time of the last known-good run, from a source outside the failing job itself: the repository listing, a monitoring series, or a success stamp file. A job that fails while writing its own status file will lie about its own history.
• A list of every other client that writes to the same repository or destination, because a shared cause shows up as several jobs failing and the single-client explanation will fit none of them.
• The name of the service owner for the protected data, and the name of whoever can authorise a change to the backup schedule. Neither role is the responder holding the terminal.
• An incident note open before the first command, holding the time each check was run and what it returned. Half of this procedure is a timeline, and a timeline cannot be reconstructed afterwards.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish the age of the newest recovery point before you look at the failure at all. The failed job is the cause; the exposure is how long this system has been without a new recovery point. Those are different quantities, and only the second one tells the incident owner how much trouble the business is in.
• · Count how many consecutive runs have failed, not just the one that alerted. A single miss and a fortnight of silent misses produce the same notification. The repository listing and the schedule together give the answer; the notification alone never does.
• · Check whether other clients writing the same repository or destination also failed. One client failing is a client problem. Several failing at once is a repository, network, credential or capacity problem, and every minute spent on the client is wasted.
• · Confirm the job is genuinely failing rather than genuinely not running. A disabled timer, a decommissioned host and an expired schedule all present as "no recent backup" and none of them writes an error anywhere. Look at the schedule state before hunting for a message that does not exist.
• · Do not restart the job yet. A speculative rerun overwrites the client log you are about to read, may take a lock that then outlives it, and if the destination is short of space it can consume what remains. Read first.
• · Do not run any repair, prune or forget operation while the cause is unknown. These write. If the failure turns out to be repository damage, a write is the one action that can convert a recoverable situation into an unrecoverable one.
• · Confirm the tool version on the client and note it in the incident record. A version change on the client is a common and easily missed cause, and every message you are about to interpret belongs to a specific version.
• · Confirm that the repository or destination is reachable read-only from the client, using the same credential the job uses. This single check separates credential and destination failures from everything else in about thirty seconds.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Record the age of the newest recovery point, in hours, beside the agreed recovery point objective for this data set. This is the finding the incident is actually about. Everything below explains it; nothing below replaces it. Write both numbers down before the investigation moves on.
2. 2List the recovery points from the repository itself rather than from the job history. The measured listing showed each snapshot with its own timestamp, host, tags and paths, so the newest recovery point and its age are read from the data that would be restored, not from a status table that the failing job maintains.
3. 3Read the client log, not the scheduler result. A scheduler reports the exit status of the process it launched. When that process is a wrapper script, the exit status is the exit status of its last command, which is frequently a notification, a log rotation or a cleanup that succeeded after the backup failed. A green scheduler line above a failed backup is the commonest way this incident stays invisible for weeks.
4. 4Find the first error in the client log, not the last. The last line is usually a consequence: a summary, an abort, a retry giving up. The first error names the actual condition, and it is often several hundred lines above the one quoted in the alert.
5. 5Classify the failure by signature before touching anything: credential, capacity, lock, source unavailable, destination unreachable, or repository damage. Each has a distinct fingerprint and a different next action. Choosing an action before the classification is how a lock gets treated as damage, and how a full destination gets retried all night.
6. 6For a suspected lock, establish which process holds it and whether that process still exists. The measured tool prints create exclusive lock for repository when it takes one. A lock left behind by a killed run looks exactly like a lock held by a run still working, and the difference is a live process on a named host, not a guess.
7. 7For a suspected capacity failure, measure free space at the destination and at the client staging path, and state which one is short. Deleting recovery points to make room is an anti-pattern this course treats as an incident in its own right, and it must never be the first move of a capacity investigation.
8. 8For suspected repository damage, run the structural check first and then the check that reads and re-hashes the stored data, and record both exit statuses. The measured structural check reported no errors were found at exit 0 on a repository that had already been damaged. Only the data-reading pass found it. A single check that passes is not the answer to this question.
9. 9Treat the repository as under suspicion the moment a data-reading check reports damage, and stop writing to it. In the measured capture the tool named the damaged pack, said the damaged files must be removed to repair the repository, pointed at its troubleshooting guide first, and then printed two repair commands. Both are repair actions rather than read-only ones, and neither belongs in an investigation whose scope is not yet known.
10. 10Select the recovery point you would restore from if you had to restore right now, and name it. Never "the latest". Name it by repository, identifier and timestamp, and state in plain words what work it does not contain, measured from the incident window rather than from the job schedule.
11. 11Decision point — is the system inside its recovery point objective, outside it, or unknowable? Inside: fix the job, log the exposure, no escalation. Outside: escalate now, before the repair, because the exposure is already a business fact and somebody other than the responder has to own it. Unknowable, because the repository is under suspicion and no recovery point has been proven readable: escalate as if outside. Record which of the three, the numbers that decided it, and the time.
12. 12Decision point — repair in place, redirect the job to a second destination, or stop. Repair in place is right only when the classification is credential, lock, capacity or source, and nothing has cast doubt on the stored data. Redirect to a second destination when the repository is under suspicion and a second one exists, because a new recovery point is worth more than a diagnosis. Stop and escalate when neither is available.
13. 13Apply the fix that the classification indicates, one change at a time, and record each one as you make it. A credential renewal, a lock removal, a capacity addition and a source remount are four different changes, and applying them together makes the eventual write-up guesswork.
14. 14Run the job once, deliberately, and read its client log from the first line rather than reading only its exit status. One controlled run tests the classification. A retry loop tests nothing and consumes the destination.
15. 15Validate by performing a restore from a recovery point written after the fix, into an isolated target, and comparing it against a known digest. The measured successful restore reported Summary: Restored 7 files/dirs (60.000 MiB) at exit 0 and every file verified against digests taken before the backup. That pairing is the validation. A green job on its own is not evidence that the data can be recovered.
16. 16Have the service owner confirm, against a sample they choose, that the restored content is the content they expect. The responder can prove that bytes match a digest. Only the owner can say the data is the right data.
17. 17Publish the exposure window as a single number: from the last known-good recovery point to the first verified one after the fix, beside the agreed objective. That comparison is the output of this runbook. It is the only part of the investigation the organisation can act on, and it exists only if somebody writes down both ends.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The age of the newest recovery point at the start of the investigation is recorded in hours, beside the agreed recovery point objective for this data set, and the comparison between them is stated explicitly. It is read from the repository itself: for a restic repository, restic -r "$RBDR_REPO" snapshots exits 0 and prints one row per recovery point above a trailing count line. An empty listing is a finding, not a failed command.
• ✓The number of consecutive failed runs is recorded, established from the repository listing and the schedule rather than from the alert that opened the incident. systemctl show "$RBDR_UNIT" --property=Result,ExecMainStatus,ExecMainExitTimestamp exits 0 and its three lines are pasted into the record whole rather than summarised.
• ✓Whether other clients sharing the destination also failed is recorded as a yes or a no, with the evidence that settled it.
• ✓The first error in the client log is quoted verbatim in the incident record, with its timestamp, and it is identified as the first rather than the last. Note the exit status of the search that found it: grep exits 0 when it matched and 1 when it did not, and an exit of 1 is itself a finding — the failure left no message in the file you searched, so you are reading the wrong file or the run never started.
• ✓The failure is assigned to exactly one signature — credential, capacity, lock, source unavailable, destination unreachable, or repository damage — with the evidence that assigned it.
• ✓Where repository damage was suspected, both checks have been run and both exit statuses recorded: restic -r "$RBDR_REPO" check, then restic -r "$RBDR_REPO" check --read-data. In the measured capture on an already-damaged repository the first printed no errors were found and exited 0 while the second exited 1 on the same repository, so a single passing check does not satisfy this line.
• ✓The recovery point that would have been used for a restore during the exposure window is named by repository, identifier and timestamp, with a written statement of what it does not contain.
• ✓Each decision point has a recorded outcome, the numbers that decided it, the time, and the name of whoever decided.
• ✓A restore from a post-fix recovery point has completed into an isolated target and been verified file by file: restic -r "$RBDR_REPO" restore "$RBDR_SNAP" --target "$RBDR_TARGET" prints a Summary: Restored line and exits 0, and md5sum -c "$RBDR_BASELINE" prints OK against every file and exits 0. Both exit statuses are recorded; the restore's alone does not satisfy this line.
• ✓The service owner has confirmed, against a sample they chose, that the restored content is usable in business terms.
• ✓The total exposure window is published as one number and compared against the agreed objective, and the incident record says whether the objective was met.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Remove every monitoring silence, alert suppression and maintenance window created during the investigation, and record the time each was removed. A silence that outlives the incident is the reflex mutation this runbook most expects, and it hides the next occurrence of exactly this failure.
• ↶Re-enable any schedule, timer or job that was paused while the cause was being established, and confirm it is active rather than assuming it. A backup schedule left disabled after an investigation is a second, quieter incident with the same consequence as the first.
• ↶Undo any speculative restart of the backup service, agent or daemon that was made before the log was read, and note that it happened. A restart that is not recorded turns the client log into evidence nobody can date.
• ↶Remove any temporary credential, token or access grant issued to reach the destination during the investigation, and record when it was revoked.
• ↶Remove any lock, staging directory or partial archive created by a controlled test run, and record what was removed. Anything left behind under a plausible name will be mistaken for real state by the next responder.
• ↶Return any configuration changed while probing — a retry count raised to reproduce a timeout, a verbosity setting, a temporarily widened exclusion — to its previous value, and confirm from the file rather than from memory.

6 · Escalation

When the runbook isn't enough, contact:

• · The newest recovery point is older than the agreed recovery point objective: escalate to the incident owner immediately, before attempting the repair. The exposure already exists and the decision to accept it is not the responder's to make.
• · A data-reading check has reported damage in the repository: escalate before any repair, prune or forget runs. Every repair path writes, and the decision to write to a repository holding the only copies belongs to the data owner.
• · Several clients sharing one destination failed in the same window: escalate to whoever owns that destination. A shared cause is not a client incident, and treating it as one delays the fix for every client at once.
• · The failures have been running for longer than the retention period of any independent copy, so no unaffected recovery point remains inside the objective: escalate with the exact gap in hours rather than continuing to repair quietly.
• · The credential the job uses cannot be renewed by the responder — an expired key held elsewhere, a revoked token, a role nobody now owns: escalate to whoever owns that credential. This is an identity incident wearing a backup incident's clothes.
• · A restore from a post-fix recovery point fails or fails verification: stop treating this as a job failure, escalate, and move to the failed-restore procedure. The system is unrecoverable until a restore proves otherwise.

References

1. restic documentation, Backing up
2. restic documentation, Working with repositories
3. restic documentation, Troubleshooting
4. restic documentation, Scripting
5. BorgBackup documentation, borg check
6. systemd manual, journalctl
7. systemd manual, systemctl
8. NIST SP 800-184, Guide for Cybersecurity Event Recovery