medium riskservice affecting~40 min

Runbook: Take and verify a backup

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm what is being backed up and who owns it - name the data, not the host
· Confirm the destination has space: repository free space plus the expected delta
· Confirm the repository passphrase or key is escrowed somewhere you can reach without this host
· Confirm the retention policy so this run does not prune something still required
· Confirm whether the data needs quiescing (database, VM, anything with an open write path)
· Confirm the previous run succeeded; a chain of failures is its own incident

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the start time and the exact source paths
2Quiesce or snapshot the data so the capture is consistent
3Run the backup with an explicit exclude list and a named archive
4Release the quiesce or remove the snapshot as soon as the read is done
5Verify the archive: structural check plus data verification, not just exit status 0
6Confirm the archive appears in the repository listing with the expected size
7Apply retention/prune, and confirm what it removed before it removes it
8Record the archive name, size, duration and verification result in the backup log

4 · Verification

Confirm the procedure actually fixed the problem.

✓The backup tool exited 0 AND the archive is listed in the repository
✓A full verification pass (borg check --verify-data, restic check --read-data-subset) succeeded
✓A sample file has been restored to a scratch path and compared against the source
✓The archive size is within the expected range for the delta - a suspiciously small archive is a failed backup that reported success
✓The monitoring check for backup age has cleared

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A backup run itself changes no source data; there is nothing to roll back on the source
↶If a snapshot was taken, remove it - a forgotten LVM snapshot fills its pool and then invalidates itself
↶If a database was placed in backup mode, take it out of backup mode; a host left in backup mode accumulates WAL until the filesystem fills
↶If prune removed archives in error, stop immediately - prune is not reversible. Escalate.

6 · Escalation

When the runbook isn't enough, contact:

· Verification fails on an archive: escalate immediately and do not prune anything
· The repository is unreachable or its passphrase cannot be found: escalate to the backup owner, this is a business-continuity incident
· A database cannot be quiesced within the window: escalate to the data team rather than taking an inconsistent copy
· Repository free space insufficient for the run: escalate to capacity rather than shortening retention under pressure

A backup that has never been verified is a hypothesis. This runbook takes one and proves it, in the order that matters: quiesce, capture, verify, record. The verification step is not optional and is not the same as a zero exit status.

When to use this runbook

An on-demand backup before a risky change.
A scheduled backup that failed and must be re-run by hand.
The first backup of a new dataset, where nothing is proven yet.

Step 1: Know what you are protecting

Read-only / Safescope the source— Inspects the source and the destination. No writes.

# What is actually there, and how big
sudo du -sxh /srv/app /var/lib/postgresql 2>/dev/null

# Destination capacity
df -h /backup
borg info ssh://backup@vault/./repo 2>/dev/null | head -20

# When did the last successful run finish
borg list ssh://backup@vault/./repo | tail -5

Name the dataset and its owner before you start. “Back up the host” is how directories that matter get excluded and directories that do not get included.

Step 2: Quiesce, or accept an inconsistent copy

Copying a live database file while it is being written produces an archive that restores into a corrupt database. The archive will verify fine — it is a faithful copy of an inconsistent state.

Service impact possiblequiesce— Places the application in a consistent state for capture. May pause writes.

# PostgreSQL - use the tool, do not copy the data directory live
sudo -u postgres pg_dump -Fc app > /backup/staging/app-$(date -u +%F).dump
# or, for a physical base backup
sudo -u postgres pg_basebackup -D /backup/staging/base -Ft -z -Xs -P

# MySQL/MariaDB - consistent logical dump
sudo mysqldump --single-transaction --routines --triggers \
--all-databases > /backup/staging/mysql-$(date -u +%F).sql

# Filesystem-level: snapshot, then back up the snapshot
sudo lvcreate -L 10G -s -n app-snap /dev/vg0/app
sudo mkdir -p /mnt/app-snap
sudo mount -o ro /dev/vg0/app-snap /mnt/app-snap

Read-only / Safelvs— Lists logical volumes and snapshot fill percentage.

sudo lvs -o lv_name,origin,lv_size,data_percent,snap_percent

Step 3: Run the backup

Configuration changeborg create— Writes a new archive to the repository. Reads the source; does not modify it.

export BORG_REPO=ssh://backup@vault/./repo
export BORG_PASSCOMMAND='cat /etc/borg/passphrase'   # mode 0600, root only

sudo -E borg create \
--stats --progress \
--compression zstd,6 \
--exclude-caches \
--exclude '/srv/app/tmp/*' \
--exclude '/srv/app/cache/*' \
::'app-{now:%Y-%m-%dT%H:%M:%S}' \
/srv/app /etc /backup/staging

Configuration changerestic backup— Alternative tool. Writes a new snapshot to the repository.

export RESTIC_REPOSITORY=s3:s3.example.com/backups/app
export RESTIC_PASSWORD_FILE=/etc/restic/passphrase

sudo -E restic backup \
--exclude-caches \
--exclude /srv/app/tmp \
--tag app --tag scheduled \
/srv/app /etc /backup/staging

Release the quiesce as soon as the read completes — not at the end of the runbook:

Service impact possiblerelease quiesce— Removes the snapshot and returns the application to normal operation.

sudo umount /mnt/app-snap
sudo lvremove -y /dev/vg0/app-snap
sudo lvs -o lv_name,origin | grep -c snap   # expect 0 strays

Step 4: Verify - this is the step that makes it a backup

Read-only / Safeverify archive— Reads and validates the repository. Restores nothing.

# Structural check plus full data verification
sudo -E borg check --verify-data ::app-2026-08-11T02:00:00

# restic: check metadata, plus re-read a sample of the data
sudo -E restic check --read-data-subset=10%

# Is the archive there, and is it a plausible size
sudo -E borg list
sudo -E borg info ::app-2026-08-11T02:00:00

Step 5: Prove it restores

An archive that has never been restored is still a hypothesis. Restore one file, to a scratch path, and compare it.

Read-only / Safesample restore— Extracts a single file to a scratch directory. Does not touch the source.

sudo mkdir -p /var/tmp/restore-test && cd /var/tmp/restore-test

sudo -E borg extract --strip-components 0 \
::app-2026-08-11T02:00:00 srv/app/config/app.yaml

sudo diff -u /srv/app/config/app.yaml \
/var/tmp/restore-test/srv/app/config/app.yaml && echo 'MATCH'

A full restore rehearsal belongs in the restore runbook and should be scheduled, not improvised. This sample check is the minimum that belongs in every backup run.

Step 6: Apply retention

Data-loss riskprune— Permanently removes archives. Run the dry run first and read the list.

# ALWAYS dry-run and read the output before the real prune
sudo -E borg prune --dry-run --list \
--keep-daily=14 --keep-weekly=8 --keep-monthly=12

# Only after reading the list above
sudo -E borg prune --list \
--keep-daily=14 --keep-weekly=8 --keep-monthly=12
sudo -E borg compact

Step 7: Record the run

Read-only / Saferecord— Reads final state for the run log.

sudo -E borg info ::app-2026-08-11T02:00:00
sudo -E borg list | tail -3
df -h /backup

Write down, in the backup log or ticket:

Dataset and owner.
Archive name, start and end time, duration.
Original, compressed and deduplicated size.
Verification method used and its result.
What the sample restore compared, and whether it matched.
Anything excluded, and why.

Common patterns

Symptom	Likely cause	Resolution
Archive far smaller than the last one	An exclude matched too much, or a source path was not mounted	Compare `borg info` sizes; check the mount before the run
Backup succeeds, restore produces a corrupt database	Live data directory copied without quiescing	Use `pg_dump`/`pg_basebackup` or snapshot first
Repository fills up	Retention longer than capacity, or `compact` never run	Add capacity; never shorten retention under pressure
Snapshot-based backup fails part-way	LVM snapshot filled and was invalidated	Size the snapshot for the write rate, not the data size
WAL or binlog fills the filesystem after a backup	Database left in backup mode	Take it out of backup mode in the same procedure
Repository lock left behind	A previous run was killed	`borg break-lock` only after confirming no run is active
Nobody can find the passphrase	Key not escrowed	This is a business-continuity incident; escalate now