Volume backups — tar, restic, Borg

A named volume is a directory on the host, normally /var/lib/docker/volumes/<name>/_data. To back it up, copy the directory somewhere durable. That framing is correct for a config directory and quietly wrong for anything with a process writing into it, which is most of what you put on a volume.

The mechanics below are easy. The judgement — which of three consistency levels you are actually getting — is the lesson.

The three consistency levels

Get this straight before touching a command, because every tool on the page lands in one of these boxes and the boxes are not interchangeable.

The distinction that costs people money is the first two. They are routinely described with the same word, “crash-consistent”, and they are not the same thing at all.

Backing up a volume with tar

The canonical form. A helper container mounts the volume read-only and a destination directory read-write, and writes an archive.

VOLUME=app_uploads
DEST=/backup
STAMP=$(date +%F-%H%M)

docker run --rm -v "$VOLUME":/source:ro -v "$DEST":/backup alpine:3.20 tar czf "/backup/$VOLUME-$STAMP.tar.gz" -C /source .

Each part earns its place:

• --rm removes the helper container, so this does not accumulate one stopped container per night.
• :ro on the source is not decoration. It is the guarantee that a mistyped command cannot write into production data.
• -C /source . archives the contents of the directory rather than a source/ prefix, which is what makes the restore a plain extraction into the target volume.
• The timestamp in the filename is what turns one file into a series. A backup that overwrites itself has a retention of one.

The verification step, which is not optional

VOLUME=app_uploads
ARCHIVE=/backup/app_uploads-2026-08-12-0200.tar.gz

# 1. The archive is structurally readable end to end
gzip -t "$ARCHIVE" && echo 'gzip stream OK'

# 2. It contains a plausible number of entries, not zero
tar tzf "$ARCHIVE" | wc -l

# 3. The count matches the live volume, within reason
docker run --rm -v "$VOLUME":/source:ro alpine:3.20 sh -c 'find /source -type f | wc -l'

gzip -t decompresses the whole stream and discards the output, so a truncated archive — the disk filled at 02:41 and tar was killed — fails here rather than during a recovery. This is the check that fails, and it is why it belongs in the backup script rather than in a runbook.

$ gzip -t /backup/app_uploads-2026-08-12-0200.tar.gz

gzip: /backup/app_uploads-2026-08-12-0200.tar.gz: unexpected end of file
gzip: /backup/app_uploads-2026-08-12-0200.tar.gz: uncompressed data length is not correct

Restoring a volume

Restore is the operation nobody rehearses, and it has a sharp edge that the backup does not.

ARCHIVE=/backup/app_uploads-2026-08-12-0200.tar.gz
TARGET=app_uploads_restored

# 1. Stop anything writing to the data, if restoring in place
#    docker compose -f /srv/app/compose.yaml stop app

# 2. Restore into a fresh volume rather than over the live one
docker volume create "$TARGET"

docker run --rm -v "$TARGET":/target -v /backup:/backup:ro alpine:3.20 tar xzf "/backup/$(basename "$ARCHIVE")" -C /target

# 3. Prove the contents are there before pointing anything at it
docker run --rm -v "$TARGET":/target:ro alpine:3.20 sh -c 'find /target -type f | wc -l; du -sh /target'

Restoring into a new volume and re-pointing the service at it, rather than extracting over the live one, is the single habit that makes restores survivable. A tar extraction over an existing tree does not remove files that are not in the archive — so extracting an older backup over a newer volume leaves you with a hybrid of both, which is a state that has never existed and is far harder to reason about than either. It also means a botched restore has not destroyed the thing you were trying to recover, so you get a second attempt.

restic

tar gives you a full copy every night. For a 40 GB volume that is 40 GB of transfer and storage per run, which caps your retention and therefore your RPO. restic is the usual answer: content-defined chunking means only changed blocks are stored, and every snapshot is independently restorable.

# Credentials come from the environment, never from the command line,
# so they do not land in shell history or the process table.
export RESTIC_REPOSITORY='s3:s3.example.com/backups/app-01'
export RESTIC_PASSWORD_FILE=/etc/restic/password
export AWS_ACCESS_KEY_ID='REPLACE_ME'
export AWS_SECRET_ACCESS_KEY='REPLACE_ME'

restic init

VOLUME=app_uploads

docker run --rm -v "$VOLUME":/source:ro -v /etc/restic:/etc/restic:ro -v restic-cache:/root/.cache/restic -e RESTIC_REPOSITORY='s3:s3.example.com/backups/app-01' -e RESTIC_PASSWORD_FILE=/etc/restic/password -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY restic/restic:latest backup /source --host app-01 --tag "$VOLUME"

The details that are easy to get wrong:

• RESTIC_REPOSITORY must be set, by environment or by -r. A restic backup with neither fails; there is no implicit default repository.
• Mount the cache as a volume. restic keeps an index cache under /root/.cache/restic. In a --rm container that cache is discarded every run, so every backup re-downloads the index from the remote. On a large repository this turns a two-minute job into a twenty-minute one.
• --host and --tag are what make restic restore latest --host app-01 --path /source select the right snapshot later. Without them, “latest” means the latest of anything in the repository, including another machine’s.

Restoring, and the integrity check

TARGET=app_uploads_restored
docker volume create "$TARGET"

docker run --rm -v "$TARGET":/target -v /etc/restic:/etc/restic:ro -e RESTIC_REPOSITORY='s3:s3.example.com/backups/app-01' -e RESTIC_PASSWORD_FILE=/etc/restic/password restic/restic:latest restore latest --host app-01 --path /source --target /target

# Structural check of the repository - cheap, run it nightly
restic check

# Actually re-read and re-hash a sample of the pack files.
# Accepts n/t, a percentage, or a size such as 10G.
restic check --read-data-subset=5%

restic check on its own validates the repository’s structure: that every referenced blob exists and the trees are intact. It does not read the data. --read-data-subset downloads and re-hashes real pack files, which is what detects bit rot or a storage backend quietly returning corrupt objects. A nightly check plus a weekly --read-data-subset=5% covers the whole repository over a few weeks at a bandwidth cost you can budget for.

Note where restore lands: restore --target /target recreates the snapshot’s path structure beneath it, so files arrive at /target/source/.... Check the layout on your first restore rather than discovering it during one.

Borg

Borg predates restic and solves the same problem: deduplicated, compressed, encrypted, append-capable archives. The operational difference that usually decides it is the transport. Borg needs a borg serve process on the far end (over SSH) to get server-side deduplication, which makes it excellent against a machine you control and awkward against object storage. restic speaks S3, Azure Blob, Backblaze B2, REST and SFTP natively.

VOLUME=app_uploads

# One-time, on the backup host
borg init --encryption=repokey /srv/borg/app-01

docker run --rm -v "$VOLUME":/source:ro -v /srv/borg:/srv/borg -e BORG_PASSCOMMAND='cat /run/secrets/borg-passphrase' -v /run/secrets:/run/secrets:ro --entrypoint borg ghcr.io/borgbackup/borg:1.4 create --stats --compression zstd "/srv/borg/app-01::${VOLUME}-{now}" /source

Two Borg-specific traps. --encryption=repokey puts the key in the repository itself, so a repository copy plus the passphrase is everything an attacker needs; keyfile mode keeps the key on the client, which is stronger and means losing the client means losing the archives. And {now} is Borg’s own placeholder syntax, expanded by Borg, not the shell — it must survive quoting intact, which is why the archive name is in double quotes with the brace form left alone.

Database volumes: use the database

For anything with a transaction log, the volume is the wrong unit of backup.

STAMP=$(date +%F)
DEST=/backup

# Postgres: custom format is compressed and restorable selectively with pg_restore
docker exec -t app-db pg_dump -U app --format=custom --file=/tmp/app.dump app
docker cp app-db:/tmp/app.dump "$DEST/app-$STAMP.dump"
docker exec app-db rm -f /tmp/app.dump

# MySQL: --single-transaction is what makes it consistent for InnoDB
docker exec -t app-mysql mysqldump --single-transaction --routines --triggers --user=app --password="$MYSQL_PASSWORD" app > "$DEST/app-$STAMP.sql"

--single-transaction on mysqldump is not optional and is not a performance flag. Without it, mysqldump reads each table independently and you get a smeared dump — the exact failure this lesson opened with, produced by a tool people assume is safe because it is the database’s own. It works by taking a consistent read snapshot for the whole dump, and it applies to transactional engines only: a MyISAM table in the same database is still smeared.

Then treat the dump file as the thing you back up with restic. Dump for consistency, restic for durability and retention. The two tools are solving different problems and neither substitutes for the other.

Knowledge check