high riskdata loss risk~45 min

Runbook: Restore a named volume from backup

1 · Prerequisites

Confirm every item is in place before any state change.

The backup archive is on the host and you know its full path
You know every container and Compose service that mounts the target volume
The consuming service can be stopped, or you have an agreed maintenance window
Free space on the Docker data root exceeds the uncompressed size of the archive
You know the UID and GID the application runs as inside the container
You can reach a person who can confirm the application data is correct after the restore

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· VOL=app-data; docker volume inspect "$VOL" prints Name, Driver and Mountpoint
· docker ps -a --filter volume="$VOL" --format "{{.Names}} {{.State}}" lists every container holding the volume
· ARCHIVE=/var/backups/docker/app-data-2026-08-10.tar.gz; ls -lh "$ARCHIVE" shows a non-zero size
· gzip -t "$ARCHIVE" exits 0, proving the compression stream is intact
· tar -tzf "$ARCHIVE" | head -20 lists the paths you expect, at the depth you expect
· df -h /var/lib/docker shows free space greater than the uncompressed archive size
· docker system df -v shows the LINKS count for the volume, which must reach 0 before any restore

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Set your variables once: VOL=app-data, NEWVOL=app-data-restore, ARCHIVE=/var/backups/docker/app-data-2026-08-10.tar.gz, SERVICE=app
2Verify the archive before you trust it: gzip -t "$ARCHIVE" exits 0 and tar -tzf "$ARCHIVE" lists the expected top-level directory
3Create an empty target volume: docker volume create "$NEWVOL" prints the volume name
4Extract into the new volume as root so ownership is restored: docker run --rm --mount type=volume,src="$NEWVOL",dst=/restore --mount type=bind,src="$ARCHIVE",dst=/backup.tar.gz,ro alpine:3.20 tar -xzf /backup.tar.gz -C /restore --strip-components=1 --same-owner --numeric-owner
5Confirm ownership survived: docker run --rm --mount type=volume,src="$NEWVOL",dst=/restore alpine:3.20 ls -lan /restore prints the numeric UID and GID the application expects, not 0 0
6Stop the consumer before any switch-over: docker compose stop "$SERVICE" leaves the container in Exited state
7Confirm nothing still holds the old volume: docker ps --filter volume="$VOL" --format "{{.Names}}" prints nothing
8Point the service at the restored volume: edit the compose file so the service mounts "$NEWVOL", or rename by keeping both volumes and changing only the source name
9Start the service: docker compose up -d "$SERVICE" recreates the container with the new mount
10Confirm the mount actually changed: docker inspect -f "{{range .Mounts}}{{.Name}} {{end}}" the container prints "$NEWVOL"
11Prove the application reads the restored data, using the application own query path (see Verification), not a file listing
12Keep the original volume untouched for the agreed retention window; it is your rollback

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker inspect -f "{{range .Mounts}}{{.Name}}:{{.Destination}} {{end}}" "$SERVICE" prints the restored volume name at the expected destination
✓The application health endpoint returns 200: docker compose exec "$SERVICE" curl -fsS -o /dev/null -w "%{http_code}\n" http://localhost:8080/health prints 200
✓The application reads its own data: for PostgreSQL, docker compose exec db psql -U app -d app -c "select count(*) from orders;" returns a row count matching the pre-incident figure
✓A record known to exist before the incident is retrievable through the application API, not just present on disk
✓docker compose logs --since 5m "$SERVICE" contains no permission denied and no corrupt or checksum errors
✓File ownership inside the running container matches the application UID: docker compose exec "$SERVICE" stat -c "%u %g %n" /data prints the expected numeric IDs
✓The application writes as well as reads: create a throwaway record through the API and read it back

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Stop the service again: docker compose stop "$SERVICE"
↶Revert the compose file to mount the original volume "$VOL"
↶Start the service: docker compose up -d "$SERVICE"
↶Confirm the revert: docker inspect -f "{{range .Mounts}}{{.Name}} {{end}}" the container prints "$VOL" again
↶The restored volume "$NEWVOL" can be removed once the revert is confirmed: docker volume rm "$NEWVOL"
↶If you overwrote the original volume in place against this runbook advice, there is no rollback. The prior contents are gone. Restore from an older archive and open an incident.

6 · Escalation

When the runbook isn't enough, contact:

· The archive fails gzip -t or tar -tzf: escalate to the backup owner and try the previous archive before anything else
· The application starts but the data is stale by more than the agreed RPO: escalate to the service owner with the archive timestamp before deciding to keep it
· Ownership cannot be restored because the archive was written without --numeric-owner: escalate to the backup owner; the fix is in the backup job, not the restore
· The volume is backed by a non-local driver (NFS, cloud block storage): escalate to the storage team, since the driver has its own snapshot and restore path
· Data loss is already confirmed and customer-visible: raise an incident and notify the service owner before continuing

A volume restore is the one operation where the recovery attempt is more dangerous than the failure. The data you are trying to recover is usually still partly there. Extracting an archive over a live volume destroys whatever survived, and it does it while the application is still writing.

This runbook never overwrites in place. It restores into a new volume, switches the service over, and leaves the original alone until somebody has confirmed the restored data is correct.

Set your variables first

Every command below uses these. Set them once, in the shell you are going to work in.

Read-only / Safevariables— Defines shell variables. No side effects.

VOL=app-data
NEWVOL=app-data-restore
ARCHIVE=/var/backups/docker/app-data-2026-08-10.tar.gz
SERVICE=app

Step 1: Find out who is holding the volume

Read-only / Safeconsumers— Reads container and volume metadata. No side effects.

# Does the volume exist, and where does it live on disk?
docker volume inspect "$VOL"

# Every container that mounts it, running or not
docker ps -a --filter volume="$VOL" --format '{{.Names}}	{{.State}}	{{.Image}}'

# LINKS is the number of containers referencing each volume
docker system df -v

docker ps --filter volume= is the honest answer to “is anything using this?”. docker volume rm refuses a volume that is in use, which is a useful safety net, but it only protects the volume from deletion — it does nothing to stop you extracting a tar archive over the top of one that is mounted and live.

Step 2: Verify the archive before you trust it

The archive is the only copy of the data you are betting on. Check it before you make the live volume irrelevant.

Read-only / Safearchive integrity— Reads and tests the archive. Does not extract.

ls -lh "$ARCHIVE"

# The compression stream is intact end to end
gzip -t "$ARCHIVE" && echo 'gzip stream OK'

# The archive contains what you expect, at the depth you expect
tar -tzf "$ARCHIVE" | head -20
tar -tzf "$ARCHIVE" | wc -l

# Uncompressed size, so you can compare it with free space
gzip -l "$ARCHIVE"
df -h /var/lib/docker

Two things go wrong here more often than corruption does. The archive is a zero-byte file because the backup job failed and nobody watched the exit code. Or the archive was written from / and every path inside it is data/... rather than ./..., so an extract without --strip-components puts the data one directory too deep and the application starts with an empty volume.

tar -tzf ... | head -20 answers both in two seconds.

Step 3: Restore into a new volume

Configuration changecreate and extract— Creates a new volume and extracts into it. Does not touch the existing volume.

docker volume create "$NEWVOL"

docker run --rm \
--mount type=volume,src="$NEWVOL",dst=/restore \
--mount type=bind,src="$ARCHIVE",dst=/backup.tar.gz,ro \
alpine:3.20 \
tar -xzf /backup.tar.gz -C /restore \
   --strip-components=1 --same-owner --numeric-owner

Three details in that command earn their place:

--strip-components=1 removes the leading directory that the backup job added. Set it to match what tar -tzf actually showed you — if the archive paths already start at the data root, drop the flag entirely.
--same-owner --numeric-owner restores the recorded UID and GID rather than mapping through the helper image’s /etc/passwd. Without --numeric-owner, tar looks up the name in the helper image, and postgres in alpine is not the same UID as postgres in the Postgres image. The files land owned by the wrong user and the application fails on startup with permission denied.
The bind mount is ro. The helper container has no business writing to your archive.

--same-owner is the default when tar runs as root, which it does here, but stating it makes the intent legible to whoever reads this at 03:00.

Step 4: Check ownership before you switch anything

Read-only / Safeownership— Lists the restored files with numeric IDs. No side effects.

docker run --rm \
--mount type=volume,src="$NEWVOL",dst=/restore \
alpine:3.20 ls -lan /restore

docker run --rm \
--mount type=volume,src="$NEWVOL",dst=/restore \
alpine:3.20 du -sh /restore

ls -lan prints numeric IDs. If everything is 0 0 and the application runs as UID 999, you have found the problem now rather than after the switch-over. Re-extract with --numeric-owner, or fix ownership explicitly with a chown in a helper container using the numeric ID.

The du -sh figure should be within a few percent of what the old volume held. An order of magnitude difference means you restored the wrong archive or stripped the wrong number of components.

Step 5: Stop the consumer, then switch over

Service impact possiblestop and switch— Stops the service and recreates it against the restored volume. Service is down for the duration.

docker compose stop "$SERVICE"

# Nothing must still hold the old volume
docker ps --filter volume="$VOL" --format '{{.Names}}'

# Edit the compose file so the service mounts $NEWVOL instead of $VOL,
# then recreate:
docker compose up -d "$SERVICE"

docker inspect -f '{{range .Mounts}}{{.Name}}:{{.Destination}} {{end}}' \
"$(docker compose ps -q "$SERVICE")"

Switching the mount rather than overwriting the volume is what gives you a rollback. The original volume is still on disk, still complete, and still exactly as it was when the incident started. Reverting is a compose edit and a restart, not another restore.

Step 6: Prove the application reads the data

This is the step people skip, and it is the only one that matters. Files existing on disk is not evidence. The application successfully reading them is.

Read-only / Safeapplication-level verification— Queries the running application. Read paths only.

# Ownership as the application sees it
docker compose exec "$SERVICE" stat -c '%u %g %n' /data

# Health endpoint
docker compose exec "$SERVICE" \
curl -fsS -o /dev/null -w '%{http_code}\n' http://localhost:8080/health

# The application own read path - example for PostgreSQL
docker compose exec db psql -U app -d app -c 'select count(*) from orders;'

# Errors since the switch-over
docker compose logs --since 5m "$SERVICE" | grep -iE 'denied|corrupt|checksum|fatal'

Pick a query whose answer you knew before the incident. “The orders table has 41,802 rows” is a test. “The container is running” is not — a container with an empty volume runs perfectly.

Step 7: Retire the old volume, later

Data-loss riskdelete the original— Permanently deletes the pre-restore volume. Irreversible. Do not run on the day of the incident.

# Only after the service owner has confirmed the restored data is correct,
# and only after the agreed retention window has passed.
docker volume inspect "$VOL"
docker ps -a --filter volume="$VOL" --format '{{.Names}}'

docker volume rm "$VOL"

Common failure patterns

Symptom	Likely cause	Resolution
Application starts, volume looks empty	Archive paths were one level deeper than assumed	Re-extract with the correct `--strip-components`
Permission denied on every file	Extracted without `--numeric-owner`, names resolved in the helper image	Re-extract with `--numeric-owner`, or `chown` by numeric UID
`docker volume rm` refuses	A stopped container still references the volume	`docker ps -a --filter volume=` then remove the container
Restore succeeded, data is a day old	Backup job last succeeded a day ago	Escalate on RPO before accepting writes
Database starts then fails hours later	Archive was taken from a live volume with no quiesce	Restore from an archive taken with the database stopped or via its own dump tool
`gzip -t` fails	Truncated transfer or a failed backup job	Use the previous archive; fix the backup job