Back up and restore Docker volume data, and find what was missed

Objective

A process inside a container writes a file and gets 0 back. It cannot tell you whether those bytes landed in the container’s writable layer, which dies with the container, or in a mounted volume, which does not. Your backup procedure has to make that distinction for it.

By the end of this lab you will have taken two different backups of one running container, destroyed the container and its volume together, and restored one half byte-perfectly while watching the other half stay gone. You will have the exit codes for both.

Architecture

Two storage locations, two capture mechanisms, and each mechanism reaches exactly the one the other misses.

flowchart LR
    subgraph RUN["running container rbdr-app"]
        LAYER["writable layer<br/>/etc/app-marker"]
        VOL["volume rbdr-data<br/>mounted at /var/lib/app<br/>orders.csv"]
    end
    LAYER -->|docker commit| IMG["rbdr-committed:v1"]
    VOL -.->|never captured| IMG
    VOL -->|tar from a :ro helper| TGZ["rbdr-data.tgz<br/>160 bytes"]
    LAYER -.->|never captured| TGZ
    IMG --> P1["probe: marker prints, exit 0<br/>/var/lib/app empty, exit 0"]
    TGZ --> P2["restore: md5 matches, exit 0<br/>marker absent, exit 1"]

Requirements

• Docker Engine and the CLI. The capture quoted throughout was taken on Docker version 29.7.2, build a7dcaa6. That is the only engine this lab was run on; on another version, treat the quoted lines as the shape of the answer and your own output as the evidence.
• Permission to remove containers, volumes and images. Task 5 is destructive by design and there is no point running the lab without it.
• A host carrying no rbdr- objects. Everything created here uses that prefix so Cleanup can be scoped to it and asserted; Task 1 records the starting inventory so the assertion has something to compare against.

Scenario

A small order service runs in a container. Its data lives in a named volume. The team’s documented protection is a nightly docker commit pushed to a registry, and it has never once reported an error.

The host dies. You have every image. Somebody asks whether the orders are in them.

Tasks

Task 1 — Record pre-lab state

LAB="$HOME/rbdr-lab-18"
OUT=/tmp/rbdr-out
mkdir -p "$LAB" "$OUT"

{
  docker --version
  echo "--- containers ---"
  docker ps -a --filter 'name=rbdr-' --format '{{.Names}}' | sort
  echo "--- volumes ---"
  docker volume ls --filter 'name=rbdr-' --format '{{.Name}}' | sort
  echo "--- images ---"
  docker image ls --filter 'reference=rbdr-*' --format '{{.Repository}}:{{.Tag}}' | sort
} | tee "$LAB/pre-state.txt"

All three inventories must be empty. Cleanup diffs against this file at the end, so if something rbdr- already exists here, use another host rather than deleting an object that belongs to someone else.

Task 2 — Write to the volume and to the layer

docker volume create rbdr-data
docker run -d --name rbdr-app -v rbdr-data:/var/lib/app alpine:3 sleep 3600

docker exec rbdr-app sh -c 'printf "ORDER-1001,4500.00\nORDER-1002,1250.00\n" > /var/lib/app/orders.csv'
docker exec rbdr-app sh -c 'printf "written-into-the-container-layer\n" > /etc/app-marker'

docker exec rbdr-app cat /var/lib/app/orders.csv
docker exec rbdr-app md5sum /var/lib/app/orders.csv

Nothing is mounted over /etc, so the marker goes into the writable layer. The volume is mounted at /var/lib/app, so orders.csv goes outside the container entirely. Record the checksum now; it is the only thing that will prove the restore later.

$ the capture harness reporting where each write landed, then the volume contents and their md5

  data written to the VOLUME  : /var/lib/app/orders.csv
data written to the LAYER   : /etc/app-marker

--- volume contents ---
ORDER-1001,4500.00
ORDER-1002,1250.00
orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0

Task 3 — Commit the container and probe what you got

docker commit rbdr-app rbdr-committed:v1

{
  docker run --rm rbdr-committed:v1 cat /etc/app-marker
  echo ">>> exit code: $?"
  docker run --rm rbdr-committed:v1 ls -la /var/lib/app
  echo ">>> exit code: $?"
} | tee "$LAB/layer-vs-volume.txt"

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

Read the exit codes before the text. Both probes returned 0. The first printed the marker, so the commit captured the writable layer. The second listed /var/lib/app and found . and .. — the directory exists in the image because it is a mount point, and its contents were never part of the image at all. The commit captured the half nobody was worried about.

Task 4 — Archive the volume through a helper container

docker volume inspect rbdr-data --format 'Mountpoint: {{.Mountpoint}}'
docker volume inspect rbdr-data --format 'Driver: {{.Driver}}'

BACKUP_AT=$(date +%s)
docker run --rm -v rbdr-data:/src:ro -v "$OUT":/out alpine \
  tar czf /out/rbdr-data.tgz -C /src .
ls -l "$OUT/rbdr-data.tgz"

$ docker volume inspect rbdr-data, reporting the mount point and the driver

  Mountpoint: /var/lib/docker/volumes/rbdr-data/_data
Driver: local

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

The mount point is real, and reading it directly is still the wrong habit. The path the daemon reported sits under /var/lib/docker, which is the local driver’s own layout — a backup job written against it is written against an implementation detail rather than against the volume. The helper container asks the daemon for the volume by name instead, and mounts the source :ro, which keeps a mistake in the backup job from becoming a mistake in production data.

Task 5 — Destroy both, and confirm it by exit code

docker rm -f rbdr-app
docker volume rm rbdr-data

docker ps -a --filter 'name=rbdr-app' --format '{{.Names}}'
docker volume inspect rbdr-data
echo ">>> exit code: $?"

$ docker rm -f rbdr-app, docker volume rm rbdr-data, then docker volume inspect rbdr-data

  container removed, volume removed
[]
Error response from daemon: get rbdr-data: no such volume
>>> exit code: 1

This is the lab’s failing case, and it is worth more than the happy path that follows. docker volume inspect on a volume that does not exist prints [] on stdout, sends no such volume to stderr, and exits 1. A monitoring check that parses stdout and ignores the exit code sees an empty list and reports nothing wrong.

Task 6 — Restore into a volume that has never existed

RESTORE_START=$(date +%s)

docker volume create rbdr-data-restored
docker run --rm -v rbdr-data-restored:/dst -v "$OUT":/in alpine \
  tar xzf /in/rbdr-data.tgz -C /dst
docker run -d --name rbdr-app-restored -v rbdr-data-restored:/var/lib/app alpine:3 sleep 3600

RESTORED_MD5=$(docker exec rbdr-app-restored md5sum /var/lib/app/orders.csv | cut -d' ' -f1)

{
  docker exec rbdr-app-restored cat /var/lib/app/orders.csv
  echo "restored md5 : $RESTORED_MD5"
  echo "original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0"
  echo "restore seconds: $(( $(date +%s) - RESTORE_START ))"
  echo "unprotected window seconds: $(( RESTORE_START - BACKUP_AT ))"
} | tee "$LAB/restore-proof.txt"

Take the checksum into a variable before printing it, so the comparison below is between two labelled lines rather than between a labelled line and whatever md5sum chose to append.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

  ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

Two checksums taken on two different volumes, on either side of a deletion, over infrastructure that did not exist when the first one was computed. A row count or a file listing would have said the same reassuring thing about a truncated file.

Task 7 — Ask the restored container for the other file

docker exec rbdr-app-restored cat /etc/app-marker
echo ">>> exit code: $?"

{
  echo "/etc/app-marker | container writable layer | in rbdr-committed:v1 only | in no archive"
} | tee "$LAB/missed.txt"

$ docker exec rbdr-app-restored cat /etc/app-marker

  cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

Validation

Run each check and compare the printed string and the exit code against the table. Anything else is a finding, not a variation.

LAB="$HOME/rbdr-lab-18"

docker exec rbdr-app-restored md5sum /var/lib/app/orders.csv | cut -d' ' -f1
docker exec rbdr-app-restored cat /etc/app-marker
echo "marker exit: $?"
docker volume inspect rbdr-data
echo "old volume exit: $?"
docker volume inspect rbdr-data-restored --format '{{.Name}}'
echo "restored volume exit: $?"
grep -c 'rbdr-' "$LAB/pre-state.txt"
grep -c '>>> exit code: 0' "$LAB/layer-vs-volume.txt"

The checksum is the load-bearing line. 9eb4e2ad8e08e1dcaaf87ababab964b0 is the md5 of exactly those two order lines with a trailing newline, so any mismatch means the archive, the extraction or the original write differed.

Expected Outcome

The volume data survived deletion of both the container and the volume and returned byte-identical on infrastructure created after the loss. The container-layer file did not return, because the only artefact that ever held it was the committed image — and that image held no orders.

Record both measurements from your own run:

• Actual restore time: _______ seconds, taken from RESTORE_START in Task 6 to the checksum comparison. It counts the volume creation, the extraction and the container start. It does not count noticing the loss or finding the archive, which in an incident are usually larger.
• Actual RPO observed: _______ seconds, the unprotected window seconds printed in Task 6 — the gap between the tar czf in Task 4 and the destruction in Task 5. Every write inside that window is absent from the archive. This number is a property of when you ran the archive, never of tar or of Docker.

Both blanks are blanks on purpose. The captured transcript this lab quotes covers Tasks 2 through 7 — the writes, the commit probe, the archive, the destruction and the restore — and it recorded no timings at all. Nobody can fill those two lines in for you, and a lab that printed somebody else’s seconds there would be teaching the exact habit this course exists to break.

Troubleshooting

Error response from daemon: get rbdr-data: no such volume, exit 1, while still in Task 4. The volume was destroyed before it was archived. Task 5 deliberately runs after Task 4; restart from Task 2.

docker volume rm reports the volume is in use. A container still references it, and a stopped container counts. docker ps -a --filter 'volume=rbdr-data' names the holder; remove it with docker rm -f first.

The restored md5sum is not 9eb4e2ad8e08e1dcaaf87ababab964b0. The file written in Task 2 was not exactly those two lines with one trailing newline, usually because it was retyped by hand or an editor adjusted the final newline. Rewrite it with the printf from Task 2.

tar: /out/rbdr-data.tgz: Cannot open: Permission denied. The bind mount target is not writable by the helper container’s user, or a mandatory access control label is denying the write. On an SELinux host the bind mount usually needs a relabelling suffix; the Docker volumes documentation cited above describes the :z and :Z options and which of the two is safe on a shared directory.

tar: can't create /dst/...: Read-only file system, exit 1, in Task 6. The :ro from the backup mount in Task 4 was carried into the restore command. The source is read-only; the destination must not be.

The restored container starts and /var/lib/app is empty. The extraction wrote into a different volume from the one mounted, or ran before docker volume create. Confirm the volume name matches on both docker run lines.

cat /etc/app-marker succeeds in the restored container. You built the restored container from rbdr-committed:v1 instead of alpine:3. That proves the opposite point: the committed image carries the marker and no orders.

Cleanup

LAB="$HOME/rbdr-lab-18"
OUT=/tmp/rbdr-out

docker rm -f rbdr-app rbdr-app-restored 2>/dev/null
docker volume rm rbdr-data rbdr-data-restored 2>/dev/null
docker image rm rbdr-committed:v1 2>/dev/null
rm -f "$OUT/rbdr-data.tgz"

{
  echo "--- containers ---"
  docker ps -a --filter 'name=rbdr-' --format '{{.Names}}' | sort
  echo "--- volumes ---"
  docker volume ls --filter 'name=rbdr-' --format '{{.Name}}' | sort
  echo "--- images ---"
  docker image ls --filter 'reference=rbdr-*' --format '{{.Repository}}:{{.Tag}}' | sort
} > "$LAB/post-state.txt"

diff <(grep -v '^Docker version' "$LAB/pre-state.txt") "$LAB/post-state.txt" \
  && echo "CLEAN: post-state matches pre-state"

diff exiting 0 with CLEAN printed is the assertion that the host is back where Task 1 found it. Any other output names the objects still present.

Production notes

• Enumerate the mounts before you write a container backup job. docker inspect --format '{{json .Mounts}}' lists them; treat every path outside that list as unprotected until something proves otherwise.
• docker commit is not a backup of a service. It captures the writable layer, pauses the container while it works, and produces an image no Dockerfile explains.
• State written into a container layer is state nobody is protecting. Either move it into a volume or accept in writing that it is disposable — those are the only two honest options.
• Back volumes up through a helper container with the source mounted :ro. The host path under /var/lib/docker/volumes is an implementation detail of the local driver.
• Time the restore and write the number down beside the archive. An untimed restore is an estimate, and estimates in this area are reliably optimistic.

What You Learned

• Two writes that look identical from inside a container landed on two different storage systems, and only the exit codes of a later probe could tell them apart.
• docker commit captured the writable layer and left the volume mount point empty, with both probes exiting 0 — a capture that reports success and holds none of the service’s data.
• The volume archive restored byte-identical into a volume created after the original was destroyed, proven by md5 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides.
• docker volume inspect on a missing volume prints [] to stdout and exits 1, so a stdout-only check can be written that never notices the loss.
• The marker file was gone at exit 1, and the measurable window between archive and destruction is the RPO you actually achieved, not one any tool provided.