Restore a complete service onto clean infrastructure

Objective

Containers make hidden dependencies visible faster than anything else, because destroying one is cheap. That is why this lab uses them as the clean infrastructure: the rebuild is genuinely clean, with no leftover files and no configuration somebody applied by hand in 2023.

You will run a small service that keeps state in two places, take the backup most teams take, destroy both the container and the volume, and rebuild. The volume data will come back byte-identical. Everything else will not, and the deliverable of this lab is the written list of what “everything else” turned out to be.

Architecture

Three places hold state; one backup mechanism reaches each of them, and one place is reached by neither.

flowchart TB
    subgraph SVC["running service rbdr-app"]
        LAYER["container layer<br/>/etc/app-marker"]
        VOL["named volume rbdr-data<br/>mounted at /var/lib/app<br/>orders.csv"]
        CFG["name, image tag, mount spec,<br/>network, restart policy"]
    end
    LAYER -->|docker commit| IMG["rbdr-committed:v1<br/>layer captured<br/>/var/lib/app is empty"]
    VOL -->|tar via helper container| TGZ["rbdr-data.tgz, 160 bytes"]
    CFG -->|captured by nothing| GAP["lives only in operator memory"]
    TGZ --> NEW["rbdr-data-restored<br/>md5 identical to the original"]
    IMG -.->|not used in the rebuild| NEW
    GAP -.->|retyped by hand| NEW

Requirements

• Docker Engine and the CLI, with permission to create and destroy containers, volumes and images. The capture this lab quotes was taken on Docker 29.7.2, build a7dcaa6.
• Roughly 500 MB of free disk for the alpine image and the archives.
• A host with no rbdr- objects on it. Every object here carries that prefix so cleanup can be scoped and asserted; Task 1 records what was there first.

Scenario

An application team runs a small order service in a container. Their documented backup is a nightly docker commit of the running container, pushed to a registry. It has run without error for eleven months.

The host is lost. You have the images. You are asked how long the restore will take.

Tasks

Task 1 — Record pre-lab state

LAB="$HOME/rbdr-lab-10"
OUT=/tmp/rbdr-out
mkdir -p "$LAB" "$OUT"

{
  docker --version
  echo "--- containers ---"
  docker ps -a --filter 'name=rbdr-' --format '{{.Names}}' | sort
  echo "--- volumes ---"
  docker volume ls --filter 'name=rbdr-' --format '{{.Name}}' | sort
  echo "--- images ---"
  docker image ls --filter 'reference=rbdr-*' --format '{{.Repository}}:{{.Tag}}' | sort
} | tee "$LAB/pre-state.txt"

Cleanup at the end of this lab compares against pre-state.txt, so the three inventories must be empty now. If they are not, rename the objects this lab creates or use a different host; do not delete somebody else’s rbdr- objects on the strength of a prefix collision.

Task 2 — Run the service and write state in two places

docker volume create rbdr-data
docker run -d --name rbdr-app -v rbdr-data:/var/lib/app alpine:3 sleep 3600

docker exec rbdr-app sh -c 'printf "ORDER-1001,4500.00\nORDER-1002,1250.00\n" > /var/lib/app/orders.csv'
docker exec rbdr-app sh -c 'printf "written-into-the-container-layer\n" > /etc/app-marker'

docker exec rbdr-app cat /var/lib/app/orders.csv
docker exec rbdr-app md5sum /var/lib/app/orders.csv

/var/lib/app/orders.csv lands in the volume. /etc/app-marker lands in the container’s writable layer, because nothing is mounted over /etc. Both look identical from inside the container, and that is the whole problem: a process writing a file cannot tell you which storage it landed on.

$ the harness reporting where each write landed, then the volume contents and their checksum

  data written to the VOLUME  : /var/lib/app/orders.csv
data written to the LAYER   : /etc/app-marker

--- volume contents ---
ORDER-1001,4500.00
ORDER-1002,1250.00
orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0

Task 3 — Take the backup the team believes in

docker commit rbdr-app rbdr-committed:v1

{
  docker run --rm rbdr-committed:v1 cat /etc/app-marker
  echo ">>> exit code: $?"
  docker run --rm rbdr-committed:v1 ls -la /var/lib/app
  echo ">>> exit code: $?"
} | tee "$LAB/commit-evidence.txt"

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

Both commands exited 0. The marker file printed. /var/lib/app exists and holds nothing but . and ... The commit captured the container’s writable layer; the volume was never part of the image, because /var/lib/app is a mount point and its contents live outside the image entirely.

Task 4 — Find the volume and archive it

docker volume inspect rbdr-data --format 'Mountpoint: {{.Mountpoint}}'
docker volume inspect rbdr-data --format 'Driver: {{.Driver}}'

docker run --rm -v rbdr-data:/src:ro -v "$OUT":/out alpine \
  tar czf /out/rbdr-data.tgz -C /src .
ls -l "$OUT/rbdr-data.tgz"

$ docker volume inspect rbdr-data, reporting the mount point and the driver

  Mountpoint: /var/lib/docker/volumes/rbdr-data/_data
Driver: local

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

The helper-container route is the portable one. Reading the host path directly works today, requires root, and breaks the moment the volume moves to a driver that has no host path at all.

Task 5 — Destroy the container and the volume

docker rm -f rbdr-app
docker volume rm rbdr-data

docker ps -a --filter 'name=rbdr-app' --format '{{.Names}}'
docker volume inspect rbdr-data
echo ">>> exit code: $?"

$ docker rm -f rbdr-app, docker volume rm rbdr-data, then docker volume inspect rbdr-data

  container removed, volume removed
[]
Error response from daemon: get rbdr-data: no such volume
>>> exit code: 1

This is the failing case the lab needs. docker volume inspect prints an empty JSON array on stdout, the error on stderr, and exits 1. A check that reads only stdout sees [] and can be written to conclude that nothing is wrong.

Task 6 — Rebuild on clean infrastructure and prove the bytes

RESTORE_START=$(date +%s)

docker volume create rbdr-data-restored
docker run --rm -v rbdr-data-restored:/dst -v "$OUT":/in alpine \
  tar xzf /in/rbdr-data.tgz -C /dst
docker run -d --name rbdr-app-restored -v rbdr-data-restored:/var/lib/app alpine:3 sleep 3600

{
  docker exec rbdr-app-restored cat /var/lib/app/orders.csv
  docker exec rbdr-app-restored md5sum /var/lib/app/orders.csv
  echo "original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0"
  echo "restore seconds: $(( $(date +%s) - RESTORE_START ))"
} | tee "$LAB/restore-proof.txt"

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

  ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

Two checksums, taken on different volumes on either side of a destruction. That is a restore proof. A file count would have said the same thing about a truncated file.

Task 7 — Ask the restored service for the other file

docker exec rbdr-app-restored cat /etc/app-marker
echo ">>> exit code: $?"

$ docker exec rbdr-app-restored cat /etc/app-marker

  cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

It was never in the volume, and the volume archive is the only thing that was taken. State written into a container layer is state nobody is protecting.

Task 8 — Write the list

The deliverable of this lab is not the archive. It is gaps.txt.

cat > "$LAB/gaps.txt" <<'EOF'
# Everything the rebuild needed that did NOT come out of rbdr-data.tgz.
# Columns: WHAT | WHERE IT LIVED | HOW IT WAS RECREATED | WHO KNEW
/etc/app-marker | container writable layer | retyped from memory | one engineer
image reference alpine:3 | operator memory | retyped | one engineer
container name rbdr-app | operator memory | retyped | one engineer
mount spec rbdr-data:/var/lib/app | operator memory | retyped | one engineer
process command sleep 3600 | operator memory | retyped | one engineer
EOF

wc -l < "$LAB/gaps.txt"

Extend it from your own run. Every line you add is a dependency your production procedure also has and does not mention. Ask of each one: if the person who knew it were unavailable, what would the rebuild do?

Validation

Run each check and compare against the stated string and exit code.

grep -c 'rbdr-' "$LAB/pre-state.txt"
docker exec rbdr-app-restored md5sum /var/lib/app/orders.csv | cut -d' ' -f1
docker exec rbdr-app-restored cat /etc/app-marker
echo "marker exit: $?"
docker volume inspect rbdr-data-restored --format '{{.Name}}'
test -s "$LAB/gaps.txt" && echo "OK gaps.txt"
grep -c '|' "$LAB/gaps.txt"

The checksum is the load-bearing check. 9eb4e2ad8e08e1dcaaf87ababab964b0 is the md5 of exactly the two order lines with a trailing newline, so a mismatch means the archive, the extraction or the write differed.

Expected Outcome

The volume data survived a full destruction of both the container and the volume, and returned byte-identical. The container-layer file did not survive, and the image that a nightly docker commit produced was the only artefact that ever held it — an artefact the restore did not use, because it carried no orders.

Record both numbers from your own run:

• Actual restore time: _______ seconds, measured in Task 6 from docker volume create to the checksum comparison. It excludes the time to notice the loss, to find the archive, and to remember the five lines now sitting in gaps.txt.
• Actual RPO observed: _______ seconds, the interval between the tar czf in Task 4 and the docker volume rm in Task 5. Any write inside that window is not in the archive. This is a property of when the archive was taken, never of the tool that took it.

Troubleshooting

Error response from daemon: get rbdr-data: no such volume, exit 1, during Task 4. The volume was removed before it was archived. Task 5 runs after Task 4 for that reason; restart from Task 2.

docker volume rm reports volume is in use. A container still references it, including a stopped one. docker ps -a --filter 'volume=rbdr-data' names the holder; docker rm -f it first.

The restored md5sum differs from 9eb4e2ad8e08e1dcaaf87ababab964b0. The file written in Task 2 was not the exact two lines with a trailing newline — most often because a shell history expansion or an editor stripped or added one. Rewrite it with the printf from Task 2 rather than by hand.

tar: /out/rbdr-data.tgz: Cannot open: Permission denied. The bind mount target directory is not writable by the helper container’s user, or SELinux is denying the mount. Add :z to the -v "$OUT":/out mount on an SELinux host.

The restored container starts and /var/lib/app is empty. The extraction ran into a different volume from the one mounted, or ran before the volume existed. docker volume inspect rbdr-data-restored and confirm the name matches on both docker run lines.

cat /etc/app-marker succeeds in the restored container. You rebuilt from rbdr-committed:v1 rather than from alpine:3. That is worth seeing once, but it proves the opposite point: the committed image carries the marker and no orders.

Cleanup

docker rm -f rbdr-app rbdr-app-restored 2>/dev/null
docker volume rm rbdr-data rbdr-data-restored 2>/dev/null
docker image rm rbdr-committed:v1 2>/dev/null
rm -f "$OUT/rbdr-data.tgz"

{
  echo "--- containers ---"
  docker ps -a --filter 'name=rbdr-' --format '{{.Names}}' | sort
  echo "--- volumes ---"
  docker volume ls --filter 'name=rbdr-' --format '{{.Name}}' | sort
  echo "--- images ---"
  docker image ls --filter 'reference=rbdr-*' --format '{{.Repository}}:{{.Tag}}' | sort
} > "$LAB/post-state.txt"

diff <(grep -v '^Docker version' "$LAB/pre-state.txt") "$LAB/post-state.txt" \
  && echo "CLEAN: post-state matches pre-state"

diff exiting 0 with CLEAN printed is the assertion. Anything else lists the objects still present, by name.

Production notes

• A containerised service keeps state in the image, the writable layer and every mount. Enumerate the mounts with docker inspect --format '{{json .Mounts}}' and treat anything outside them as unprotected.
• docker commit is not a backup of a service. It captures the layer, pauses the container while it works, and produces an image whose provenance no Dockerfile explains.
• Back volumes up through a helper container, not from the host path. The host path is a local driver implementation detail.
• The rebuild instructions are part of the backup. An archive with no record of the image tag, the mount specification and the command line is bytes waiting for somebody to remember what to do with them.
• Time the restore and write the number down. An untimed restore is an assumption, and the assumption is always optimistic.

What You Learned

• A docker commit captured the container layer and left the volume mount point empty, with both probe commands exiting 0 — a backup that reports success and holds none of the service data.
• A volume-only archive restored byte-identical across a full destruction, proven by two checksums rather than by a file count.
• The layer file was gone entirely, exit 1, because nothing had ever been protecting it.
• docker volume inspect on a missing volume prints [] to stdout and fails on stderr, so a stdout-only health check can be written to miss the loss completely.
• The gap list is the finding. Five dependencies came back only because a human retyped them, and none of them appeared in any archive.