What is actually stateful — and why /var/lib/docker is not the answer · Docker & Containers

Disaster recovery starts with an inventory, not with a backup tool. The question is narrow and answerable:

If this host disappeared right now, what could not be recreated from a registry, a git repository and a configuration management run?

Whatever is on that list is your entire DR problem. Everything else is a rebuild.

Sorting the host

Containers are not on the list. A container is a process plus a disposable writable layer; you recreate it by running the image again. Images are not on the list either, provided they are in a registry you also protect.

Thing	Recreatable from	On the list?
Containers	`docker compose up -d`	No
Images	Registry, or a rebuild from the Dockerfile	No
Build cache	Nothing — and nothing needs it	No
Compose files, Dockerfiles	Git	No
`daemon.json`, systemd drop-ins	Configuration management	No
Named volume contents	Nothing	Yes
Anonymous volume contents	Nothing	Yes
Bind mount source directories	Nothing	Yes
Secrets and TLS private keys	A secret manager, if you have one	Yes
Self-hosted registry storage	Nothing	Yes
Anything written into a container and not into a volume	Nothing	Yes, and it should not exist

That last row is the one that turns a clean rebuild into a data loss event. A container’s writable layer is discarded when the container is removed — which docker compose up does routinely, on every image change. If an application writes state there, the state was already living on borrowed time before any disaster.

Anonymous volumes: state with no name

An image with a VOLUME instruction creates a volume whether or not you asked for one. If nothing is mounted at that path, Docker creates an anonymous volume with a 64-character hex name.

Read-only / Safea real host— 47 volumes. 37 have machine-generated names. 39 are attached to no container at all.

$ docker volume ls -q | wc -l
docker volume ls -q | grep -Ec '^[0-9a-f]{64}$'
docker volume ls -qf dangling=true | wc -l

47
37
39

Read those three numbers together. Most of the volumes on this host have names nobody chose, and most are referenced by nothing. Some of them are empty scratch space from an image’s VOLUME declaration. Some of them are the only surviving copy of something.

There is no command that tells you which is which. You have to look.

Read-only / Safetriage the orphans— For each unreferenced volume, print its mountpoint and size. Anything non-trivial deserves an explanation before it is pruned.

for v in $(docker volume ls -qf dangling=true); do
mp=$(docker volume inspect "$v" --format '{{.Mountpoint}}')
printf '%s  %s\n' "$(sudo du -sh "$mp" 2>/dev/null | cut -f1)" "$v"
done | sort -h | tail -20

docker volume prune and compose down -v delete these permanently

docker volume prune removes every volume not attached to a container. docker compose down -v removes the project’s volumes, including anonymous ones. Neither asks which of them held data, and there is no undo.

The routine sequence that destroys production data is not exotic: stop the stack for maintenance, run a prune to reclaim disk while the containers are down, bring the stack back up. The volumes were “dangling” only because the containers were stopped.

Two rules make this safe:

Never prune volumes while the workload is down. dangling means “no container is currently using it”, not “no container needs it”.
Give every volume that holds data an explicit name in the compose file. A named volume survives down, states its purpose, and can be found by a backup job. An anonymous volume can do none of those things.

Why a `/var/lib/docker` tar is not a backup

It is the obvious idea — everything Docker owns is under one directory, so copy the directory. It fails for four independent reasons, any one of which is sufficient.

1. It is a live database

/var/lib/docker holds the daemon’s metadata: the image store, containerd’s bolt database, network state, container configuration. The daemon holds these open and writes to them continuously. A tar of a running daemon’s directory is a crash-consistent copy of a database mid-write.

It might restore. Nobody can tell you in advance whether it will, and you find out during the outage.

2. It is mostly things you do not need

On the host above:

Read-only / Safewhat is in there— 28 GB total. The 2.9 GB of volumes is the only part that cannot be recreated.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          34        10        17.98GB   14.46GB (80%)
Containers      10        9         192.5kB   49.15kB (25%)
Local Volumes   47        8         2.901GB   1.684GB (58%)
Build Cache     87        0         7.446GB   5.124GB

You would be copying roughly 25 GB of image layers and build cache — all of it reconstructible from a registry or a rebuild — in order to protect 2.9 GB that matters. Every night, over your backup link, with the retention cost multiplied by every restore point.

3. It is not portable

The layout is an implementation detail of the storage driver, and the storage driver changes. This host reports:

Read-only / Safedriver check— Docker 29 defaults to the containerd snapshotter, reported as overlayfs. A 28.x host using the classic overlay2 graph driver has a different on-disk layout.

$ docker info --format 'driver={{.Driver}} version={{.ServerVersion}}'

driver=overlayfs version=29.7.1

A /var/lib/docker captured from a 28.x host with the overlay2 graph driver, restored onto a 29.x host using the containerd snapshotter, is not a supported operation. Neither is restoring across a filesystem change, or onto a host whose kernel does not support the same overlay features. Your DR host is very often not identical to the host you lost — that is usually why you are rebuilding.

4. It leaks secrets in a form your backup system does not expect

Container configuration under /var/lib/docker includes the environment variables each container was started with. If any service takes a password through the environment, that password is now in every restore point of a filesystem-level backup, in plain text, under a retention policy written for application data.

Back up the contents, not the directory

The supported approach is to back up volume contents through a helper container. It is portable across Docker versions, storage drivers and hosts, needs no access to the host’s internals, and copies only what cannot be recreated:

VOL=veilmere-postgres-data
docker run --rm \
  -v "$VOL":/source:ro \
  -v "$PWD":/backup \
  alpine:3.20 \
  tar czf "/backup/$VOL-$(date +%F).tar.gz" -C /source .

For databases, take the application-consistent dump instead of copying files under a running engine — docker-application-consistent-backups covers that. For bind mounts, back up the host path with the host’s normal backup tool; it is an ordinary directory.

The inventory as an artefact

The output of this lesson is a file in the repository, not a mental model. Generate it, commit it, and diff it on a schedule so a new unbacked-up volume shows up as a change:

Read-only / Safegenerate the state inventory— Every mount of every container, with its type and destination. Commit the output; a diff means the DR plan needs updating.

docker ps -a --format '{{.Names}}' | sort | while read -r c; do
docker inspect "$c" --format \
  '{{range .Mounts}}{{$.Name}} {{.Type}} {{.Name}}{{.Source}} -> {{.Destination}} rw={{.RW}}
{{end}}'
done

A row with Type=volume and a 64-hex name is an anonymous volume that should be given a name. A row with Type=bind names a host path that your host backup must cover. A service with no rows at all either genuinely holds no state, or is writing to its container layer.

What if a service has no mounts at all?

Two possibilities, and they need opposite responses.

It might be genuinely stateless — a reverse proxy, a worker that reads from a queue and writes to a database elsewhere. Nothing to do; that is the goal.

Or it is writing to its own writable layer, which means the state disappears on the next docker compose up that pulls a new image. Check before assuming:

docker diff web | head -40

docker diff lists every path added, changed or deleted in the container’s writable layer since it started. Log files and caches are expected. An application data directory, an SQLite file or a directory of user uploads in that output is a data-loss bug that has not fired yet.

Sanity check

Why this matters

Most Docker DR plans fail in the same place, and it is not the restore procedure. It is that the plan protects what was easy to identify rather than what was irreplaceable — a nightly tar of a directory that is 90% regenerable and 10% inconsistent, while the anonymous volume holding six months of user uploads is not mentioned anywhere.

The inventory is the deliverable. Once you can say, in one page, exactly which paths and which volumes cannot be recreated, the backup design becomes almost mechanical and the rebuild becomes a procedure rather than an improvisation.

Write the inventory down, commit it, and diff it on a schedule. A volume that appears without a corresponding backup job is the finding you want to make on a Tuesday afternoon rather than at 04:00 on the day the host is gone.