Volumes, bind mounts, and where the state actually is · Backup & DR

The commit in the previous lesson captured a writable layer and nothing else. The marker file written into the layer travelled with the image; the orders file written into the volume did not, and a new container started from that image listed its /var/lib/app as total 8 with two directory entries and no data. That settles what an image does not contain. It does not tell you what to protect instead, and on a host running a dozen containers the answer is never a single tidy list. Before an estate can be backed up it has to be enumerated, and the enumeration is not visible in the compose file.

Four places container state hides

The first place is a named volume: storage the engine created, tracked by name, and mounted into a container at a path the container chose. This is the easy case. The engine can list them by name and report where their driver put the bytes. What it cannot tell you is which of them matter, because the engine has no idea whether a volume holds a rebuildable cache or the only copy of a customer ledger.

The second place is a bind mount: a directory that already exists on the host, grafted into the container at some destination. The engine records the source path and the destination and nothing else. The data’s lifecycle belongs entirely to the host — it survives the container, it is not created or destroyed with it, and it will never appear in a listing of volumes. A team that enumerates volumes and stops there has enumerated the storage the engine manages, not the storage the application uses.

The third place is the container writable layer: everything the process writes to a path that is not a mount. This is ephemeral by construction. In the capture behind this part, the restored service could read its recovered volume data and then failed on the layer file with cat: can't open '/etc/app-marker': No such file or directory. Nothing about that is a bug. It is the container model working as designed, and it matters here because applications write to the writable layer constantly — logs, caches, upload staging directories, sqlite files that someone put in /app/data because no volume was mounted there.

The fourth place is not on this host at all. A container that talks to a managed database, an object store, a message broker or a payment provider has state in those systems, and no amount of mount inspection will reveal it. It is discoverable only from configuration: environment variables, mounted secrets, connection strings in a config file. It is also the place where an enumeration most often stops early, because the mounts came back clean and the list felt complete.

The enumeration is `docker inspect`, not the compose file

A compose file describes what someone intended at the moment it was written. The running container describes what is actually attached. Those diverge for ordinary reasons: an image declares its own mount point and the engine supplies a volume for it; a container was started by hand during an incident and never recreated from the file; a CI job runs a docker run that nobody has read in a year; the compose file has been edited since the stack was last deployed. The enumeration has to come from the engine.

For each running container, ask the engine for its mounts and classify each one:

docker ps --quiet | while read -r cid; do
  docker inspect --format '{{.Name}}' "$cid"
  docker inspect --format \
    '{{range .Mounts}}  {{.Type}} {{.Name}}{{.Source}} -> {{.Destination}} rw={{.RW}}{{println}}{{end}}' \
    "$cid"
done

Each entry in .Mounts carries a Type of volume, bind or tmpfs. A volume has a Name; a bind mount has a Source on the host; a tmpfs has neither of consequence. Destination is where it lands inside the container, and RW says whether the container can write to it.

That is the mechanical half. The half that produces a backup design is one question asked of every row: if this were empty tomorrow, what would put the contents back? There are only four honest answers. The image — a mount that only ever receives content copied out of the image is regenerable by pulling the image again. A pipeline — configuration rendered from a repository on every deploy is regenerable, provided the repository and the pipeline are themselves recoverable, which is a claim about a different system and has to be checked rather than assumed. Another system — the mount is a cache of state whose authoritative copy lives elsewhere, in which case the enumeration follows it there. And nothing — no process, no repository and no upstream would reconstruct it. That last answer is the definition of the data this course exists to protect, and the enumeration exists to find every instance of it.

Read-only mounts are not exempt from the question. rw=false says the container cannot modify the contents; it says nothing about whether anyone else can reproduce them. A read-only bind mount of rendered configuration is regenerable. A read-only mount of a licence file issued once by a vendor is not.

`Mountpoint` and `Driver` describe one engine, not an interface

Having found a named volume, it is reasonable to ask where its bytes are. The previous lesson read these two fields to establish that a volume on the local driver is a directory on the host. The same two fields answer a second question, and this one decides how the backup is written.

Read-only / Safethe two fields worth reading from a volume inspection— EXECUTION VALIDATED on Docker version 29.7.2, build a7dcaa6. Output quoted byte-for-byte from step 3 of docker-volume-state-and-restore.txt, which recorded these two fields rather than the whole JSON document an unformatted inspection returns.

$ docker volume inspect --format '  Mountpoint: {{.Mountpoint}}{{println}}  Driver: {{.Driver}}' rbdr-data

  Mountpoint: /var/lib/docker/volumes/rbdr-data/_data
Driver: local

Two things are true about that path, and they pull in opposite directions.

It is genuinely useful. It is where you measure how much space the volume is consuming, where a forensic examination looks after a container has already been destroyed, and how you confirm that the thing you are looking at on disk is the thing the container was writing to. When a host is being recovered rather than inspected, knowing that the local driver keeps its data under the engine’s data root is what lets you check whether that data survived.

It is also entirely a property of Driver: local. The driver is what decides where the bytes go; the local driver’s answer is a directory under the engine’s data root, and another driver’s answer may be a remote filesystem, a network block device, or something with no meaningful path on this host at all. Build a backup job against that string and you have coupled the job to one driver on one engine, on the assumption that the engine’s on-disk layout is a stable public interface. It is not; it is an implementation detail that the engine is free to change, and that any volume created on a non-local driver already answers differently.

Why the mount point hides what the image put there

A container’s filesystem is assembled before its first process runs. The engine stacks the image’s read-only layers, adds a writable layer on top, and then — still before the process starts — attaches each configured mount inside the container’s mount namespace at its destination path.

That ordering explains the empty directory in the previous lesson. The image had a directory at the mount destination; the volume was mounted over it; the container’s writes landed in the volume, which is outside the image entirely. A commit then captured the image layers plus the writable layer, which is exactly what a commit is defined to capture, and the mounted data was never a candidate for inclusion. When a container starts later from that image with no volume attached, the mount is no longer covering anything and the directory the image shipped shows through — empty, because it always was.

The same ordering explains a behaviour that surprises people during recovery. Docker documents that when a container starts with a volume whose contents are empty, files already present at the destination in the image are copied into the volume. A bind mount does no such thing: it obscures whatever was at the destination and leaves it untouched underneath. So a fresh volume can come up pre-populated from the image and look restored when nothing has been restored at all, while a bind mount pointed at an empty host directory presents an empty directory to an application that expected its data.

Finally, the mount is a mount. From the host, the local driver’s volume directory is an ordinary tree that any host-level backup walking that path will sweep up — as bytes, at whatever instant the walk reaches them, with no coordination whatsoever with the container writing to them.

Three traps that survive a careless enumeration

The volume nobody named. An image can declare a mount point of its own, and a container started from that image with no explicit mount for that path gets an anonymous volume: a real volume, holding real data, with a random name that no human chose and that appears in no compose file. It is listed by the engine alongside every deliberate volume, indistinguishable in a list of dozens except that its name is meaningless. Two failure modes follow. An inventory built by matching volume names against the compose file skips it, because there is nothing to match. And when the container is removed, the volume either goes with it or is left behind referenced by nothing — at which point it is a candidate for any command whose job is to remove volumes nothing is using. State that nobody named is state that nobody will miss until it is needed.

The bind mount covered by accident. A container bind-mounts /srv/appdata, the host backup covers /srv, and the data is protected. The coverage is real. It is also incidental: it exists because two paths happen to overlap, not because anyone decided that this application’s data should be backed up. Move the data to /opt/appdata, deploy the same stack on a host that uses a different layout, or let a new compose file change the source, and the coverage disappears silently. Nothing fails. The backup job still runs, still exits 0, and now covers less — the exact shape of the problem this course opened with, where a green job carries no information about what it contains. Coverage derived from a path is coverage that expires the next time the path moves.

The mount that is meant to be lost. A tmpfs mount is deliberately volatile: Docker documents it as storage held in the host’s memory only, never written to the host filesystem, and removed when the container stops. It should not be backed up, and the useful thing the enumeration does with it is invert the question — not how do we protect this, but does anyone believe this survives? An application configured to write uploads, exports or session data to a path that turns out to be tmpfs is losing data continuously, and the mount type is where that shows up.

The helper container is the portable interface

The pattern is a throwaway container that mounts the volume read-only, mounts a destination the engine can also see, and streams an archive out.

Read-only / Safebacking up a named volume without touching the engine's data root— EXECUTION VALIDATED on Docker version 29.7.2, build a7dcaa6. Step 3 of docker-volume-state-and-restore.txt; the output line is the resulting archive on the host.

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

Written as something a schedule can run, it is one command with the volume name and the destination lifted out:

VOL=rbdr-data
OUT=/srv/backup/volumes
docker run --rm \
  -v "$VOL":/src:ro \
  -v "$OUT":/out \
  alpine tar czf "/out/$VOL.tgz" -C /src .

The :ro matters. The helper physically cannot modify the data it is copying, which removes an entire class of accident from a job that runs unattended against production state.

The restore is the same shape with the tar flags reversed and a destination volume that does not exist yet, which is how the archive above was proved: the previous lesson unpacked those 160 bytes into a brand-new rbdr-data-restored after the original container and volume had been destroyed, and both sides hashed to 9eb4e2ad8e08e1dcaaf87ababab964b0. Symmetry is the property worth noticing. A backup route whose restore is a different mechanism, run by a different person, against a path that only exists on the original host, is a route that gets tested on the day it is needed.

Prefer this to reaching into the engine’s storage directory for four reasons. It is driver-independent: the helper asks the engine for the volume by name and the engine mounts it, whatever the driver would have answered for Mountpoint. It survives an engine upgrade that reorganises the data root, because it never referenced the data root. It works against an engine reached over a socket, where the path in Mountpoint names a filesystem the backup job cannot see at all. And it needs no privileged read of a directory the engine owns — the helper is granted exactly one volume, read-only, by name, which is a far narrower grant than a backup agent walking the whole data root.

What the pattern does not give you is consistency. A tar of a live volume is a crash-consistent copy of files that a running container may be part-way through writing, which is adequate for some workloads and useless for a database. That question is the subject of the next lesson.

Three shortcuts that produce an unrestorable estate

“We back up the whole host, so /var/lib/docker is covered.” It is covered as bytes, taken while the engine and every container were writing. Walking a live data root also sweeps in image layers and engine metadata, which inflates the backup with material a registry already holds, and it produces exactly the crash-consistent copy of a running database that a restore will refuse.

“The images are in the registry, so we’re covered.” The registry holds what was built. It holds no volume, no bind mount and no writable layer, which the previous lesson measured directly.

“The compose file is in Git, so we can rebuild it.” Git holds the declaration. The anonymous volume the image created is not in the declaration, and neither is a single byte the application wrote.

Production discipline

Enumerate from the engine, not from the repository. Inspect the mounts of every running container and classify each as volume, bind or tmpfs. The compose file is a statement of intent; the container is the fact.
Ask the regeneration question of every mount and write down the answer. Image, pipeline, another system, or nothing. Only the last requires a backup, and only an explicit answer distinguishes it from the third, which moves the enumeration to a system nobody has looked at yet.
Name every volume you intend to keep. An anonymous volume holding real state is state your inventory cannot match and your retention tooling cannot distinguish from an abandoned one.
Convert incidental bind-mount coverage into a stated dependency. If a host job protects a container’s data because two paths overlap, record that the overlap is load-bearing and re-run the enumeration after any change to either path. A backup that silently covers less still exits 0.
Back volumes up through a container, not through the data root. Mount the volume read-only into a helper, stream an archive out, and prove the restore into a volume that did not exist before — the measured run produced a 160-byte archive and returned 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides after the source volume had been destroyed.

Cross-course references

Docker & Containers for Production Sysadmins — Part IX (Docker Compose) covers what a compose file declares and how the engine turns it into running containers. That is the gap this lesson turns into a method: the compose file states an intention, the engine states what is attached now, and an enumeration built for backup has to be taken from the second.
Kubernetes for Production Sysadmins — Part L (PersistentVolumes and Claims) is where the same enumeration moves once a stack becomes a cluster workload. The four hiding places survive the move — the claim replaces the named volume, hostPath replaces the bind mount, the pod filesystem replaces the writable layer — and the regeneration question is asked unchanged.
Linux for Production Sysadmins — Part XV (/etc/fstab and Mount Management) explains the mount semantics underneath all of this, including a mount covering an existing directory rather than merging with it. That is precisely the mechanism that makes an image’s directory reappear empty once the volume over it is gone.