A container image is not a backup · Backup & DR

Losing a whole virtualisation cluster left an estate that had to be rebuilt before anything could be restored into it, and every unit of state in that exercise had a name somebody could point at: a disk image, a datastore, a configuration file on a host. Containers break that habit. They put state in places that are created and destroyed by the same command that starts and stops a process, and estate inventories rarely list those places at all. That is why container estates are so routinely under-protected, and the correction is not a new tool. It is knowing which of three layers a given byte is sitting in.

Three places, three lifetimes, one command that captures the wrong one

A container image is an ordered stack of read-only layers. Each layer is a filesystem changeset — a set of added, changed and removed paths recorded against the layer beneath it — and the stack is produced from a definition, usually a Dockerfile, then addressed by digest. An image is immutable in one narrow sense: the bytes of a layer cannot be altered without changing the digest that names it. Immutable there does not mean retained, protected or recoverable. It is a property of content addressing, and it says nothing about whether that digest will still resolve in a registry next quarter.

When the daemon starts a container from an image it adds one more layer on top, and that one is writable. Every file the process creates or modifies at a path that is not covered by a mount lands there. The writable layer belongs to the container rather than to the image, and it is discarded when the container is removed — which is an ordinary event, not an incident, because replacing the container is how a containerised service gets upgraded, rescheduled, or brought back after a host reboot.

The third place is not a layer at all. A volume or a bind mount is storage attached to the container at a path, supplied at run time. It outlives the container, and — this is the part that gets missed — it is not part of the image and never becomes part of one. The image knows a directory exists at that path. It does not know, and cannot carry, what is behind the mount.

So ask where a file lives before asking how it is protected. The answer decides which mechanism can reach it, and two of the three places are invisible to the mechanism people reach for first.

Measured: what `docker commit` captured, and what it did not

The claim is testable, so it was tested rather than asserted. On Docker 29.7.2, build a7dcaa6, an application container was started with a named volume rbdr-data mounted at /var/lib/app. Two files were written: orders.csv into the volume, holding two order lines and hashing to md5 9eb4e2ad8e08e1dcaaf87ababab964b0, and /etc/app-marker into the container’s own filesystem at a path with no mount over it — the writable layer.

Then docker commit, which is the capture most people reach for when they want “a copy of this container before we change it”.

Configuration changedocker commit — the backup people think they are taking— EXECUTION VALIDATED on Docker 29.7.2, build a7dcaa6. A new container was then started from the committed image with no volume attached.

$ docker commit rbdr-app rbdr-committed:v1

  image created

--- start a NEW container from that committed image, with NO volume ---
written-into-the-container-layer
>>> exit code: 0

--- and the file that was in the volume? ---
total 8
drwxr-xr-x    2 root     root          4096 Aug 28 13:48 .
drwxr-xr-x    1 root     root          4096 Aug 28 13:48 ..
>>> exit code: 0

The two halves point in opposite directions. The file written into the writable layer came back: a fresh container started from rbdr-committed:v1 printed written-into-the-container-layer and exited 0, so the commit captured something real and captured it faithfully.

The directory that had been the volume’s mount point came back with total 8 and two entries, . and ... Empty. Not corrupt, not partial — the order data was never in the image, so there was nothing for the new container to show. One image, and the two files ended up on opposite sides of the boundary from the one an operator would guess.

The volume is a directory on the host with a name attached

If the volume contents are not in the image, they are somewhere, and it is worth being precise about where. The capture asked the daemon: the volume’s Mountpoint was /var/lib/docker/volumes/rbdr-data/_data and its Driver was local. That is the whole trick of the local driver — a named volume is a directory on the host, in a path the daemon manages, presented to the container at whatever target path the run command asked for.

Which means it can be backed up like any other directory, by either of two routes. The host route reads /var/lib/docker/volumes/... directly and needs the writer quiesced to be worth anything. The portable route mounts the volume into a short-lived helper container and streams an archive out, which behaves the same way whatever driver is behind the volume.

Read-only / Safebacking up a named volume through a helper container— EXECUTION VALIDATED on Docker 29.7.2, build a7dcaa6. The volume held one two-line CSV, so the resulting archive is small.

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

The :ro on the source mount is deliberate: the helper has no business writing to what it is reading. Note also what the archive is not. Those 160 bytes are volume contents — not the application, its image, its configuration or the container-layer file — and a restore procedure that assumes otherwise is wrong in a way that only shows up when it is used.

The two routes also fail differently. The host route is an artefact of the driver: the capture’s volume reported Driver: local, and it is the local driver that puts the bytes under /var/lib/docker/volumes/rbdr-data/_data. The Docker volumes documentation describes volume drivers that place a volume’s storage elsewhere, including on a remote host, and under one of those the host path is not where the data is — so a job hard-coded to that directory would exit 0 and archive nothing. The helper-container route asks the daemon for the mount rather than guessing at a path, which is why it survives that change and belongs in any procedure written for an estate somebody else configured.

Neither route says anything about the state of the writer. tar czf walks a tree and reads each file when it reaches it, so a volume written during the walk yields files captured at different instants, and whether those reassemble into something the application will start is a property of the application, not the archive. Nor should default flags be assumed to carry everything a directory holds: measured elsewhere in this course on GNU tar 1.35, an archive taken with default flags restored names, sizes and contents intact while the source’s ACL came back with zero entries and its extended attribute came back ABSENT. Whether a given helper image’s tar behaves the same way is worth testing. For a volume holding one CSV it does not matter. For one holding a mail spool or a certificate store, where ownership and ACLs decide who may read a file, it is the difference between a restore and a directory of correct bytes nobody can open.

Why the mount point listed as empty

Two different mechanisms are stacked at that path, and the capture separated them cleanly.

The union filesystem builds what the container sees. Under the OverlayFS storage driver the image’s read-only layers are the lower directories and the container’s writable layer is the upper directory; the kernel presents the merged result as one tree. A write to /etc/app-marker goes into the upper directory. If a file of that name already existed in a lower layer, it is copied up first and then modified — the lower layer is never touched, which is what makes it shareable between every container running that image.

docker commit freezes that upper directory and appends it to the image as one more layer. That is the entire operation. It is not a snapshot of the container, it is a snapshot of the container’s diff against its image, and the diff is exactly where /etc/app-marker was.

The changeset format makes that diff character explicit. Per the OCI Image Layer Filesystem Changeset specification, a layer tar carries additions and modifications as ordinary entries and expresses a removal as a whiteout: an empty file named with the target’s basename prefixed by .wh., so deleting /etc/my-app-config records ./etc/.wh.my-app-config in the layer rather than removing anything. Whiteouts apply only to lower layers, and a whole directory can be blanked with the opaque marker .wh..wh..opq. Two consequences follow for anyone treating a commit as a copy. Deleting data and committing does not shrink the result — the original bytes remain in the layer beneath, hidden rather than gone. And a layer is a set of instructions for changing a filesystem rather than a filesystem, so what an image reconstructs depends entirely on which layers sit underneath it.

A volume is not in that union. The daemon bind-mounts the volume’s host directory over the target path inside the container’s mount namespace. Reads and writes under /var/lib/app resolve through that mount to the host directory and never reach any layer — so when the commit walked the upper directory, the order data was not there to find. It was not skipped or filtered. It was on the other side of a mount boundary.

That also explains the empty listing. The image carries whatever the directory held when it was built, and here that was nothing; with no volume attached the mount is absent, so the path shows the image’s own empty directory. (Per the Docker volumes documentation, an empty named volume mounted over a path that does hold files in the image copies them into the volume on first use, which makes the boundary harder still to see.)

The general principle outlives Docker: a capture that walks a filesystem captures what that filesystem contains, and a mount point contains a mount, not the data behind it.

Destroy both, restore one

An image and an archive that have never been exercised are two claims. The capture converted them into results by removing the container and the volume, which is what a host rebuild, a docker volume prune on the wrong machine, or a mistyped compose teardown does in production.

Data-loss riskafter the container and the named volume were removed— EXECUTION VALIDATED on Docker 29.7.2, build a7dcaa6. Both the container and the volume were destroyed before this lookup.

$ docker volume inspect rbdr-data

  []
Error response from daemon: get rbdr-data: no such volume
>>> exit code: 1

Nothing recoverable from the daemon. What existed at that point was one 160-byte archive on a different filesystem, and it was unpacked into a brand-new volume — clean infrastructure, not the original.

Service impact possiblerestoring the archive into a new volume on clean infrastructure— EXECUTION VALIDATED on Docker 29.7.2, build a7dcaa6, immediately after the removal above.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

--- the restored service reads its data ---
ORDER-1001,4500.00
ORDER-1002,1250.00
restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

--- but what about the file that lived in the container layer? ---
cat: can't open '/etc/app-marker': No such file or directory
>>> exit code: 1

The volume half of this is a clean result and worth stating plainly: the same md5 on both sides, 9eb4e2ad8e08e1dcaaf87ababab964b0, from an archive taken before the destruction and unpacked after it. That is a proven restore of the volume, by the standard this course has used since Part I — bytes compared, not a job that exited 0.

The other half is the lesson. /etc/app-marker is gone, with cat: can't open '/etc/app-marker': No such file or directory and exit code 1. It had been captured, once, into an image still sitting in the local image store. But the procedure that was written and rehearsed was the volume procedure, so the volume is what came back, and everything written into the container layer was outside every mechanism running on a schedule.

What the registry does and does not tell you

Three conclusions follow, and each contradicts something people say about container estates.

An image registry is not a backup system. It stores definitions — the artefact you would rebuild a container from, which is worth protecting and is protected by ordinary means. It stores no runtime state, because runtime state was never pushed to it. A registry holding every image an estate runs tells you the estate can be reconstructed empty, and nothing about the orders, uploads or databases those containers were holding.

docker commit captures the wrong half. It is not that commit fails; it succeeded above and its result was verified. It captures the container’s diff against its image, which is where accumulated drift lives, and structurally cannot reach what is behind a mount. Used as a pre-change safety copy it produces something that looks like a backup, runs under docker run, and omits the data.

State in a container layer is state nobody is protecting. Not because the tools are inadequate, but because nothing points at it. It is not in the image CI builds, not in the volume the backup job archives, and it disappears on the next docker rm — an event that happens on every deploy. The only sound handling is to keep durable data out of it: mount the path, and back up the mount.

What to take from this

Container state lives in three places with three lifetimes: read-only image layers, the container’s writable layer, and attached storage. Only the first is in the image, and only the third survives docker rm on its own.
Measured on Docker 29.7.2: docker commit captured /etc/app-marker from the writable layer — a new container printed written-into-the-container-layer and exited 0 — and captured nothing from the volume, whose mount point listed as total 8 with only . and ...
A named volume on the local driver is a host directory; the capture reported Mountpoint: /var/lib/docker/volumes/rbdr-data/_data. Streaming it out through a helper container produced a 160-byte archive that was independent of both the container and the volume.
After both were destroyed, docker volume inspect returned Error response from daemon: get rbdr-data: no such volume at exit 1, and the archive restored into a new volume with md5 9eb4e2ad8e08e1dcaaf87ababab964b0 matching on both sides.
The container-layer file was gone in the same restore, with cat: can't open '/etc/app-marker': No such file or directory at exit 1. It was captured into an image and it was still not part of any restore that was being practised.

Cross-course references

Docker & Containers for Production Sysadmins — Part VIII (Storage) develops the writable-layer, volume and bind-mount distinction that this lesson measures, and Part XXI (Backup) builds the volume backup and restore procedure whose boundary is exactly the one established here.
Kubernetes for Production Sysadmins — Part XLIX (Volumes) and Part XCVI (Workload Backup) carry the same split into an orchestrator, where a rescheduled pod replaces the container layer routinely and the only surviving state is what a PersistentVolume holds, which makes the distinction taught here operationally unavoidable rather than merely correct.
Linux for Production Sysadmins — Part LXXVIII (Containers from the Linux Perspective) explains the mount namespace and union filesystem underneath the behaviour above, which is why a mount point in a committed image contains a mount rather than the data behind it.