Backing up container data with consistency

Knowing where container state lives tells you which directory holds the bytes; it does not tell you when you are allowed to read them. A named volume is an ordinary directory on the host, and the boundary that makes a workload feel self-contained gives you nothing in exchange — no quiescing, no fixed instant, no notion of a moment the data was coherent. Every consistency problem from Part IV arrives here intact, in a smaller box and with more confident vocabulary around it. The question is narrow: given a volume something is writing to, what does copying it actually produce, and what does something better cost?

Four copies of the same volume, in increasing order of correctness

There are four things an operator can do with a volume that holds live data, and they are not four styles of the same job. They differ in what each one knows about the application’s write state at the moment the bytes are read.

1. Archive the volume while the workload runs. A helper container mounts the volume and writes a tar archive; nothing is told anything. The archive is crash-consistent at best, and for a database usually not even that, because tar reads files one at a time over several seconds while pages underneath it move. It costs nothing, which is why most estates start here.
2. Stop the container, archive the volume, start it again. The workload completes its shutdown, flushes what it was holding and closes its files, so the tree tar reads is closed and quiet. The result is genuinely application-consistent. The price is an outage lasting the whole copy, which grows with the data.
3. Snapshot the filesystem under the volume and archive the snapshot. The pause shrinks from the length of the copy to the length of a snapshot creation, and the archive is taken from the frozen instant while the service runs. This needs storage that can snapshot, and on its own it still yields a crash-consistent image unless the workload was quiesced first.
4. Have the application produce its own backup and archive that output. The database writes a base backup, or a dump, into a directory of its own, and the backup job copies a file the application declared complete. Nothing is appending to that file while it is read.

The list is ordered by ignorance rather than by effort: the first option knows nothing about the workload, the fourth is executed by it. Two and three buy correctness with an outage and with a storage requirement, which is why estates use all four in different places rather than standardising on one.

The copy that started, ran crash recovery and returned all 45000 rows

The first option is dangerous precisely because it usually appears to work, and the clearest demonstration of that is not a container at all. A PostgreSQL 18.6 cluster was running a write workload while a plain cp -a walked its data directory into a second location, with no database involvement of any kind. That copy was then started as a cluster of its own.

$ pg_ctl -D /work/naive-copy start

  waiting for server to start.... done
server started
>>> exit code: 0

2026-08-28 13:34:37.611 UTC [94] LOG:  database system was interrupted; last known up at 2026-08-28 13:34:36 UTC
2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

Every observable signal is green: exit code 0, database system is ready to accept connections, and a row count identical to the live database. An operator who tested this backup by starting it and running a SELECT count(*) would record a successful restore and move on.

The result is the problem rather than the reassurance. The log says the cluster was not properly shut down and went through automatic recovery — the path it would take after a power cut. Replay ran from redo starts at 0/17615F8 to redo done at 0/256A8B0, halting on an invalid record length, which is how redo recognises the end of the written log rather than a sign of corruption. All that establishes is that this copy happened to hold every record redo asked for. PostgreSQL’s documentation is explicit that a file-system copy of a running cluster is not a valid backup unless it is bracketed by pg_backup_start and pg_backup_stop and accompanied by all the WAL generated during the copy, and cp did none of that while walking the tree over several seconds with pages moving underneath it. What happened was luck, and luck has no failure signal.

The same directory without pg_wal did not start at all

The second half of the capture removes the luck. The same copy was started with pg_wal emptied, which models an arrangement that is extremely common: the data on one volume, the write-ahead log on another, and a backup job pointed at the data volume because that is the one that obviously holds the database.

$ pg_ctl -D /work/nowal start

  waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
>>> exit code: 1

2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/archive_status"
2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/summaries"
2026-08-28 13:34:37.879 UTC [132] LOG:  invalid checkpoint record
2026-08-28 13:34:37.879 UTC [132] PANIC:  could not locate a valid checkpoint record at 0/2F20158
2026-08-28 13:34:37.941 UTC [126] LOG:  startup process (PID 132) was terminated by signal 6: Aborted
2026-08-28 13:34:37.941 UTC [126] LOG:  terminating any other active server processes
2026-08-28 13:34:37.942 UTC [126] LOG:  shutting down due to startup process failure
2026-08-28 13:34:37.943 UTC [126] LOG:  database system is shut down

The first two log lines are the trap in this outcome. Startup finds pg_wal/archive_status and pg_wal/summaries absent and creates them, which reads like a system repairing itself; what it has repaired is the shape of the directory, and nothing can repair the contents. The copy’s pg_control names the LSN of the last checkpoint — 0/2F20158 — but that checkpoint is itself a WAL record, and WAL is what the copy left behind. Without a redo point there is no defined way to reconcile the data files against anything, so the startup process aborts under signal 6 instead of guessing: exit code 1, no cluster, and the PANIC naming the LSN it wanted.

Both transcripts used the same copy method within the same second; the only difference is what the copy contained. The method does not decide whether you get your data back — the contents do, and the method cannot guarantee them. A database in a container carries that requirement unchanged: if a Compose file gives it one volume for PGDATA and another for the write-ahead log, a reasonable layout for I/O separation, and the backup job archives the volume named after the data, the restore is the second transcript. The container boundary changes nothing, because the requirement belongs to the database rather than to the packaging.

Buying consistency with an outage, or with a snapshot

The second option is the honest version of the file copy, and it is short enough to write down completely: stop the container, archive the volume through a helper, start the container. With the workload exited, the tree tar reads is closed and quiet, and the archive is application-consistent for the same reason a backup of a powered-off machine is.

set -euo pipefail

CONTAINER=orders-db
VOLUME=orders-pgdata
STAMP=$(date +%Y%m%dT%H%M%S)

docker stop -t 300 "$CONTAINER"
docker run --rm -v "$VOLUME":/src:ro -v /srv/backup:/out alpine \
  tar czf "/out/$VOLUME-$STAMP.tgz" -C /src .
docker start "$CONTAINER"

The -t 300 is easy to leave out and expensive to leave out. The CLI reference for docker container stop states that the daemon determines the default timeout and that it is 10 seconds for Linux containers, after which the process is killed with SIGKILL. A database needing longer than that to write its shutdown checkpoint is killed mid-shutdown, and the volume you archive is then a crash-consistent directory recorded as an application-consistent backup: the outage was paid for and the correctness was not received.

The timeout is only half of it. The reference is precise about what a stop does: the main process inside the container receives SIGTERM, and after the grace period, SIGKILL. The main process is PID 1 in the container, which is whatever the image’s entrypoint started — and if that is a shell wrapper that launched the database as a child, PID 1 is a shell. The shell exits on SIGTERM without passing anything on, the database is killed later with no chance to write a shutdown checkpoint, and the stop still looks clean from the outside while producing exactly the volume the first option produces. Whether a container shuts its workload down cleanly is a property of the image’s entrypoint rather than of the stop command, and it is worth establishing once per image instead of assuming it on every backup run.

The third option keeps the reasoning and shortens the window. Snapshot the filesystem carrying the volume — the capture reported the volume at /var/lib/docker/volumes/rbdr-data/_data, so that is whichever filesystem holds /var/lib/docker/volumes — then archive from the snapshot while the service is back up.

set -euo pipefail

SNAP=/mnt/dockervol-snap

docker stop -t 300 orders-db
lvcreate --size 10G --snapshot --name dockervol-snap /dev/vg0/dockervol
docker start orders-db

mount -o ro /dev/vg0/dockervol-snap "$SNAP"
tar czf /srv/backup/orders-pgdata.tgz -C "$SNAP/orders-pgdata/_data" .
umount "$SNAP"
lvremove -f /dev/vg0/dockervol-snap

The service is down for the stop and the lvcreate, not for the archive, which is the entire point. Drop the stop and the start and the sequence still runs and still produces an archive — a crash-consistent one, an image of a real instant holding whatever had reached disk by then, including transactions partway through.

The helper container moves bytes; it does not make them consistent

The portable way to capture a named volume is to mount it into a throwaway container alongside a destination and let that container write the archive. It needs no tooling on the host and never reaches into the daemon’s storage directory.

$ docker run --rm -v rbdr-data:/src:ro -v $PWD:/out alpine tar czf /out/rbdr-data.tgz -C /src .

  -rw-r--r-- 1 root root 160 Aug 28 13:49 /tmp/rbdr-out/rbdr-data.tgz

The capture then did the part that turns a rehearsal into a test: it destroyed the container and the volume — a lookup afterwards returned Error response from daemon: get rbdr-data: no such volume with exit code 1 — and unpacked the archive into a volume that had never held anything.

$ docker run --rm -v rbdr-data-restored:/dst -v /tmp/rbdr-out:/in alpine tar xzf /in/rbdr-data.tgz -C /dst

  restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the volume data was recovered byte-identical

Read what that verification proves and what it does not. The md5 recorded before the volume was destroyed and the md5 read after the restore are the same string, so the archive is a faithful transport of a two-line CSV nothing was writing to. Run the same comparison against a live database volume and it still passes, telling you only that the archive preserved whatever inconsistent set of pages tar happened to read. Transport fidelity and application consistency are separate properties, and the helper container supplies one of them.

That distinction retires the most common container restore test. Unpacking an archive into a fresh volume, starting the container against it and watching the service come up establishes that the image runs and that the volume is mountable. It is the container-shaped version of the first transcript, where a cluster started, announced ready to accept connections and told you nothing about the copy it started from. A restore test has to compare recovered content against something recorded before the backup — a hash, a row count, a business checksum — because that is the only comparison a workload cannot pass by merely starting.

Back up what the application produces

The fourth option removes the question rather than answering it. Instead of archiving the directory the database writes to, run the database’s own backup command and archive its output. In the same capture, pg_basebackup -D /work/base -X stream -c fast exited 0 and produced a backup containing 45000 rows. The difference from the naive copy is not the row count, which was also 45000 there; it is that the server took part, so the start and the end of the copy are boundaries the database chose rather than whenever cp happened to begin and finish.

set -euo pipefail

STAMP=$(date +%Y%m%dT%H%M%S)

docker exec orders-db pg_basebackup -D "/backup/base-$STAMP" -X stream -c fast
docker run --rm -v orders-backup:/src:ro -v /srv/backup:/out alpine \
  tar czf "/out/orders-base-$STAMP.tgz" -C /src "base-$STAMP"

The backup volume is written by one process, at a moment that process chooses, and read by the archiver afterwards; nothing is appending to it while tar walks it. The same shape applies beyond PostgreSQL — a message broker exporting its queues and definitions, a key-value store writing a point-in-time file, an application with an export endpoint. In each case the artefact is produced by the component that knows what a complete state is, and the backup system’s job shrinks to moving a finished file and proving it arrived.

Production discipline

1. Name the consistency level of every container backup you own, in writing. For each volume, record which of the four options produced it and therefore whether the archive is untimed, crash-consistent or application-consistent. An estate that cannot answer this per volume does not know what it can recover.
2. Set the stop timeout to exceed the workload’s shutdown, and verify the exit. The docker container stop reference gives the daemon default as 10 seconds for Linux containers; a database killed at that point leaves a crash-consistent volume labelled as a clean one. Check that the container stopped rather than was killed before the archive step runs.
3. Never split a data directory across volumes you back up separately. The capture panicked with could not locate a valid checkpoint record at 0/2F20158 when only the data volume was present. If the layout separates data from write-ahead log, the file-level backup covers both or neither.
4. Verify restores against recorded content, not against container startup. The naive copy started, reported ready to accept connections and returned 45000 rows while guaranteeing nothing. Compare a hash or business checksum captured before the backup, the way the volume restore matched 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides.
5. Back up what a stateful container produces, not the directory it writes to. For a database or a message queue, archive the output of a backup command the application declared complete. That is the only one of the four options where the consistency question is answered by the component able to answer it.

Cross-course references

• Docker & Containers for Production Sysadmins — Part XXI (Backup) develops the volume capture and restore procedures for a Docker estate in full, including the helper-container pattern used above; this lesson supplies the consistency test that decides whether those procedures yield a restorable database or merely a restorable directory.
• PostgreSQL for Production Sysadmins — Part XII (WAL, Checkpoints and Crash Recovery) explains the checkpoint record and the redo mechanism that produced both outcomes quoted here, and it is the reason a data volume separated from its write-ahead log is not a backup of anything.
• Linux for Production Sysadmins — Part XVI (LVM) covers the snapshot mechanism the third option depends on, including the copy-on-write cost of holding a snapshot open on the filesystem that carries /var/lib/docker/volumes while the container keeps writing to it.

Quiz