Crash-consistent, application-consistent, and the gap between them

Capacity planning closed Part III by treating a backup as a volume of bytes to be sized, retained and paid for, and every calculation in it was indifferent to what those bytes actually contained. This part asks the question those calculations skipped: which moment did the contents come from? A copy can be complete, verified, correctly sized and held in three places, and still describe a state the system was never in. There are two established terms for this and an enormous unnamed gap between them, and most of the data quietly lost to bad backups is lost inside that gap.

Crash-consistent is the state a power cut would have left

Crash-consistent has a precise meaning and it is worth holding to it. A crash-consistent copy captures the contents of a storage volume as they stood at a single instant, with no participation from anything running above it. Every write that had reached the device before that instant is present, every write that had not is absent, and the copy contains no ordering that did not exist on the device. That is exactly the state the media would be in if power had been removed at that instant, which is where the name comes from.

What the term excludes is everything still held above the storage layer. Data sitting in an application buffer, a page in the kernel page cache that has not been written back, a transaction the client believes is still in progress — none of that is in the copy, because none of it had reached the device.

The reason crash consistency is useful rather than merely descriptive is that serious software is engineered to start from it. A journalling filesystem replays or discards its journal on mount and returns to a structurally coherent state. A database with a write-ahead log replays that log from its last checkpoint, reapplying the committed transactions the log covers and discarding the rest. Recovery from a crash-consistent copy is a designed code path with a defined outcome, not a lucky escape.

Two limits travel with the term. The first is that recovery is only as capable as the software doing it: a journalling filesystem guarantees that its metadata is coherent, not that the contents of a file that was mid-write are the contents anybody wanted, and an application that keeps state in memory and writes no log has nothing to replay and gets nothing back. The second is that the single instant applies only to what was captured in that instant. Two volumes snapshotted by two separate operations are two instants, and a database whose data files and log are split across them has no crash-consistent image at all — which one of the transcripts below shows failing very loudly.

Application-consistent is a state a component agreed to

Application-consistent means something categorically different. It means that a component which understands the data was told a copy was about to be taken, and did something about it: flushed what it was holding, wrote whatever marker its own recovery path will need, and in the strongest form held new writes still until it was told the copy had finished. The consistency is not a property the storage produced. It is a claim the application made, and it exists only if some interface was actually used to ask for it.

PostgreSQL is explicit about what that interface is. Its documentation states that a file-system-level backup of a running cluster is valid only when the copy is bracketed by pg_backup_start and pg_backup_stop, and accompanied by all the WAL generated between the two calls. The bracketing is not a formality: an online backup is always started at the beginning of a checkpoint — by default the start call waits for the next regularly scheduled one, and it can be asked to request an immediate one instead — and the location that checkpoint establishes is where replay must begin. The WAL collected across the interval is what turns a smeared set of files into a coherent one, and pg_backup_stop identifies the last segment needed to complete the set. pg_basebackup performs the same handshake on your behalf; the documentation is unambiguous about what its -X option adds, which is that all the write-ahead log required to use the backup is included in the backup automatically, so no special action is required at restore time to find it somewhere else.

$ pg_basebackup -D /work/base -X stream -c fast

  >>> exit code: 0
rows contained in the base backup: 45000

Replaying that base backup later produced the log line consistent recovery state reached at 0/3000120, immediately after completed backup recovery with redo LSN 0/3000028 and end LSN 0/3000120. The consistency point is the backup end location, not the redo location, and the gap between the two — 0/3000028 to 0/3000120 — is exactly the interval the bracketing covered. Replay becomes consistent when it has passed the end of the copy, because that is the first instant at which the copied files plus the replayed log describe one coherent state. Before that point the file set describes nothing in particular, which is the honest description of any copy taken while writes continue.

The useful operational question is therefore not whether a copy is application-consistent, but which component declared it and through which call. If nothing was told anything, the copy is at best crash-consistent, and possibly not even that.

The third state, which has no name and no instant

Between those two terms sits a third state that neither describes, and it is the state most copies of live data actually have.

cp -a, rsync, tar and every file-level backup agent walk a directory tree. They open one file, read it, close it and move to the next, and the walk takes as long as it takes — seconds for a small tree, hours for a large one. If nothing is writing, the result is a faithful copy of a stable state. If something is writing, the result is a set of files each captured at a different moment: one file as it stood at the start of the walk, another as it stood several seconds later, a third caught halfway through being rewritten. There is no instant in the past at which the storage held that particular combination.

PostgreSQL’s own documentation says this about its own data directory in the plainest possible terms: the server must be shut down to get a usable file-system-level backup, and half-way measures such as disallowing connections will not work — in part because tar and similar tools do not take an atomic snapshot of the state of the file system, and in part because of buffering inside the server. Both halves of that sentence matter, and they are the two independent failures a tree walk commits at once.

This is weaker than crash-consistent, not stronger, and the vocabulary hides it because there is no common word for it. A power cut at least produces a real state, one the recovery machinery was designed for. A tree walk under load produces a composite of many states that no recovery machinery was designed for, because it never occurred. Nor is the problem confined to the boundaries between files: a large file read over several seconds can have a beginning from one moment and an end from another, with internal structure matching neither.

Nothing in the copying tool detects any of this. Every file opened successfully, every byte requested came back, and the exit status is 0. The tool answered the question it was asked, which was whether the reads succeeded. No tool that walks a tree can answer the question that matters, which is whether the tree stood still.

Measured: cp -a of a live PGDATA started, recovered and returned 45000 rows

The honest way to teach this is to run it and report what happened, including the part that is inconvenient. A PostgreSQL 18.6 cluster was seeded and put under a write workload. While the workload ran, the data directory was copied with cp -a — no pg_backup_start, no snapshot, no cooperation from the database whatsoever — and the copy was then started as an independent cluster.

$ pg_ctl -D /work/naive-copy start

  waiting for server to start.... done
server started
>>> exit code: 0

2026-08-28 13:34:37.611 UTC [94] LOG:  database system was interrupted; last known up at 2026-08-28 13:34:36 UTC
2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

Read that carefully before drawing the wrong conclusion from it. The copy started. It went through crash recovery exactly as it would have after a power cut — database system was not properly shut down; automatic recovery in progress, redo from 0/17615F8 to the end of the written log, then database system is ready to accept connections. And it returned 45000 rows, the same count as the live database it was taken from.

That result is the problem, not the reassurance. Nothing in the exercise established that the copy is a transaction-consistent image of any instant. cp walked the tree over several seconds while pages were being written underneath it, and this particular copy happened to contain every WAL record that the redo point it carried turned out to need. A different workload, a larger directory or a checkpoint landing at a different moment during the walk, and the same command produces a directory that starts just as cheerfully while missing changes nobody will go looking for.

The redo has to be inside the same captured set

The same capture shows the other outcome, and it is the one to hope for. A very common arrangement puts the data on one volume and the write-ahead log on another, then snapshots the data volume — because that is where the data is, and the log volume looks like scratch space. The copy above, with pg_wal emptied to model that arrangement, was started the same way.

$ pg_ctl -D /work/nowal start

  waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
>>> exit code: 1

2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/archive_status"
2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/summaries"
2026-08-28 13:34:37.879 UTC [132] LOG:  invalid checkpoint record
2026-08-28 13:34:37.879 UTC [132] PANIC:  could not locate a valid checkpoint record at 0/2F20158
2026-08-28 13:34:37.941 UTC [126] LOG:  startup process (PID 132) was terminated by signal 6: Aborted
2026-08-28 13:34:37.941 UTC [126] LOG:  terminating any other active server processes
2026-08-28 13:34:37.942 UTC [126] LOG:  shutting down due to startup process failure
2026-08-28 13:34:37.943 UTC [126] LOG:  database system is shut down

The startup process panicked with could not locate a valid checkpoint record at 0/2F20158, and the command exited 1. Notice the two lines above it in the log: the server created the missing write-ahead log directories quite happily, which is exactly the sort of detail that persuades a hopeful operator the problem is cosmetic. It is not. The data files describe a state that only the redo can complete, and the redo was not in the set that was captured.

That is the general rule underneath both transcripts. Crash consistency is a property of a captured set at an instant, and the set has to contain everything the recovery path will read. A snapshot that is atomic over one volume and a database that spans two volumes do not compose into a recoverable copy, however atomic each snapshot was. The documentation states the constraint directly: where a database is spread across multiple file systems there may be no way to obtain exactly-simultaneous frozen snapshots of every volume, and because the snapshots must be simultaneous, that arrangement can rule out snapshot backup altogether. Simultaneity is not an optimisation here; it is the whole property. The same reasoning applies to a virtual machine disk snapshotted separately from a second disk holding the log, and to a filesystem snapshot taken while the data that matters is still sitting in an application buffer nobody asked to be flushed.

Instant and scope are two separate questions

Consistency is a property of a set of bytes captured together, so before asking when the set was captured it is worth asking what was in it. A container capture makes the point cleanly. In the Docker 29.7.2 evidence recorded for this course, an application wrote one file into a named volume and one into the container writable layer. The volume was archived through a helper container, the container and the volume were then destroyed, and the archive was restored onto a fresh volume: restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 matched original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 exactly. The file that had lived in the container layer produced cat: can't open '/etc/app-marker': No such file or directory.

The restored bytes were perfect. The copy was still incomplete, because the set it covered had been chosen wrongly, and no amount of consistency machinery repairs that. Every copy of live data therefore needs two answers rather than one: which instant, if any, do these bytes correspond to, and which files, volumes and components were inside the capture. The rest of this part is about producing evidence for both.

What to take from this

• Crash-consistent means one instant across the captured set — the state a power cut leaves behind — and journalling filesystems and write-ahead logs are built to start from exactly that. redo starts at 0/17615F8 in the capture is that machinery running as designed.
• Application-consistent means a component was told and responded. PostgreSQL documents the interface: a file-system-level backup of a running cluster is valid only when bracketed by pg_backup_start and pg_backup_stop and accompanied by all WAL generated during the copy.
• A tree-walk copy of live data is weaker than crash-consistent, because its files come from different instants and no instant ever held that combination.
• The measured cp -a copy started, logged database system was not properly shut down; automatic recovery in progress, and returned 45000 rows — the same count as the live database. It proved that this run happened to have the WAL it needed, and nothing whatsoever about transaction consistency.
• The same copy with pg_wal emptied panicked with could not locate a valid checkpoint record at 0/2F20158 and exited 1, after cheerfully creating the missing write-ahead log directories first.
• Scope is a separate question from instant: the Docker 29.7.2 capture restored its volume byte-identical at md5 9eb4e2ad8e08e1dcaaf87ababab964b0 and still lost /etc/app-marker, which lived in the container layer nobody was backing up.

Cross-course references

• PostgreSQL for Production Sysadmins — Part XII (WAL, Checkpoints and Crash Recovery) is the full treatment of the replay machinery this lesson only sketches, and it is what separates believing the naive copy worked from understanding what it was relying on; Part XIII (Backup, Archiving and Point-in-Time Recovery) covers the bracketed interface itself and the archive the bracketing depends on.
• Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) covers the backup modes a hypervisor offers for a running guest, which is this same crash-consistent versus application-consistent choice made one layer down, where a guest agent is the only thing that can turn a block-level snapshot into a state the guest itself declared.
• Linux for Production Sysadmins — Part XIV (Filesystems) explains what a journalling filesystem does and does not restore after an unclean shutdown, which sets the ceiling on what crash consistency buys you before any application-level recovery starts.

Quiz