Filesystem consistency: open files, buffered writes and quiescing · Backup & DR

Crash-consistent and application-consistent, the distinction the previous lesson drew, is a statement about what a copy contains. This lesson is about where the missing part was sitting at the instant the copy was taken. Between the moment an application calls write() and the moment those bytes are durable, the data passes through several staging areas that a block-level snapshot cannot see into and that a file-level walk sees into only by accident. Knowing which layer holds what, when your backup fires, is the whole of filesystem consistency.

Four layers between `write()` and durable storage

A successful write() does not mean the data is on a disk. It means the kernel has copied the bytes into the page cache and marked those pages dirty. The call returns immediately. Writeback happens later — when kernel threads get round to it, when memory pressure forces it, or when something explicitly asks for it. Until then the only copy of that data is in volatile RAM.

Below the page cache sits the filesystem journal. Its job is narrower than most people assume: it records metadata changes (and, in data-journalling modes, the data too) so that after an unclean shutdown the filesystem can be replayed into a structurally valid state. Journal replay guarantees that directories, inodes, extents and free-space maps agree with each other. It guarantees nothing about whether the contents of your files form a coherent application state. A filesystem can be perfectly clean and hold a half-written record.

Below that sits the device write cache. The fsync(2) manual page says that flushing a file to the disk device includes writing through or flushing a disk cache if present, and that the call blocks until the device reports the transfer complete — but the same page records that implementations in older kernels and lesser-used filesystems do not know how to flush disk caches, and that there the caches must be disabled with hdparm(8) or sdparm(8) to guarantee safe operation.

And then there is the block layer, where LVM, ZFS, Btrfs, a SAN array and a cloud volume service all take their snapshots. A snapshot at that layer is taken beneath the filesystem and beneath the page cache. It contains precisely the blocks the device has been handed. Everything still dirty in RAM is absent. Everything the application has buffered in its own memory and not yet written is absent twice over.

That is why an unassisted block snapshot produces a crash-consistent image and nothing better. It is, byte for byte, what a power cut at that instant would have left behind, and whether it is usable depends on what the filesystem journal and the application’s own recovery can make of it.

Three hazards a walk of a live filesystem cannot represent

The temptation, when a snapshot layer is unavailable, is to copy harder: tar, rsync, cp -a over the live tree, perhaps twice, perhaps with a checksum pass. Three hazards defeat all of it.

A file rewritten in place while the walk is reading it. A tree walk reads files one at a time over seconds or hours. Any file modified after the walk started and before the walk reached it is captured in its new state; any file modified while the walk is partway through is captured as a mixture of old and new — a state that never existed on the origin at any instant. The resulting archive records a normal-looking size and mtime for it.

The measured version of this is a plain copy of a running PostgreSQL data directory, taken with no involvement from the database, while a workload ran.

Data-loss riskstarting a naive copy of a live data directory— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2). The copy was taken with cp -a mid-workload, with no database involvement at all.

$ pg_ctl -D /work/naive-copy start

waiting for server to start.... done
server started

>>> exit code: 0

2026-08-28 13:34:37.611 UTC [94] LOG:  database system was interrupted; last known up at 2026-08-28 13:34:36 UTC
2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

It started. It ran crash recovery, exactly as it would after a power cut, and happened to have every WAL record it needed inside the copy. It returned the same 45000 rows as the live database. And none of that established that the copy was a transaction-consistent image of any instant — cp walked the tree over several seconds while pages were being written underneath it. The capture puts the danger plainly: a practice that usually appears to work, and has no defined failure signal when it does not, is more dangerous than one that fails loudly.

Two things people say when the snapshot layer is missing

“Just rsync it” is the first. Because the copy usually starts and the row counts usually match, the practice accumulates years of apparent success and produces no signal on the occasions it does not work. PostgreSQL’s own documentation is blunt: the server must be shut down to get a usable file-system backup, and half-way measures such as disallowing all connections will not work, because tar and similar tools do not take an atomic snapshot of the file system and because of internal buffering within the server.

“The snapshot is enough” is the second, and it fails differently. A snapshot solves the changing-underneath-you problem and solves nothing else. It is not an application-consistent image unless the application was quiesced, and it is not a backup at all: in the LVM capture used below, destroying the physical volume removed the origin and the snapshot together, because the snapshot was never an independent copy.

Files that must agree, captured at different instants. A service’s state is rarely one file. A database’s data files and its write-ahead log must correspond. When those live on separate volumes and only one volume is captured, the parts disagree. This arrangement is common: data on one volume, WAL on another, and a snapshot schedule covering the data volume.

Data-loss riskthe same copy, with pg_wal missing — a separate-volume snapshot— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), from the same capture, with pg_wal emptied to model a data-volume-only snapshot.

$ pg_ctl -D /work/nowal start

waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.

>>> exit code: 1

2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/archive_status"
2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/summaries"
2026-08-28 13:34:37.879 UTC [132] LOG:  invalid checkpoint record
2026-08-28 13:34:37.879 UTC [132] PANIC:  could not locate a valid checkpoint record at 0/2F20158
2026-08-28 13:34:37.941 UTC [126] LOG:  startup process (PID 132) was terminated by signal 6: Aborted
2026-08-28 13:34:37.941 UTC [126] LOG:  terminating any other active server processes
2026-08-28 13:34:37.942 UTC [126] LOG:  shutting down due to startup process failure
2026-08-28 13:34:37.943 UTC [126] LOG:  database system is shut down

This failure is loud, which is the lucky case. The checkpoint record telling the server where redo must begin lived in WAL the copy never contained, so start-up panicked and stopped. The dangerous variant is the one where both volumes are captured, but a few seconds apart — nothing panics, the service starts, and the disagreement between the two halves surfaces later as corruption nobody can date.

Open files with no name. A process can hold an open descriptor on a file that has already been unlinked. The data is still readable through the descriptor and the blocks are still allocated, but the file has no entry in any directory. A tree walk cannot see it at all, because a walk enumerates directory entries. A block-level snapshot does capture the blocks, but it captures them under an inode with a link count of zero that nothing in the snapshot holds open — the exact condition a filesystem’s orphan handling exists to clean up. Either way the data is gone from the copy, and the only symptom beforehand is that df and du disagree about how full the filesystem is. Log files rotated while a daemon still holds the old descriptor are the everyday version.

`fsync`, `sync` and `fsfreeze` close different parts of the gap

None of those hazards is closed by copying more carefully. Three mechanisms narrow the distance between the page cache and the block layer, and they are not interchangeable.

fsync(2) acts on a single file descriptor and is called by the application itself. It is the precise tool: the process that knows which file matters, and in what order relative to others, asks for that one to be made durable. A database’s durability guarantee is built out of fsync calls in a specific sequence, and nothing outside the application can issue them correctly, because nothing outside it knows the ordering.

sync is the blunt system-wide instrument: flush every dirty page belonging to every filesystem. It closes the page-cache gap for data that has already been written, at one instant. What it does not do is stop anything: the moment it returns, writers are dirtying pages again, and a snapshot a quarter of a second later meets a fresh gap. sync before a snapshot is better than nothing and is not a consistency mechanism.

fsfreeze is the one that gives you a window instead of an instant. In the words of its manual page it halts any new access to the filesystem and creates a stable image on disk, and it holds that state until the matching fsfreeze -u releases it.

Service impact possiblefreezing a mount point suspends every writer on it— DOCUMENTATION VERIFIED against fsfreeze(8). No output prop: this course captured no fsfreeze run, so only the command form is shown, quoted from the manual page.

# fsfreeze -f /srv/data

Three properties of fsfreeze matter operationally. Its argument is a mount point, not an arbitrary directory, so it affects every writer on that filesystem and not only the service you care about. It is not universal either: the manual page lists btrfs, ext2/3/4, f2fs, jfs, nilfs2, reiserfs and xfs as the filesystems supporting the freeze feature, and notes the list may be incomplete — so on a filesystem outside that list the freeze step cannot be assumed to exist, and the design has to hold consistency elsewhere. And the same page records that fsfreeze is unnecessary for device-mapper devices, because device-mapper and therefore LVM automatically freeze a filesystem on the device when a snapshot creation is requested. If your snapshot mechanism is LVM, the freeze is already happening; the thing that is still missing is the application.

What a freeze actually accomplishes, and what it cannot

“Creates a stable image on disk” is a stronger promise than halting new access alone would deliver, and the gap between the two is where the mechanism lives. A call that merely turned new writers away and returned would leave the writes already in flight, and the dirty pages already accumulated, to land on the device afterwards — which is precisely the moving target a snapshot needs removed. So the freeze cannot return until the writers already inside have finished, until the filesystem’s dirty pages have been written back, and until its own on-disk bookkeeping agrees with them. From that point until the thaw, this filesystem is not modifying the block device underneath at all.

That is why a snapshot taken between fsfreeze -f and fsfreeze -u is qualitatively different from one taken on a live filesystem. It is not merely a plausible crash image that recovery will probably cope with; it is the image the filesystem declared stable before the copy began.

What the freeze cannot do is know anything about your data: a filesystem’s idea of consistency stops at its own metadata. An application that is three files into a five-file update when the freeze lands gets frozen exactly there. The resulting snapshot is clean at the filesystem level and incoherent at the application level, and no amount of freezing harder will change that. Closing that gap requires the application to be told, in its own language, to reach a point it can be restarted from — which is what a quiescing hook is, whether it is pg_backup_start, a freeze hook fired by a guest agent inside a virtual machine, or a vendor writer framework.

The other thing the freeze cannot do is last. Every process writing to that filesystem blocks for the duration, and a freeze that is never thawed is indistinguishable from a hung disk. The window should be the time it takes to create a snapshot, not the time it takes to copy the data.

Quiesce, snapshot, release, then copy from the snapshot

The three mechanisms above assemble into one sequence, and it is the sequence rather than any single mechanism that closes all three hazards.

Quiesce first. Ask the application, in its own protocol, to reach a state it can be restarted from and to hold there. This is pg_backup_start for PostgreSQL, a read lock plus flush for other engines, a guest-agent freeze hook inside a virtual machine, a writer framework on Windows. The application defines what consistent means; the storage layer cannot.

Snapshot second, with the filesystem frozen — explicitly with fsfreeze, or implicitly by device-mapper if you are on LVM. The snapshot is an instant, not an interval, which is what removes the rewritten-in-place hazard. If the service spans several volumes, every one of them must be captured in the same operation, or you have manufactured the second hazard yourself.

Release third, and treat it as mandatory rather than cleanup: a freeze blocks every writer on the filesystem, so an error between freeze and thaw is an outage.

MNT=/srv/data
SNAP_CMD=/usr/local/sbin/take-volume-snapshot

# Register the thaw before taking the freeze: an error in between must not
# leave the mount point frozen against every writer on the system.
trap 'fsfreeze -u "$MNT"' EXIT

sync
fsfreeze -f "$MNT"
"$SNAP_CMD" "$MNT"
fsfreeze -u "$MNT"
trap - EXIT

Copy from the snapshot, finally — never from the live filesystem. This is the step that converts a brief freeze window into an unhurried copy, however long it takes, because the snapshot does not change while it is read. The LVM capture demonstrates the property the whole pattern rests on: a snapshot taken at 09:00 still served the 09:00 ledger after an operator truncated the file on the origin at 09:30.

Read-only / Safethe snapshot's view is stable while the origin moves on— EXECUTION VALIDATED on LVM 2.03.31(2) (2025-02-27), driver 4.50.0, Debian 13 (trixie). Step 2 of the capture, which prints both files and both md5 sums; the labels are the capture script's.

$ md5sum /mnt/snap/orders.csv

origin  orders.csv now:
ORDER-9999,0.00
snapshot orders.csv still:
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 read from snapshot : 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded at 09:00  : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the snapshot still holds the 09:00 ledger

That stable view is the one thing a snapshot genuinely gives you, and it is what makes a long copy meaningful. What makes it restorable is the quiesce before it.

Production discipline

Involve the application before you involve the storage. A quiescing hook that returns success is evidence about application state; a snapshot with no hook is evidence only about block state. If a service has no quiesce mechanism, record that its backups are crash-consistent rather than pretending otherwise.
Capture every volume a service spans in one operation. Data on one volume and WAL on another, snapshotted separately, produced PANIC: could not locate a valid checkpoint record at 0/2F20158 on PostgreSQL 18.6 — and the silent version, where both are captured seconds apart, is worse.
Guarantee the thaw before you take the freeze. Register the release path first, so that a failure in the snapshot step cannot leave a mount point frozen against every process writing to it.
Copy from the snapshot, never from the live tree. The snapshot does not move under the reader; the live filesystem is what produced the mixed-state file in the first place.
Prove the copy starts, then prove it is right. The naive copy in this lesson started, completed crash recovery and returned 45000 rows. Starting is the weakest possible evidence, and it is the evidence most restore tests stop at.

Cross-course references

Linux for Production Sysadmins — Part XIV (Filesystems) covers journal modes, writeback and mount-time recovery in the depth this lesson treats as a single layer; you need it to reason about what a freeze had actually flushed before the snapshot was taken.
PostgreSQL for Production Sysadmins — Part XII (WAL, Checkpoints and Crash Recovery) explains why the checkpoint record in pg_wal is what makes a data directory startable at all, which is precisely what the missing-WAL capture above could not supply.
Proxmox VE for Production Operators — Part IX (Virtual Machines) covers the guest-agent path that runs freeze and thaw hooks inside a guest, which is the quiescing sequence of this lesson implemented from one layer below the guest filesystem.