Guest quiescing and application consistency inside a VM · Backup & DR

Separating a VM snapshot from a VM backup settled where a copy lives and which failures it survives, and every part of that answer sat on the storage side of the guest boundary. This lesson crosses the boundary. Whatever a hypervisor writes into a snapshot, it produced it by capturing a virtual disk, and a virtual disk contains only what the guest handed down to it. So the question that decides whether the copy is worth anything is not how it is stored. It is what the guest was in the middle of when the capture fired, and whether anything inside the guest was asked first.

The hypervisor captures a block device, not a database

A snapshot taken at the hypervisor is taken at the layer that owns the virtual disk. At the instant it fires it records the blocks the guest’s storage stack has already written. What it cannot contain is everything still above that line: pages the guest kernel has dirtied but not yet written back, filesystem journal entries held in memory, an application’s own buffers, a transaction a client believes is still in flight. None of that had reached the virtual disk, so none of it is in the capture.

For a single virtual disk captured in one atomic operation, that description has a name. It is crash-consistent: the state the media would be in if power were removed at that instant, containing every write that reached the device and no ordering that did not exist there. This is a genuinely useful state rather than a consolation prize, because serious software is built to start from it. A journalling filesystem replays or discards its journal on mount. A database with a write-ahead log replays that log forward from its last checkpoint, reapplying committed transactions and discarding the rest. Starting from a crash-consistent image is a designed code path with a defined outcome.

Two qualifications travel with the phrase, and both matter more inside a VM than they do on bare metal. The first is that recovery is only as capable as the software performing it: a filesystem journal guarantees that metadata is coherent, not that a file caught mid-write contains anything sensible, and a process holding state in memory with no log of its own has nothing to replay and gets nothing back. The second is that one instant covers only what was captured in that one operation. A VM with a data disk and a log disk, captured by two operations, has two instants and therefore no single crash-consistent image of the pair.

Underneath both qualifications is the layering itself. The hypervisor does not know it is looking at a database, a message queue or a mail spool. It sees writes to a block device, and the knowledge that certain files must be captured together, or that a checkpoint must be forced before the capture, exists only inside the guest. Nothing about that is a defect in any product. It is where the information lives.

Three levels of coordination, and what each asks of the guest

Because the necessary knowledge is inside the guest, every mechanism for improving on crash consistency works by asking something in there to participate. There are exactly three levels of participation in common use, and naming them precisely is most of the work, because backup jobs at all three levels finish the same way and print the same kind of success.

Level	What is asked of the guest	What the disk holds afterwards	What is still unproven
No coordination	Nothing at all	Whatever had reached the virtual disk at the capture instant	Which instant each file came from, once more than one disk is involved
Filesystem freeze	A guest agent freezes each mounted filesystem, then thaws it	Dirty pages written back, with writes held still across the capture	Anything the application held in its own memory or expected to bracket
Application quiescing	The application flushes, records its own marker and holds	A set the application’s documented recovery path is designed to accept	That the set was really captured and can be restored, which is a separate exercise

The first level is the default everywhere. Nothing in the guest is told anything; the disk is captured as it stands. The result is the crash-consistent image described above, and the reason this survives as a practice is that it usually appears to work, for the entirely real reason that filesystems and databases are engineered to recover from exactly this.

The second level introduces an agent running inside the guest. The QEMU guest agent is the example this course refers to, because it is what a Proxmox VE host talks to: the agent must be installed inside the guest and the VM must be configured to use it, and its documented protocol includes the filesystem freeze and thaw calls guest-fsfreeze-freeze and guest-fsfreeze-thaw. When the backup path drives those calls, the capture happens between the freeze and the thaw, so the filesystems on the disk are structurally settled at a known point rather than merely at an arbitrary one.

The third level is the only one where the software that owns the data has been told anything. For PostgreSQL the documented interface is explicit: a file-system-level copy of a running cluster is a valid backup only when it is bracketed by pg_backup_start and pg_backup_stop and accompanied by all the write-ahead log generated between them. pg_basebackup performs that handshake for you and can stream the required log along with the copy. Either way the consistency is not something the storage produced. It is a claim the database made, and it exists only if some interface was actually used to ask for it.

The gap that swallows real estates sits between the first two levels, because both of the conditions the second level needs — an agent present in the guest, and the VM configured to use it — can be false without the backup job failing. The job runs, the snapshot exists, the dashboard is green, and the difference between a frozen capture and an uncoordinated one is not visible in the result. Job success does not distinguish them, so the only way to know which level a VM is actually protected at is to look at the guest and at the VM’s configuration.

What a filesystem freeze does inside the guest, and what it cannot reach

The agent is an ordinary process running in the guest, and the request reaches it over a virtio-serial channel supplied by the hypervisor rather than over the guest’s network. That detail matters operationally: freezing does not depend on the guest’s routing, firewall or name resolution being healthy, which is why it keeps working on hosts where nothing else about the guest is reachable.

On receiving the freeze request, the agent asks the guest kernel to freeze each mounted filesystem. The kernel writes back dirty pages belonging to that filesystem, brings it to a settled on-disk state, and then blocks further modification until a thaw arrives. The hypervisor takes its capture inside that window and the thaw releases the filesystem again.

Three limits follow directly from that mechanism. First, a freeze is a filesystem operation: it pushes the page cache down to the device, and it says nothing to any application about flushing its own buffers or writing a marker. A database inside a frozen filesystem has not been consulted, so what lands on the disk is still a state its crash recovery must reconstruct.

Second, writes inside the guest block for the duration. A freeze window is service-impacting by construction, which is why it is kept short, and why an aborted backup that never issues the thaw leaves a guest that appears hung while every write waits.

Third, freezing covers the filesystems the agent can see and freeze. A raw block device handed straight to a guest process, or a mount the agent does not handle, is outside the operation, and the capture of that disk is exactly as coordinated as it would have been with no agent at all.

Crash consistency that starts, recovers and returns 45000 rows

The reason uncoordinated capture survives is not that people have not heard the argument. It is that the failure it produces is usually silent. The PostgreSQL 18.6 capture recorded for this course makes that concrete by copying a live data directory with cp -a while a workload was running — no database involvement of any kind — and then starting the copy.

Data-loss riskan uncoordinated copy of a live data directory, started— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), from the application-consistency capture recorded for this course.

$ pg_ctl -D /work/naive-copy start

  waiting for server to start.... done
server started
>>> exit code: 0

2026-08-28 13:34:37.611 UTC [94] LOG:  database system was interrupted; last known up at 2026-08-28 13:34:36 UTC
2026-08-28 13:34:37.613 UTC [94] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-28 13:34:37.613 UTC [94] LOG:  redo starts at 0/17615F8
2026-08-28 13:34:37.634 UTC [94] LOG:  invalid record length at 0/256A8D8: expected at least 24, got 0
2026-08-28 13:34:37.634 UTC [94] LOG:  redo done at 0/256A8B0 system usage: CPU: user: 0.01 s, system: 0.00 s, elapsed: 0.02 s
2026-08-28 13:34:37.639 UTC [88] LOG:  database system is ready to accept connections

rows readable from the naive copy : 45000
rows in the live database         : 45000

That is the transcript of a VM you power on at the recovery site. The cluster announced that it had been interrupted, ran automatic recovery — the same code path a power cut triggers — and came up ready to accept connections. Then it returned 45000 rows, matching the live database it was copied from exactly.

The measured result is a warning, not a reassurance. Nothing in it established that the copy is a transaction-consistent image of any instant. cp walked the directory tree over several seconds while pages were being rewritten underneath it, so the files in the copy come from a range of moments rather than one, and the capture notes precisely why it nevertheless worked: the copy happened to have every write-ahead log record it needed inside it. A hypervisor snapshot of a single virtual disk is in fact stronger than this, because it really is one instant. What it shares with the copy above is the evidential problem, and that is the part that transfers: the outcome the operator observes is “it started and the numbers look right”, which is a weaker statement than “this is the state the database was in at a known time”, and no amount of repeating the first produces the second.

Two sentences that sound like evidence and are not

“The VM backup ran last night, so the database is covered” and “we take hourly snapshots, so snapshots are enough for this VM” are anti-patterns, and both are comfortable to say because both are usually followed by a recovery that starts. Neither sentence contains a claim about which instant the disks came from, or about whether anything inside the guest was asked to participate.

Replace them with statements that can be checked: which level of coordination this VM is protected at, what evidence exists that the agent is installed and in use, and when a guest recovered from one of these snapshots was last compared against a value recorded before the capture. A practice that usually appears to work, and has no defined failure signal when it does not, is more dangerous than one that fails loudly.

The multi-disk case fails loudly, and that is the lucky version

The same capture supplies the other half of the picture. A very common real arrangement puts a database’s data files on one volume and its write-ahead log on another; in a VM that means two virtual disks. The capture models what happens when only the data volume is present by emptying pg_wal and starting the cluster again.

Data-loss riskthe same directory without its write-ahead log — a data-disk-only capture— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), from the same capture, with pg_wal emptied to model a snapshot that covered only the data volume.

$ pg_ctl -D /work/nowal start

  waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
>>> exit code: 1

2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/archive_status"
2026-08-28 13:34:37.879 UTC [132] LOG:  creating missing WAL directory "pg_wal/summaries"
2026-08-28 13:34:37.879 UTC [132] LOG:  invalid checkpoint record
2026-08-28 13:34:37.879 UTC [132] PANIC:  could not locate a valid checkpoint record at 0/2F20158
2026-08-28 13:34:37.941 UTC [126] LOG:  startup process (PID 132) was terminated by signal 6: Aborted
2026-08-28 13:34:37.941 UTC [126] LOG:  terminating any other active server processes
2026-08-28 13:34:37.942 UTC [126] LOG:  shutting down due to startup process failure
2026-08-28 13:34:37.943 UTC [126] LOG:  database system is shut down

The startup process panicked with could not locate a valid checkpoint record at 0/2F20158 and the command exited 1. Read the two lines above the panic as well: the server cheerfully created the missing write-ahead log directories, which is exactly the sort of detail that persuades a hopeful operator during an incident that the problem is cosmetic. It is not. The data files describe a state that only the redo can complete, and the redo was not in the set that was captured.

Three ordinary VM arrangements produce this shape. A backup job configured years ago excludes one of the guest’s disks and nobody has re-read the configuration since. A second disk was added to a running VM and never added to the job. Or both disks are captured, but by two operations a few seconds apart. The third is the one worth losing sleep over, because it does not panic. Both halves are present, each internally plausible, describing moments that never coexisted — and a cluster in that state can start, accept connections and serve queries while carrying a disagreement that surfaces later as corruption nobody can date. The loud failure above is the version that tells you.

What the application’s own mechanism adds

The constructive answer is in the third part of the same capture, and it is worth being precise about what it improves. A base backup was taken through the database’s own interface, with the write-ahead log streamed alongside it.

Read-only / Safea copy the database participated in— EXECUTION VALIDATED on PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), from the same capture.

$ pg_basebackup -D /work/base -X stream -c fast

  >>> exit code: 0
rows contained in the base backup: 45000

Business then continued: 5,000 more orders arrived, bringing the table to 50000 rows with sum(amount)=825025000 recorded as an independent property of the data, and then somebody ran an unqualified DELETE and the table held 0 rows. Recovering the base backup forward through the archived log to a target time just before the mistake produced this, copied from the capture:

  rows recovered  : 50000   (expected 50000)
  sum(amount)     : 825025000   (expected 825025000)
  RECOVERED - row count and business checksum both match the pre-DELETE state

Two things separate this from the first transcript, and only the second is about consistency. The first is that recovery could be steered to a chosen moment at all, which requires the log the database itself wrote and archived — a capability that exists at the application layer and has no equivalent at the hypervisor layer, where the available moments are the instants somebody happened to snapshot. The second is that the result was accepted on figures recorded before the incident rather than on the fact that the server started. 50000 says the right number of rows came back; 825025000 says the values inside them are the business’s values. A guest that boots proves neither.

Which mechanism, decided per VM

For a VM whose guest holds only files — a web server, a reverse proxy, a document store where a torn file is an acceptable loss — a snapshot with a filesystem freeze is a reasonable protection, and the freeze is what makes the restored mount predictable.

For a VM running a database, a queue or anything else that maintains its own log and its own invariants, the choice is between quiescing that application explicitly around the capture and letting the application take the copy itself. The second is usually simpler to operate and is the only one of the two that gives you a point-in-time capability, because it is the only one that keeps the log. The hypervisor snapshot then protects the machine, and the database backup protects the data, and neither is asked to do the other’s job.

Production discipline

State the coordination level for every VM you protect, in writing. No coordination, filesystem freeze, or application quiescing — and treat “the backup job is green” as evidence for none of the three, since all three produce the same successful job.
Verify the agent is installed in the guest and enabled on the VM, as two separate checks. Either can be false while the backup still completes, and the failed condition changes what the snapshot contains without changing what the job reports.
Never assume a freeze reached the application. A filesystem freeze writes back the page cache and holds writes still; it does not tell PostgreSQL to flush, to bracket the copy with pg_backup_start and pg_backup_stop, or to record where redo must begin.
Treat every VM with more than one virtual disk as a multi-instant capture unless the mechanism documents that the disks are captured together. The measured single-volume case stopped at PANIC: could not locate a valid checkpoint record at 0/2F20158; the version that panics is the one you find out about.
Accept a recovered guest only against a value recorded before the capture. The uncoordinated copy started, cleared crash recovery and returned all 45000 rows; the point-in-time recovery was accepted because rows recovered : 50000 and sum(amount) : 825025000 matched figures written down beforehand.

Cross-course references

Proxmox VE for Production Operators — Part XIII (Proxmox Backup Server) is where the backup path discussed here is actually configured, including whether the guest agent participates, so it is where the abstract question “which of the three levels is this VM protected at?” turns into settings you can read off a specific host.
PostgreSQL for Production Sysadmins — Part XIII (Backup, Archiving and Point-in-Time Recovery) covers in full the application-owned mechanism this lesson recommends for any VM running a database, including the archiving configuration that made the measured recovery to a chosen target time possible at all.
Linux for Production Sysadmins — Part LXXVI (Virtualisation and Linux) describes the guest side of the boundary crossed here, which is what determines whether a freeze request can reach the filesystems in question and what remains outside it.