Backup modes: snapshot, suspend and stop · Proxmox VE

--mode is one word in a backup job definition and it is the setting that determines what your backup is actually a backup of. Get it wrong and the job still succeeds, the notification is still green, the snapshot still appears in the datastore, and the restored database still refuses to start.

Almost every cluster runs every guest in snapshot mode because it is the default and because it does not interrupt anything. That is usually correct. The lesson is about knowing why it is correct, and recognising the workloads where it is not.

The three modes, for virtual machines

Mode	What happens to the guest	Downtime	Documented as
`stop`	Orderly shutdown, then a background QEMU process backs up the data. The guest is started again once the backup process has taken over.	Shutdown + start, not the backup duration	“provides the highest consistency of the backup, at the cost of a short downtime in the VM operation”
`suspend`	The VM is suspended, then snapshot mode is invoked	Longer than snapshot, without a matching consistency gain	“provided for compatibility reason, and suspends the VM before calling the snapshot mode”
`snapshot`	Nothing. Data blocks are copied while the VM runs	None	“provides the lowest operation downtime, at the cost of a small inconsistency risk”

Stop mode does not stop the VM for the length of the backup

This is the single most common misconception in this lesson, and it changes whether stop mode is usable at all.

Stop mode shuts the guest down, starts a background QEMU process to read the disks, and the guest resumes. The outage is one clean shutdown and one boot — typically a minute or two — not the forty minutes the backup takes.

Which means “we cannot use stop mode, the backup runs for an hour” is a sentence based on a false premise. The real objection to stop mode is that the guest reboots, and for something with a long start-up or a clustered dependency that is a genuine objection. For a small appliance VM with a database that does not survive crash consistency, a two-minute nightly restart may be a much better trade than the alternative.

The three consistency levels

The modes are a means. The thing you are choosing between is consistency, and there are three levels, not two.

Level	What it means	What produces it
Crash consistent	The image is what the disk would look like if you pulled the power at that instant	`snapshot` mode with no guest agent
Filesystem consistent	The guest filesystem journal is quiesced and no writes are in flight at the freeze point	`snapshot` mode + guest agent `fs-freeze`
Application consistent	The application has flushed its own state and knows a backup point exists	Guest agent + application participation (VSS on Windows, `fs-freeze` hooks on Linux), or `stop` mode

Crash consistency is not nothing. Any modern journalling filesystem — ext4, XFS, NTFS, ZFS — recovers from a crash-consistent image. A restored guest will boot, replay its journal, and come up. The Linux course works through the same distinction from the guest side in application-consistent versus crash-consistent backups, which is worth reading alongside this if you are the person who has to convince an application owner that an image is not a database backup.

What crash consistency does not give you is the application’s invariants. A database that has written half of a multi-page update, an application that keeps an index file and a data file that must agree, a message queue mid-ack: these are consistent on disk in the filesystem sense and incoherent in the application sense. The filesystem replays cleanly and the application refuses to start, or worse, starts and serves wrong answers.

What snapshot mode is doing while the guest keeps running

There is no storage snapshot involved, which surprises people who assume the mode name refers to a ZFS or LVM snapshot. Proxmox VE uses a QEMU-internal mechanism that works on any storage backend, including ones with no snapshot support at all.

From the documentation:

When a backup for a VM is started, QEMU will install a “copy-before-write” filter in its block layer. This filter ensures that upon new guest writes, old data still needed for the backup is sent to the backup target first.

So the sequence for any given block is: the backup job is reading the disk linearly; the guest issues a write to a block the job has not reached yet; QEMU intercepts that write, sends the old contents of the block to the backup target first, and only then lets the guest write proceed.

The result is an image that represents a single instant — the moment the backup started — regardless of how long the copy takes or how much the guest writes meanwhile. That is exactly the semantics of a snapshot, which is where the mode gets its name.

The cost is in that word “first”. The guest write waits for the backup target to accept the old data. If the target is slow, guest writes are slow. That consequence is large enough to have its own lesson: see xiii-pbs-fleecing-and-throughput.

Making snapshot mode filesystem consistent

The agent is the whole difference, and it has two halves that people conflate: the agent option on the VM (does PVE talk to the guest at all) and the qemu-guest-agent package inside the guest (is there anything listening).

Configuration changeenable the agent on both sides— Both halves are required. Enabling the option without the in-guest package produces a VM that hangs on shutdown and backs up crash-consistently while appearing configured.

set -euo pipefail
VMID=100

# Host side: enable the channel. freeze-fs defaults to 1, so the freeze is on
# unless somebody has turned it off - which is worth checking rather than
# assuming, because a disabled freeze is invisible in the job report.
qm set "$VMID" --agent enabled=1,freeze-fs=1

# Guest side, Debian/Ubuntu:
#   apt install qemu-guest-agent && systemctl enable --now qemu-guest-agent
# Guest side, RHEL family:
#   dnf install qemu-guest-agent && systemctl enable --now qemu-guest-agent
# Windows: install the guest agent from the virtio-win ISO.

# The channel only appears after a full stop/start, not a reboot from inside.
qm stop "$VMID" && qm start "$VMID"

With both halves present, a snapshot-mode backup issues guest-fsfreeze-freeze before starting and guest-fsfreeze-thaw afterwards. Inside the guest the kernel flushes and quiesces every frozen filesystem, so the image is taken at a point where nothing is in flight.

Read-only / Safeverify the agent is actually answering, before you rely on it— A response means the channel works. An error means the backup you took last night was crash consistent, whatever the job report said.

# qm guest cmd 100 get-fsinfo

[
{
  "name": "sda1",
  "mountpoint": "/",
  "type": "ext4",
  "used-bytes": 4831838208,
  "total-bytes": 20961034240
}
]

Illustrative output

Read-only / Safefleet sweep - which guests claim an agent and which actually answer— Read-only. Run it weekly and treat any row in the mismatch column as a defect, not as a warning.

set -euo pipefail

for vmid in $(qm list | awk 'NR>1 {print $1}'); do
configured=$(qm config "$vmid" | awk -F'[ ,=]' '/^agent:/ {print $2}')
[ "$configured" = "1" ] || { printf '%-6s agent-not-configured\n' "$vmid"; continue; }

status=$(qm status "$vmid" | awk '{print $2}')
[ "$status" = "running" ] || { printf '%-6s not-running-skip\n' "$vmid"; continue; }

if qm guest cmd "$vmid" ping >/dev/null 2>&1; then
  printf '%-6s ok\n' "$vmid"
else
  printf '%-6s CONFIGURED-BUT-NOT-ANSWERING\n' "$vmid"
fi
done

Container modes are a different mechanism with the same names

The names are shared; the implementations are not, and the differences matter operationally.

Mode	Container behaviour
`stop`	The container is halted for the whole backup. Potentially long unavailability — unlike VM stop mode, this really is the full duration
`suspend`	rsync copies the container to a temporary location, the container is suspended, a second rsync pass copies what changed, then it resumes. Short downtime, but it needs extra storage for the temporary copy
`snapshot`	The container is suspended, a temporary snapshot of its volumes is taken, the snapshot content is archived, and the temporary snapshot is deleted. Requires a storage backend that supports snapshots

Two consequences that are easy to miss:

Container stop mode is not the cheap operation VM stop mode is. There is no background process taking over. Budget the whole backup duration as downtime.

Container snapshot mode requires snapshot-capable storage. A container on a plain directory storage cannot use it, and the job will fall back or fail. That is a storage design constraint arriving through the backup configuration, which is why it usually surprises people.

Choosing per workload

Configuration changemode is a per-job setting, so split the jobs— One nightly job for the fleet, one small job for the guests that need a stronger guarantee. This is the shape almost every mature cluster converges on.

set -euo pipefail

# The fleet: live, no downtime, agent-quiesced.
vzdump --all 1 --exclude 205,206 \
--mode snapshot \
--storage pbs-main \
--prune-backups keep-daily=14,keep-weekly=8,keep-monthly=12 \
--notes-template '{{guestname}} nightly'

# The exceptions: two guests whose application cannot be reconstructed from a
# crash-consistent image, and which tolerate a two-minute restart at 03:00.
vzdump 205 206 \
--mode stop \
--storage pbs-main \
--prune-backups keep-daily=14

Workload	Mode	Why
Stateless web/app servers	`snapshot`	Crash consistency is sufficient; the state is elsewhere
Linux guest with a journalling FS and no database	`snapshot` + agent	Filesystem consistency is the whole requirement
PostgreSQL / MySQL / MSSQL	`snapshot` + agent, plus a native dump, or `stop`	The image restores; the database engine is what decides whether it starts cleanly. A native backup inside the guest is the application-consistent artefact
Windows with VSS-aware applications	`snapshot` + guest agent	The agent drives VSS, which is what makes the copy application consistent
Active Directory domain controller	`snapshot` + agent, and read the `vmgenid` material first	Restoring a DC from an image has failure modes beyond consistency
Appliance VM with an opaque embedded database	`stop`	You cannot reason about its consistency, so remove the question

What happens if you back up a busy database in snapshot mode with no agent?

Most of the time: nothing visible. You restore it in a test, it starts, it replays its write-ahead log, and everyone concludes the backup is fine. That is the dangerous part — crash consistency succeeds often enough to build confidence.

The cases where it does not:

Multi-file invariants. A database whose data files and log files live on separate virtual disks. The copy-before-write filter gives each disk a consistent instant, but there is no guarantee the two disks represent the same instant relative to the guest. Recovery can find a log that references pages the data file does not have.
Storage that lies. A guest with writeback caching in front of it, or a filesystem mounted with barriers disabled, has already discarded the ordering guarantees the database was relying on. The crash-consistent image is consistent with a state the database never intended to be recoverable from.
Applications that are not databases. A message broker that has acknowledged a message and not yet persisted it. A file-based index that is rebuilt from a data file. These fail in ways that pass every filesystem check.

The correct posture is not “snapshot mode is unsafe for databases”. It is that a VM image is a disaster-recovery artefact for a database, and a native dump or log-shipping arrangement inside the guest is the operational recovery artefact. Most estates need both, and the mistake is believing the first one replaces the second.

The three modes, for virtual machines

The three consistency levels

Making snapshot mode filesystem consistent

Container modes are a different mechanism with the same names

Choosing per workload

Knowledge check