Dirty bitmaps and why incremental backups are fast · Proxmox VE

Here is a ticket that arrives in every Proxmox estate that runs PBS, usually on a Monday:

The nightly backup of VM 214 normally finishes in four minutes. Last night it took two hours and forty minutes. Nothing changed. Is the backup server broken?

Nothing is broken. The VM was rebooted on Sunday for a kernel update. This lesson is why that sentence is a complete explanation, and why the operator who cannot give it ends up opening a support case, restarting PBS, or — worst outcome — disabling verification to “make the window fit”.

Two optimisations, not one

Every PBS backup is a full backup: each snapshot is independently restorable and there is no chain of increments to replay. What makes the second one cheap is two distinct mechanisms working at different layers.

	Dirty bitmap	Chunk deduplication
Where it lives	In the running QEMU process on the PVE node	In the chunk store on the PBS server
What it saves	Reading the disk	Uploading the data
Scope	One VM, one running QEMU process	The entire datastore, across all guests
Survives a VM restart	No	Yes
Survives losing the previous snapshot	No	Yes, if another snapshot references the chunk
Applies to containers	No	Yes

They are frequently described as one thing — “PBS does incrementals” — and the conflation is precisely what makes the Monday ticket confusing. Losing the bitmap costs you the read; it never costs you the storage.

Deduplication: what the server does

From the technical overview:

For block based backups (like VMs), fixed-sized chunks are used. The content (disk image), is split into chunks of the same length (typically 4 MiB) … Since the image is always split into chunks of the same size, unchanged blocks will result in identical checksums for those chunks, so such chunks do not need to be backed up again.

Fixed-size chunking is what makes this work for disk images. Because every chunk boundary is at a fixed offset, an unchanged region of a disk produces byte-identical chunks with identical checksums every time. The server already has them, so they are not sent.

The client does not even need to guess. The backup protocol describes the negotiation:

If there is a previous Snapshot in the backup group, the client can first download the chunk list of the previous Snapshot. If it detects a chunk that already exists on the server, it can send only the checksum instead of data and checksum.

The dirty bitmap: what the host does

Deduplication saves the upload. It does not save the read. Without a bitmap, backing up a 2 TiB disk means reading 2 TiB from local storage, hashing all of it, and discovering that 99% of it is already on the server. The network is idle and the job still takes hours.

The bitmap removes that work:

As an optimization, VMs in Proxmox VE can make use of “dirty bitmaps”, which can track the changed blocks of an image. Since these bitmaps are also a representation of the image split into chunks, there is a direct relation between the dirty blocks of the image and chunks which need to be uploaded. Thus, only modified chunks of the disk need to be uploaded to a backup.

The alignment in that quote is the elegant part. The bitmap tracks changed blocks; the chunk store addresses fixed 4 MiB chunks; the two granularities map onto each other directly, so a dirty region translates into a specific set of chunks with no ambiguity.

Where the bitmap lives, and why that determines everything

The dirty bitmap is a data structure inside the running QEMU process for that VM. It is created when a backup runs, and from that point QEMU records every subsequent guest write into it.

That single fact generates every rule about bitmaps:

It is in memory. It is not written to the VM configuration, not stored on the disk it describes, and not held on the PBS server.
It belongs to one process. When that QEMU process ends, the bitmap ends with it.
It is per disk. Each backing device gets its own, which is why the task log reports bitmap status per drive rather than per VM.
It must be created by a backup before it can help one. The very first backup of a VM has no bitmap, creates one, and reads everything. The second backup benefits.

So the question “what invalidates a bitmap?” has a mechanical answer rather than a list to memorise: anything that replaces the QEMU process. A shutdown and start, a qm stop followed by qm start, a host reboot, a crash, an HA fence. A reboot issued from inside the guest does not replace the QEMU process, so it does not lose the bitmap — which is a distinction worth knowing, because it explains why some reboots cost you a slow backup and others do not.

There is also a correctness rule underneath: a bitmap is only usable if the backup it corresponds to is still the reference point. If the previous snapshot is gone from the datastore, or a different backup target is used, the bitmap describes changes since a state PBS no longer has, and QEMU discards it rather than producing a wrong incremental. This is the mechanism choosing correctness over speed, and it is the right choice.

Reading the evidence

The vzdump task log states the bitmap status per disk. This is the single most useful thing in this lesson, because it turns the Monday ticket into a thirty-second answer.

Read-only / Safea backup that had a usable bitmap— Illustrative shape, not a verified capture. Read the dirty-bitmap line and the reused line - those two are the whole story.

# cat /var/log/pve/tasks/*/UPID:pve1:00001234:vzdump:*

INFO: Starting Backup of VM 214 (qemu)
INFO: Backup started at 2026-08-11 01:00:03
INFO: status = running
INFO: VM Name: db-primary
INFO: include disk 'scsi0' 'ceph-vm:vm-214-disk-0' 512G
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task
INFO: scsi0: dirty-bitmap status: OK (7.4 GiB of 512.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 7.4 GiB dirty of 512.0 GiB total
INFO: 100% (7.4 GiB of 7.4 GiB) in 3m 51s, read: 32.8 MiB/s, write: 30.1 MiB/s
INFO: backup is sparse: 1.2 GiB (16%) total zero data
INFO: backup was done incrementally, reused 504.6 GiB (98%)
INFO: Finished Backup of VM 214 (00:03:58)

Illustrative output

Read-only / Safethe same VM after a Sunday reboot— Same VM, same data, 40x the wall time. Note that reuse is still 98%: nothing extra was stored, only re-read.

# cat /var/log/pve/tasks/*/UPID:pve1:00005678:vzdump:*

INFO: Starting Backup of VM 214 (qemu)
INFO: include disk 'scsi0' 'ceph-vm:vm-214-disk-0' 512G
INFO: started backup task
INFO: scsi0: dirty-bitmap status: created new
INFO: 100% (512.0 GiB of 512.0 GiB) in 2h 38m 12s, read: 55.2 MiB/s, write: 1.1 MiB/s
INFO: backup was done incrementally, reused 505.9 GiB (98%)
INFO: Finished Backup of VM 214 (02:38:19)

Illustrative output

Compare the two carefully, because the second log contains the answer to the ticket in two places:

dirty-bitmap status: created new — there was no bitmap, so this run read the whole disk and built one for next time.

read: 55.2 MiB/s, write: 1.1 MiB/s — the read rate is fifty times the write rate. The host was reading 512 GiB off local storage and sending almost nothing to PBS, because deduplication was still working perfectly. If the backup server were the problem, the write rate would be the low number and the read rate would be low too, because the reads would be blocked behind it.

reused 505.9 GiB (98%) — no additional storage was consumed on PBS.

Extracting this at fleet scale

Read-only / Safewhich guests lost their bitmap last night— Read-only. Run it the morning after a maintenance window to explain the window's backup times before anyone asks.

set -euo pipefail

NODE=$(hostname -s)

# List last night's vzdump tasks, newest first.
pvesh get "/nodes/$NODE/tasks" --typefilter vzdump --limit 50 --output-format json \
| jq -r '.[] | [.upid, .status, (.starttime|todate)] | @tsv' \
| while IFS=$'\t' read -r upid status started; do
  log=$(pvesh get "/nodes/$NODE/tasks/$upid/log" --output-format json 2>/dev/null \
        | jq -r '.[].t' 2>/dev/null || true)
  vmid=$(printf '%s' "$log" | sed -n 's/.*Starting Backup of VM \([0-9]*\).*/\1/p' | head -1)
  if printf '%s' "$log" | grep -q 'dirty-bitmap status: created new'; then
    printf '%-8s %-6s %s  BITMAP RECREATED (expect a slow run)\n' "$vmid" "$status" "$started"
  else
    printf '%-8s %-6s %s  ok\n' "$vmid" "$status" "$started"
  fi
done

The container equivalent

Containers get no dirty bitmap — there is no QEMU block layer to hold one. Their equivalent is --pbs-change-detection-mode, which changes how the .pxar archive is produced:

Value	Behaviour
`legacy`	Read and encode all files into a single archive, using pxar format version 1. The default
`data`	Read and encode all files, but split data and metadata into separate streams, using pxar format version 2
`metadata`	Split streams and use format version 2 as with `data`, but use the metadata archive of the previous snapshot to detect unchanged files

metadata is the one that produces the container analogue of a bitmap win: files whose metadata is unchanged are not re-read from disk, so a container with a large static dataset stops paying for it on every run.

Configuration changechange detection for a container with a large static dataset— Applies to container backups targeting PBS only. Test on one container first: the first run rebuilds the metadata archive and is not faster.

vzdump 310 --storage pbs-main --pbs-change-detection-mode metadata

What happens if you decide the slow post-reboot backup is unacceptable?

There are three responses, and only two of them are engineering.

Accept it and schedule around it. After a planned maintenance window, expect the next run to be a full read for every guest that was restarted. If the window is monthly, that is one slow night in thirty, and the correct action is to tell people it is coming rather than to change anything.

Reduce what the read costs. The bitmap loss makes you read the disk; how long that takes is a property of your storage. A backup that reads 512 GiB at 55 MiB/s is reading slowly for a reason — contention with production, an undersized link to shared storage, or max-workers set high enough that ten guests are reading at once. Those are fixable and they also improve every other read-heavy operation on the host.

Stagger the restarts. If a maintenance window restarts sixty guests, all sixty lose their bitmaps and all sixty do a full read on the same night. That is a self-inflicted thundering herd. Restarting a third of the fleet per week spreads the cost across three ordinary nights instead of concentrating it into one alarming one.

The response that is not engineering is disabling verification, shortening retention, or moving guests off PBS to make the window fit. All three trade a real recovery guarantee for a cosmetic improvement to a graph, and all three are much harder to reverse than to introduce.

Knowledge check

Why this matters

This lesson exists because of a specific organisational failure mode. An operator who cannot explain a slow backup will eventually make it faster by weakening it — fewer verify jobs, shorter retention, a guest excluded from the run — and each of those changes is invisible in the dashboard and catastrophic in a restore.

The defence is a mechanism you can reason from rather than a list of symptoms. The bitmap saves reads and lives in the QEMU process. Deduplication saves uploads and lives in the chunk store. From those two sentences you can derive what a restart costs, why the reuse percentage stays high anyway, why containers behave differently, and which line of the task log to read first.

That is what separates an operator who closes the ticket from one who escalates it.