Fleecing, bandwidth limits and backup impact · Proxmox VE

This course already ships a break/fix scenario called breakfix-vm-slow-during-backup. Until this lesson existed, the course described the symptom without ever teaching the mechanism that causes it or the feature that removes it. That gap is the reason this lesson is here.

The symptom is specific and recognisable: during the backup window, one or more guests become slow — not slow in a general “the host is busy” way, but slow specifically on writes, with in-guest latency that looks like the disk has been replaced with something much worse. The cause is not CPU contention and usually not host disk saturation. It is that the guest write path has been extended to include the backup target.

The coupling, and why it exists

From the previous lesson, the mechanism of snapshot-mode backup:

When a backup for a VM is started, QEMU will install a “copy-before-write” filter in its block layer. This filter ensures that upon new guest writes, old data still needed for the backup is sent to the backup target first.

Read the last four words again. Sent to the backup target first. That is a synchronous dependency: for any block the backup has not yet copied, a guest write to that block cannot complete until the old contents have been written to PBS.

So the effective latency of a guest write during a backup is:

guest write latency  =  local storage latency
                      + (if the block is not yet backed up)
                        network round trip to PBS
                      + PBS chunk write latency

Local NVMe answers in tens of microseconds. A PBS server on spinning disks over a shared 1 GbE link answers in tens of milliseconds. That is three orders of magnitude, applied to a fraction of writes that starts at 100% and falls to 0% as the backup progresses through the disk.

The shape of the symptom tells you it is this and not something else

Backup-window slowness caused by copy-before-write has a distinctive signature, and recognising it saves an hour of looking at the wrong graphs:

Writes are affected far more than reads. Reads never enter the filter.
The effect is worst at the start of the backup and tapers. Early on, almost every write hits a not-yet-copied region.
It scales with how write-heavy the guest is, not how large it is. A 1 TiB mostly-idle file server barely notices. A 40 GiB database under load suffers badly.
Host CPU and host disk utilisation look unremarkable. The guest is waiting on the network, not competing for local resources.

If reads are equally affected and the host disks are saturated, you have a different problem — probably backup read I/O competing with production I/O — and bwlimit or ionice is the lever, not fleecing.

What fleecing does

Fleecing inserts a local buffer into that path. Instead of sending the old data straight to the backup target, QEMU writes it to a fleecing image on local storage, and the backup job reads from the fleecing image at its own pace.

	Without fleecing	With fleecing
Guest write to a not-yet-copied block	Waits for the write to reach PBS	Waits for a write to local fleecing storage
Latency added to the guest	Network + PBS chunk write	Local storage write
What absorbs a slow target	The guest	The fleecing image
Extra space required	None	Up to the size of the disks being backed up, worst case

The trade is explicit: you spend local capacity and a local write to stop the backup target from being in the guest’s critical path. For a cluster whose PBS is off-site, on spinning disks, or shared with other traffic, that is an excellent trade.

Configuration changeenabling fleecing— Two sub-options: enabled, and the storage the fleecing images are allocated on. Fleecing is off by default.

set -euo pipefail

# Per invocation.
vzdump 100 --storage pbs-main --fleecing enabled=1,storage=local-lvm

# As a default for every job on this node, in /etc/vzdump.conf:
#   fleecing: enabled=1,storage=local-lvm
#
# Or on a scheduled job, via the Job Detail > Advanced tab in the GUI, which
# writes the same key into /etc/pve/jobs.cfg.

Choosing the fleecing storage

This is where fleecing goes wrong in practice, because the storage requirements are real and the failure is a full storage rather than a warning.

The documentation recommends local storage with thin provisioning and discard support — LVM-thin, RBD, or ZFS with the sparse option — or file-based storages. And it states the consequence of ignoring that: non-thin-provisioned storage such as plain LVM, or ZFS without sparse, requires the full disk size to be reserved up front.

Fleecing storage	Suitable	Why
LVM-thin (local)	Yes	Thin, supports discard, local latency
ZFS with `sparse` enabled	Yes	Thin; without `sparse` it reserves the full size
Directory storage (qcow2 on local SSD)	Yes	File-based, allocates as it grows
RBD	Yes, with a caveat	Thin and discard-capable, but it puts the fleecing write on the network too, which weakens the point unless the Ceph cluster is much faster than the PBS path
Plain LVM	Works, but reserves full size	A 2 TiB guest reserves 2 TiB of volume group
The PBS storage itself	No	It is the thing you are trying to decouple from
A network share	No	Reintroduces network latency into the guest write path

Size the fleecing storage for the worst case, not the average

The fleecing image holds old data for blocks the guest overwrites before the backup reaches them. In the ordinary case that is a small fraction of the disk and the image stays small.

The worst case is a guest that rewrites its entire disk while the backup of that disk is running — and in that case the fleecing image can grow to the size of the original disk. A backup job covering several guests allocates a fleecing image per disk being backed up.

So the sizing rule is not “a few percent”. It is: the fleecing storage should have headroom for the largest concurrent set of disks in a single backup run, or you should limit that concurrency (see max-workers below). A fleecing storage that fills mid-backup fails the backup, and on non-thin-provisioned storage it can fail the allocation at job start instead, which is at least a louder failure.

Why fleecing does not simply make the backup slower

The natural objection is that this only moves the problem: the backup still has to write everything to PBS eventually, so surely the total time is the same and the guest is affected for just as long?

The difference is what is synchronous.

Without fleecing, the guest and the backup are locked together. A guest write to an uncopied block cannot proceed until PBS has acknowledged the old data. The guest runs at the speed of the slowest link in the chain, and there is no buffer between them.

With fleecing, the guest writes to local storage and returns immediately. The backup job drains the fleecing image and the source disk into PBS asynchronously. The two processes are now producer and consumer with a buffer between them, and the guest’s latency depends only on how fast the buffer accepts writes.

The backup does still take as long as PBS is capable of. That is fine — the point was never to make the backup fast. It was to stop the backup’s speed from being the guest’s speed. Those are different objectives and fleecing addresses only the second one.

The residual cost is honest and worth stating: fleecing adds one local write per copy-before-write event, plus the space. On a host whose local storage is already the bottleneck, fleecing makes things slightly worse rather than better — which is why the diagnosis in the callout above comes before the configuration.

The other levers, and which problem each solves

Fleecing addresses guest write latency. It does nothing about a backup job that saturates a link or a disk. Those need different controls, and applying the wrong one is the usual reason a first attempt does not help.

Lever	Option	Solves	Does not solve
Fleecing	`--fleecing enabled=1,storage=X`	Guest write latency caused by copy-before-write	Link saturation, host disk saturation
Bandwidth cap	`--bwlimit KiB/s` (0 = unlimited)	A backup saturating the network link and starving everything else	Latency added to individual guest writes
I/O priority	`--ionice 0-8` (default 7; requires the BFQ scheduler)	Backup read I/O competing with production I/O on the host disks	Anything on a host not using BFQ
Concurrency	`--performance max-workers=N`	Too many disks being backed up at once, multiplying every effect above	The per-disk cost
PBS entry batching	`--performance pbs-entries-max=N`	Memory pressure from very large file-based archives	VM image backups

Read-only / Safewhat scheduler is actually in use— The value in brackets is the active one. ionice is only meaningful on the sda line here.

# grep . /sys/block/*/queue/scheduler

/sys/block/nvme0n1/queue/scheduler:[none] mq-deadline
/sys/block/sda/queue/scheduler:mq-deadline [bfq] none

Illustrative output

Configuration changea job tuned for a slow, remote PBS— Fleecing for latency, a bandwidth cap sized below the link so production traffic still fits, and limited concurrency so the fleecing storage is not asked for everything at once.

set -euo pipefail

vzdump --all 1 \
--mode snapshot \
--storage pbs-offsite \
--fleecing enabled=1,storage=local-lvm \
--bwlimit 62500 \
--performance max-workers=2 \
--prune-backups keep-daily=7,keep-weekly=4

# --bwlimit is KiB/s. 62500 KiB/s is about 500 Mbit/s, which leaves half of
# a 1 GbE link for everything else. Compute it rather than guessing:
#   LINK_MBIT=1000
#   SHARE=50
#   echo $(( LINK_MBIT * SHARE / 100 * 1000 / 8 ))

Diagnosing it properly

Read-only / Safeevidence to collect while it is happening— Read-only. Run these during the backup window, on the node hosting the affected guest.

set -euo pipefail
VMID=100

# 1. Is a backup actually running against this guest right now?
pvesh get /nodes/localhost/tasks --limit 20 --typefilter vzdump

# 2. Host disk utilisation and await. If %util is near 100 on the guest's
#    storage devices, the problem is local contention, not the target.
command -v iostat >/dev/null && iostat -x 2 5

# 3. Link utilisation toward the PBS server. If this is pegged, bwlimit is
#    the lever, not fleecing.
command -v ifstat >/dev/null && ifstat -i vmbr0 2 5

# 4. Inside the guest: are reads fine and writes terrible? That asymmetry is
#    the copy-before-write signature.
qm guest cmd "$VMID" ping && echo 'agent reachable; run iostat inside the guest'

The decision that follows from the evidence:

Evidence	Cause	Remedy
Guest writes slow, reads fine, host disks idle, link not saturated	Copy-before-write to a slow target	Fleecing
Everything on the host slow, host disks at 100% util	Backup read I/O competing locally	`max-workers`, BFQ + `ionice`, or move the backup window
Link saturated, other services on the same link affected	Backup saturating the network	`bwlimit`
Only the first backup after a reboot is slow	No dirty bitmap, so it is a full read	Expected; see `xiii-pbs-incremental-mechanics`

What happens if you enable fleecing on every job on every node without checking?

It is not free, and there are two ways this bites.

Capacity. Every running job allocates fleecing images for the disks it is currently backing up. On thin storage that is usually fine. On a node whose fleecing storage is plain LVM, the full size of each disk is reserved at allocation, and a nightly job across twenty guests can fail at job start because the volume group cannot satisfy the reservation. That failure is at least visible. The version that is not visible is a thin pool quietly filling across several nodes over a few weeks until one of them tips over — and a full LVM-thin pool takes the guests on it with it, not just the backup.

A host that was never the problem. If the bottleneck was local disk rather than the PBS path, fleecing adds a local write per copy-before-write event and makes the window worse. You will see this as a backup that got slower after a change that was supposed to help.

The disciplined rollout is: enable it on the guests that show the symptom, measure the backup window and the in-guest latency before and after, then widen. /etc/vzdump.conf makes it a node default in one line, which is convenient and is exactly why it gets set before anyone has measured anything.

Knowledge check

Why this matters

Fleecing is a checkbox, and a checkbox is a bad way to learn a mechanism. An operator who knows only that “fleecing helps with slow backups” will enable it on a host whose bottleneck is local disk and make the window worse, or put the fleecing image on the backup storage and change nothing.

The mechanism is the transferable part. Once you can say the sentence — a guest write to an uncopied block waits for the old data to reach the backup target, and fleecing replaces that wait with a local write — you can predict which guests will suffer, which will not, what fleecing will cost, and which of the other four levers to reach for when it is not the answer.

That is also why this lesson exists next to breakfix-vm-slow-during-backup rather than instead of it. The scenario teaches you to recognise the symptom under pressure; this teaches you why the symptom exists.