The performance cost of snapshots and clone depth

Why this matters in production

Snapshots and clones are described as free, and their creation is. Their ongoing cost is real, appears as latency rather than capacity, and is attributed to everything except the snapshot structure — which makes it one of the harder performance problems to diagnose without knowing to look.

The snapshot write cost

The first write to any object after a snapshot triggers a copy-on-write: the OSD clones the object before modifying it.

without a snapshot:   write 4 KiB  → modify the object
with a snapshot:      write 4 KiB  → clone the whole 4 MiB object, then modify

The clone is object-sized, not write-sized. A 4 KiB write to a freshly-snapshotted object moves 4 MiB on the OSD.

Consequences:

• A burst of write latency immediately after taking a snapshot
• Proportional to how much of the image is touched, not to image size
• Transient: once an object has been cloned, subsequent writes are normal
• Worse with larger objects, since the clone unit is the object

Multiple snapshots do not multiply this — an object is cloned once per snapshot it must be preserved for, and a write after several snapshots clones against the most recent.

The clone read cost

A read to an unwritten region traverses the parent chain:

object-map removes the futile lookup at each level but not the traversal to the parent that actually holds the data.

Measuring

# write latency before and after a snapshot
fio --name=w --ioengine=rbd --pool=rbd-vms --rbdname=test \
    --rw=randwrite --bs=4k --iodepth=16 --runtime=60 --time_based

rbd snap create rbd-vms/test@perf
# re-run immediately — expect elevated latency for the first pass

# read latency on a clone versus a flattened image
rbd clone rbd-vms/test@perf rbd-vms/test-clone
fio --name=r --ioengine=rbd --pool=rbd-vms --rbdname=test-clone \
    --rw=randread --bs=4k --iodepth=16 --runtime=60 --time_based
rbd flatten rbd-vms/test-clone
# re-run and compare

Policy limits

• Snapshot count per image: bounded by retention, typically tens
• Clone depth: one level; flatten rather than chaining
• Flatten age: flatten clones that outlive a defined period
• Snapshot timing: avoid snapshotting immediately before a write-heavy window, since the copy-on-write burst lands on top of it

Quiz

Production discipline

Schedule snapshots after write-heavy windows rather than before, so the copy-on-write cost falls where latency matters least. Set policy limits on snapshot count and clone depth and enforce them in automation; both costs are bounded only by what the policy allows.

Cross-course references

• Kubernetes: snapshot-induced write amplification affects PVC-backed workloads identically
• Linux: LVM and ZFS copy-on-write snapshots have the same first-write cost profile