LVM snapshots: sizing, growth and invalidation

Copy-on-write, as the previous lesson described it, explains where a snapshot’s contents come from. It does not say how much room those contents need, and LVM is the implementation where that question has a hard answer with a hard consequence. A classic LVM snapshot is a logical volume whose size you fix at creation, and the copy-on-write exceptions it accumulates have nowhere else to go. When it runs out of room the kernel does not stall the origin and does not pause your copy: it throws the snapshot away. The sizing rule, the alert and the removal discipline in this lesson all follow from that one sentence.

A 100 MiB snapshot of a 600 MiB origin, and what it bought

The capture behind this lesson runs on a disposable loop device: a single physical volume, a volume group named rbdrvg, and a 600 MiB origin logical volume named data carrying a small business ledger. The snapshot taken against it is 100 MiB — one sixth of the origin, which is already a hint that the snapshot’s size is not a function of the origin’s size.

$ lvcreate -s -L 100M -n data_snap /dev/rbdrvg/data && lvs

  Logical volume "data_snap" created.
LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   0.01   swi-a-s---

Four things in that table are worth reading precisely, because every operational decision later in the lesson is made from them. LSize for data_snap is 100.00m: that is the size of the exception store, not the size of the view it presents. Mounted, the snapshot looks like a 600 MiB filesystem, because reads that have no exception are served straight from the origin. Origin names the volume this snapshot tracks. Data% is 0.01 immediately after creation — the store is essentially empty, holding only its own metadata. And Attr reads swi-a-s---, where the leading s marks a snapshot and the fifth character is the state field, a for active.

That fifth character is the one to memorise. It is the difference between a snapshot you can still read and a snapshot that exists only as a row in lvs.

What the snapshot bought is a stable view. Half an hour after it was taken, an operator truncated the ledger on the origin — the ordinary destructive mistake, not a hardware fault — and the snapshot went on presenting the 09:00 contents.

$ md5sum /mnt/snap/orders.csv

  md5 read from snapshot : 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded at 09:00  : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the snapshot still holds the 09:00 ledger

Identical digests. This is the entire value proposition of a snapshot and it is genuinely valuable: a consistent, readable, unmoving surface to copy from while the production volume carries on serving writes. Everything that follows is about the price of holding that surface open, and about how the surface can disappear from underneath a copy that is still running.

Writes to the origin, not writes to the snapshot, consume the snapshot

The commonest sizing mistake is to reason about the snapshot as though it were a destination. Nothing in the capture writes to data_snap at all. The origin alone is rewritten, 10 MiB at a time, and the snapshot’s data_percent is read after each increment.

$ lvs -o lv_name,data_percent rbdrvg/data_snap

Nothing is written to the snapshot. Only the origin is rewritten.
after rewriting  10 MiB of the origin: snapshot used 10.12%
after rewriting  20 MiB of the origin: snapshot used 20.17%
after rewriting  30 MiB of the origin: snapshot used 30.22%
after rewriting  40 MiB of the origin: snapshot used 40.27%
after rewriting  50 MiB of the origin: snapshot used 50.32%
after rewriting  60 MiB of the origin: snapshot used 60.37%
after rewriting  70 MiB of the origin: snapshot used 70.42%
after rewriting  80 MiB of the origin: snapshot used 80.47%
after rewriting  90 MiB of the origin: snapshot used 90.52%
after rewriting 100 MiB of the origin: snapshot used 100.00%

The progression is close to linear and slightly worse than one for one: 10 MiB of origin rewrite consumed 10.12% of a 100 MiB store, 50 MiB consumed 50.32%, and the tenth increment took it to 100.00%. The surplus above 10.00% is the store’s own bookkeeping, and it matters for sizing because it means a store exactly the size of the expected write volume is already slightly too small.

Two refinements make this usable in production. First, the cost is charged on the first change to a region, not on every change: a chunk that a workload rewrites forty times is copied into the exception store once, so a busy origin that churns the same hot set repeatedly is far cheaper than the arithmetic above suggests. The capture rewrote fresh regions each time, which is the worst case and therefore the honest one to size against. Second, the origin’s own capacity is a hard ceiling: since each chunk can be copied at most once, a snapshot as large as the origin plus a small metadata allowance cannot be made to overflow by origin writes at all. Between those two bounds sits every real sizing decision.

Overflow is destruction, not saturation

Here is where LVM parts company with the other snapshot technologies in this part. The capture kept rewriting the origin past the point where the store was full, and the outcome was not a full snapshot. It was no snapshot.

$ lvs; dmesg | grep -i snapshot

  LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   100.00 swi-I-s---

--- kernel ring buffer (last lines mentioning snapshot) ---
[739220.717450] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.

The state character has moved from a to I. The volume is still listed, still 100.00m, still shows an origin and a Data% — every field a monitoring script that only watches fill level would look at is present and unremarkable. What changed is a single letter, and it means the device is no longer serving the data it was created to serve.

$ mount -o ro /dev/mapper/rbdrvg-data_snap /mnt/snap

mount: /mnt/snap: can't read superblock on /dev/mapper/rbdrvg-data_snap.
     dmesg(1) may have more information after failed mount system call.
MOUNT FAILED - the snapshot is gone.

Read that failure carefully, because the wording invites a wrong diagnosis. A superblock that cannot be read looks like filesystem corruption, and the instinct of a tired operator at 03:00 is to reach for fsck. There is nothing for fsck to repair. The block device beneath the filesystem has stopped answering; the filesystem was never damaged. Running a repair tool against an invalidated snapshot wastes the only time in which the origin might still have held the data you wanted.

Note also what did not happen. The origin stayed owi-aos--- and kept accepting writes throughout. No application saw an error, no service degraded, no page fired from the workload’s own instrumentation. The failure is entirely invisible from the production side, and visible only to whatever was reading the snapshot — which, at 02:00 on a backup night, is a job nobody is watching.

Sizing: write volume during the snapshot’s life, plus margin

The rule that falls out of the mechanism is short, and it is not the rule most people apply. Size the snapshot for the writes you expect the origin to receive while the snapshot exists, add margin, and stop thinking about the origin’s total size except as a ceiling.

Four inputs go into that number. The first is the origin’s write rate under the workload that will actually be running — not the daily average, but the rate during the window the snapshot will be held, which for a nightly copy may coincide with batch jobs, log rotation, index maintenance or a vacuum. The second is how long the snapshot will live, which is the copy duration plus whatever slack the copy job’s scheduling introduces. The third is chunk amplification: writes are charged at chunk granularity, so a workload that scatters small writes across many untouched chunks costs far more store than its byte count implies, and the chunk size is fixed at creation by lvcreate --chunksize. The fourth is margin, which exists because the first three are estimates and the penalty for underestimating is total.

Worked through with stated assumptions: an origin of 500 GiB whose busiest hour is measured at 8 GiB of writes, with a copy expected to take three hours, faces roughly 24 GiB of worst-case first-touch writes. The capture’s 10.12% per 10.00% tells you 24 GiB of store is already marginal, so the working number is comfortably above it — 40 GiB gives room for a slow night and a scattered write pattern, and still costs 8% of the origin. If the volume group has the free extents and the workload is genuinely unpredictable, a store the size of the origin removes the failure mode entirely, at the cost of reserving that space.

Every figure in that paragraph came from a measured write rate and a stated copy duration. Change the workload or lengthen the copy and the number changes; none of it is a property of LVM.

data_percent is an alert, not a post-mortem field

The capture went from 0.01 to 100.00 under a straightforward sequential rewrite. Nothing in that progression is slow, and nothing about it produces a signal unless someone is sampling the value. lvs will report it on demand; the operational question is whether anything asks.

Treat data_percent as a first-class metric on the same footing as filesystem utilisation, scraped on the interval you would scrape a disk. Two thresholds are worth carrying: a warning that gives an operator time to react, and a second, higher one that is allowed to take action on its own — shortening the copy, or abandoning it and removing the snapshot before it is invalidated, which at least leaves you with a clean failure instead of a silent one.

set -euo pipefail

VG=rbdrvg
SNAP=data_snap
WARN=70

state=$(lvs --noheadings -o lv_attr "$VG/$SNAP" | tr -d ' ')
used=$(lvs --noheadings -o data_percent "$VG/$SNAP" | tr -d ' ')

if [ "${state:4:1}" = "I" ]; then
  printf 'CRITICAL %s/%s invalidated (attr %s) - the copy is void\n' \
    "$VG" "$SNAP" "$state"
  exit 2
fi

if [ "${used%%.*}" -ge "$WARN" ]; then
  printf 'WARNING %s/%s at %s%% of its exception store\n' "$VG" "$SNAP" "$used"
  exit 1
fi

The state check comes first deliberately. Fill percentage answers “is this heading for trouble”; the fifth attribute character answers “has the trouble already happened”, and a monitor that only watches the percentage will report a comfortable 100.00 on a volume that has been dead for an hour. Both belong in the same check.

LVM can also grow a snapshot without an operator, under a condition worth checking rather than assuming. lvcreate --monitor y places the volume under dmeventd, which lvcreate(8) describes as performing automated maintenance for an LV in response to kernel events; the autoextend thresholds that decide what that maintenance actually does are configuration, so read them on the host you are running before crediting the feature. Even where it is armed, extension can only take extents that are free in the volume group, which means it fails in exactly the situation where you needed it most — a volume group already provisioned tight — and it moves the failure from a snapshot you sized wrongly to a volume group you sized wrongly.

Snapshot, copy, remove

The performance cost of an LVM snapshot is not a fixed overhead; it is a tax on origin writes, and it is levied for exactly as long as the snapshot exists. Every first write to a chunk turns into a read of the origin, a write into the exception store and a metadata commit before the original write is allowed to land. On a read-mostly volume that is close to free. On a write-heavy origin it is a sustained increase in write latency, and it scales with the number of snapshots held against the same origin, because each one needs its own preserved copy of the chunk.

Both problems — the tax and the invalidation risk — shrink with the same lever: the snapshot’s lifetime. The correct pattern is therefore mechanical. Create the snapshot, mount it read-only, copy from it into a repository that lives somewhere else entirely, and remove it as soon as the copy has been verified. A snapshot that exists for eleven minutes cannot be filled by a workload that would have needed forty.

set -euo pipefail

VG=rbdrvg
ORIGIN=data
SNAP=data_snap
MOUNT=/mnt/snap
SIZE=40G

cleanup() {
  umount "$MOUNT" 2>/dev/null || true
  lvremove -f "$VG/$SNAP" 2>/dev/null || true
}
trap cleanup EXIT

lvcreate -s -L "$SIZE" -n "$SNAP" "/dev/$VG/$ORIGIN"
mkdir -p "$MOUNT"
mount -o ro "/dev/$VG/$SNAP" "$MOUNT"

restic backup "$MOUNT" --tag nightly

The trap is the part that earns its place. Backup scripts fail in the middle, and an abandoned snapshot is worse than a missing one: it goes on taxing the origin’s writes, goes on consuming its exception store, and eventually either invalidates itself or exhausts the volume group’s free extents while nobody is looking. Removing it on every exit path — success, failure, interruption — turns a forgotten snapshot from a slow-motion outage into a failed job.

Production discipline

1. Size the store from the origin’s write volume, never from its capacity. The capture’s 100 MiB snapshot survived a 600 MiB origin comfortably until 100 MiB of writes landed; the origin’s size never entered into it. Write the assumed write rate and the assumed copy duration into the runbook beside the number, so the next person can tell when the number stopped being right.
2. Add margin above the estimate, because the store bills at more than 1:1. Ten MiB of origin rewrite cost 10.12% of a 100 MiB store, so a snapshot sized exactly to the expected write volume is already short before chunk amplification is considered.
3. Alert on data_percent and on the fifth attribute character together. The fill level warns you; swi-I-s--- tells you it is over. A monitor that reads only the percentage sees a steady 100.00 and reports nothing wrong.
4. Treat can't read superblock on a snapshot as invalidation until proven otherwise. The filesystem is intact and fsck has nothing to repair — the device below it stopped answering. Check dmesg for Invalidating snapshot: Unable to allocate exception. before touching a repair tool.
5. Remove the snapshot the moment the copy is verified, from a trap. The copy-on-write tax and the invalidation risk both scale with lifetime, and an abandoned snapshot damages the origin’s write performance and the volume group’s free space at the same time.

Cross-course references

• Linux for Production Sysadmins — Part XVI (LVM) is where the volume group, physical volume and extent model this lesson depends on is built from first principles, including the free-extent accounting that decides whether a 40 GiB snapshot can be created at all and whether autoextension has anywhere to grow into.
• Observability for Production Sysadmins — Part XVIII (Alerting Rules) supplies the machinery for turning data_percent and the lv_attr state character into the two-threshold alert described above; this lesson tells you which values to export, and that part covers how to express them as rules that fire before the snapshot is invalidated rather than after.
• Proxmox VE for Production Operators — Part V (Storage Fundamentals) covers the LVM and LVM-thin storage types a hypervisor offers for guest disks, where the same exception-store arithmetic governs whether a snapshot-backed guest backup completes, and where the origin’s write rate is the sum of every VM sharing the volume group.

Quiz