LVM snapshots: copy-on-write growth, overflow and invalidation

Objective

Three claims about LVM snapshots are easy to read past: the exception store is consumed by writes to the origin, overflow destroys the snapshot rather than filling it, and origin and snapshot share one failure domain. You will produce all three, on a volume group you build and then destroy.

You will also do the thing a snapshot exists for: restore a file an operator truncated, proven by checksum. The failure modes tell you how long you have to do it in.

Architecture

Everything here lives inside one 1 GiB file. That is not a convenience — it is the shape of the failure the last task demonstrates.

flowchart TD
  IMG["/srv/rbdr-lab06/rbdr-pv.img, a 1 GiB file"] --> LOOP["loop device, PV of PSize 1020.00m"]
  LOOP --> VG["VG rbdrvg, one PV, one failure domain"]
  VG --> ORIG["LV data, 600 MiB origin, attr owi-aos---"]
  VG --> SNAP["LV data_snap, 100 MiB exception store, attr swi-a-s---"]
  ORIG -->|"first write to a chunk copies the OLD contents across"| SNAP
  SNAP -->|"store full, Unable to allocate exception"| INV["attr swi-I-s---, reads fail, mount fails"]
  LOOP -->|"backing store destroyed"| GONE["vgs and lvs both return nothing"]

The arrow into the snapshot is the copy-on-write tax, charged on origin writes for as long as the snapshot exists. The arrow out of the loop device is the fate the snapshot cannot escape.

Requirements

This lab drives device-mapper directly, so state the environment.

• A privileged container, or a disposable VM with root. The captured run used a Debian 13 (trixie) container on a 7.0.0-29-generic host.
• /dev bind-mounted from the host. losetup and lvcreate open device nodes; a container with a private devtmpfs cannot.
• /lib/modules bind-mounted from the host, matching the running kernel.
• modprobe dm-snapshot must succeed before any lvcreate -s. The snapshot target is a separate module from dm_mod, and lvcreate reports its absence in language that never mentions modules.
• An /etc/lvm/lvmlocal.conf that disables udev interaction. Set udev_sync = 0, udev_rules = 0 and verify_udev_operations = 0 under activation, and obtain_device_list_from_udev = 0 under devices. Without them lvcreate waits for a udev daemon the container does not run, times out, and aborts while wiping the new volume — surfacing as not found: device not cleared, which reads like a hardware fault and is not one.
• Roughly 1.5 GiB of free disk and a free loop device.

The volume group is rbdrvg and the logical volumes data and data_snap, matching the capture quoted below; everything outside LVM carries the rbdr- prefix so cleanup can be scoped.

Scenario

At 09:00 a ledger sits on a 600 MiB volume, and a 100 MiB snapshot is taken so a copy can be made without stopping the service. At 09:30 an operator truncates the ledger on the origin, leaving a readable 09:00 view and a live volume that no longer holds the data.

You restore from the snapshot, then find out how long that view would have survived on a busy volume.

Tasks

Task 1 — Record the pre-lab state

set -u
LAB="$HOME/rbdr-lab-06"
mkdir -p "$LAB"

{
  echo "== loop devices =="; losetup -a
  echo "== volume groups =="; vgs --noheadings 2>&1
  echo "== logical volumes =="; lvs --noheadings 2>&1
  echo "== device-mapper =="; dmsetup ls 2>&1
} | tee "$LAB/pre-state.txt"

Cleanup diffs post-state.txt against this file, so capture it first. If the four sections are not empty, this is not a disposable host and Task 8 will destroy something that matters.

Task 2 — Make the container capable of device-mapper

modprobe dm-snapshot
grep -E '^dm_snapshot|^dm_mod' /proc/modules

install -d /etc/lvm
cat > /etc/lvm/lvmlocal.conf <<'EOF'
activation {
    udev_sync = 0
    udev_rules = 0
    verify_udev_operations = 0
}
devices {
    obtain_device_list_from_udev = 0
}
EOF

lvm version | tee "$LAB/env.txt"

$ lvm version

  LVM version:     2.03.31(2) (2025-02-27)
Library version: 1.02.205 (2025-02-27)
Driver version:  4.50.0

Those four settings are not tuning. By default LVM creates the volume, asks udev for its device node, waits for confirmation, then zeroes the start of the volume. With no udev daemon the wait cannot be satisfied, and lvcreate abandons the volume rather than proceed.

Task 3 — Build the PV, the VG and the origin

BACKING=/srv/rbdr-lab06/rbdr-pv.img
install -d /srv/rbdr-lab06
truncate -s 1G "$BACKING"
LOOP=$(losetup --find --show "$BACKING")
printf 'backing store attached as %s\n' "$LOOP" | tee -a "$LAB/env.txt"

pvcreate "$LOOP"
vgcreate rbdrvg "$LOOP"
lvcreate -L 600M -n data rbdrvg
mkfs.ext4 -q /dev/rbdrvg/data

install -d /mnt/rbdr-origin /mnt/snap
mount /dev/rbdrvg/data /mnt/rbdr-origin
dd if=/dev/zero of=/mnt/rbdr-origin/rbdr-churn.bin bs=1M count=200 \
   conv=fsync status=none

printf 'ORDER-1001,4500.00\nORDER-1002,1250.00\n' > /mnt/rbdr-origin/orders.csv
sync
SNAP_TAKEN=$(date +%s)
md5sum /mnt/rbdr-origin/orders.csv | cut -d' ' -f1 > "$LAB/ledger-0900.txt"
cat "$LAB/ledger-0900.txt"

$ md5sum /mnt/rbdr-origin/orders.csv

orders.csv contents:
ORDER-1001,4500.00
ORDER-1002,1250.00
orders.csv md5 at 09:00: 9eb4e2ad8e08e1dcaaf87ababab964b0

rbdr-churn.bin is 200 MiB of pre-written blocks; Task 6 rewrites the first half, which is what charges the exception store.

Task 4 — Take the 100 MiB snapshot

lvcreate -s -L 100M -n data_snap /dev/rbdrvg/data
lvs

$ lvcreate -s -L 100M -n data_snap /dev/rbdrvg/data && lvs

  Logical volume "data_snap" created.
LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   0.01   swi-a-s---

LSize is the exception store, not the view: mounted, data_snap presents a 600 MiB filesystem, because a chunk with no exception is read from the origin. Data% at 0.01 is metadata only. The fifth Attr character, a for active, is the field this lab is about.

Task 5 — The accident, and the restore

mount -o ro /dev/rbdrvg/data_snap /mnt/snap

printf 'ORDER-9999,0.00\n' > /mnt/rbdr-origin/orders.csv
sync
ACCIDENT=$(date +%s)

echo "origin  orders.csv now:"; cat /mnt/rbdr-origin/orders.csv
echo "snapshot orders.csv still:"; cat /mnt/snap/orders.csv
printf '  md5 read from snapshot : %s\n' "$(md5sum /mnt/snap/orders.csv | cut -d' ' -f1)"
printf '  md5 recorded at 09:00  : %s\n' "$(cat "$LAB/ledger-0900.txt")"

$ cat /mnt/snap/orders.csv && md5sum /mnt/snap/orders.csv

origin  orders.csv now:
ORDER-9999,0.00
snapshot orders.csv still:
ORDER-1001,4500.00
ORDER-1002,1250.00
md5 read from snapshot : 9eb4e2ad8e08e1dcaaf87ababab964b0
md5 recorded at 09:00  : 9eb4e2ad8e08e1dcaaf87ababab964b0
MATCH - the snapshot still holds the 09:00 ledger

Now perform the restore, time it, and record the recovery point it cost you.

RESTORE_START=$(date +%s%N)
cp --preserve=timestamps /mnt/snap/orders.csv /mnt/rbdr-origin/orders.csv
sync
RESTORE_END=$(date +%s%N)

md5sum /mnt/rbdr-origin/orders.csv | cut -d' ' -f1 > "$LAB/restored.txt"
{
  printf 'Actual restore time: %s ms\n' "$(( (RESTORE_END - RESTORE_START) / 1000000 ))"
  printf 'Actual RPO observed: %s s of origin changes not recoverable\n' \
    "$(( ACCIDENT - SNAP_TAKEN ))"
} | tee "$LAB/timings.txt"

The RPO figure is the gap between the snapshot and the accident. Anything written to the origin inside that window exists nowhere else, and it is set by how often you take the snapshot, not by anything LVM does.

Task 6 — Copy-on-write growth, measured

umount /mnt/snap

{
  echo "Nothing is written to the snapshot. Only the origin is rewritten."
  for STEP in $(seq 1 10); do
    dd if=/dev/urandom of=/mnt/rbdr-origin/rbdr-churn.bin bs=1M count=10 \
       seek=$(( (STEP - 1) * 10 )) conv=notrunc,fsync status=none
    USED=$(lvs --noheadings -o data_percent rbdrvg/data_snap | tr -d ' ')
    printf '  after rewriting %3d MiB of the origin: snapshot used %s%%\n' \
      "$(( STEP * 10 ))" "$USED"
  done
} | tee "$LAB/cow-growth.txt"

$ lvs --noheadings -o data_percent rbdrvg/data_snap

Nothing is written to the snapshot. Only the origin is rewritten.
after rewriting  10 MiB of the origin: snapshot used 10.12%
after rewriting  20 MiB of the origin: snapshot used 20.17%
after rewriting  30 MiB of the origin: snapshot used 30.22%
after rewriting  40 MiB of the origin: snapshot used 40.27%
after rewriting  50 MiB of the origin: snapshot used 50.32%
after rewriting  60 MiB of the origin: snapshot used 60.37%
after rewriting  70 MiB of the origin: snapshot used 70.42%
after rewriting  80 MiB of the origin: snapshot used 80.47%
after rewriting  90 MiB of the origin: snapshot used 90.52%
after rewriting 100 MiB of the origin: snapshot used 100.00%

Ten MiB of origin rewrite cost 10.12% of a 100 MiB store; the surplus above 10.00% is the store’s own bookkeeping. A store sized exactly to the expected write volume is therefore already short.

Task 7 — Overflow, and the failing case

dd if=/dev/urandom of=/mnt/rbdr-origin/rbdr-churn.bin bs=1M count=100 \
   seek=100 conv=notrunc,fsync status=none

{
  echo "--- lvs now ---"
  lvs
  echo
  echo "--- kernel ring buffer (last lines mentioning snapshot) ---"
  dmesg | grep -i 'snapshot' | tail -1
} | tee "$LAB/invalidation.txt"

$ lvs; dmesg | grep -i snapshot | tail -1

--- lvs now ---
LV        LSize   Origin Data%  Attr      
data      600.00m               owi-aos---
data_snap 100.00m data   100.00 swi-I-s---

--- kernel ring buffer (last lines mentioning snapshot) ---
[739220.717450] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.

Now try to use it. This is the failing case the lab exists for.

{
  echo "--- attempt to mount the snapshot again ---"
  mount -o ro /dev/mapper/rbdrvg-data_snap /mnt/snap
  echo
  echo "Snapshot attribute flags (5th char 'I' = Invalid):"
  lvs --noheadings -o lv_name,lv_attr rbdrvg/data_snap
} 2>&1 | tee -a "$LAB/invalidation.txt"

$ mount -o ro /dev/mapper/rbdrvg-data_snap /mnt/snap

mount: /mnt/snap: can't read superblock on /dev/mapper/rbdrvg-data_snap.
     dmesg(1) may have more information after failed mount system call.
MOUNT FAILED - the snapshot is gone.

Snapshot attribute flags (5th char 'I' = Invalid):
data_snap swi-I-s---

Three independent signals, and only one is a percentage. Data% reads a steady 100.00; a monitor watching that field alone sees a full snapshot, not a dead one. The attribute character and the kernel message are what say the device has stopped answering.

Task 8 — The shared-fate test

umount /mnt/rbdr-origin
pvs

vgchange -an rbdrvg
losetup -d "$LOOP"
truncate -s 0 "$BACKING"
truncate -s 1G "$BACKING"
LOOP=$(losetup --find --show "$BACKING")

{
  printf 'Simulating permanent loss of the underlying device:\n'
  printf '  re-attached the same (now destroyed) backing store as %s\n' "$LOOP"
  echo
  echo "--- what survived? ---"
  echo "vgs:"; vgs
  echo "lvs:"; lvs
} | tee "$LAB/shared-fate.txt"

$ pvs; vgs; lvs

  PV         VG     PSize   
/dev/loop5 rbdrvg 1020.00m

Simulating permanent loss of the underlying device:
re-attached the same (now destroyed) backing store as /dev/loop5

--- what survived? ---
vgs:
lvs:

vgchange -an stands in for the machine losing power along with the disk. Without it the kernel keeps device-mapper tables pointing at storage that is gone — an artefact of tearing a device out from under a running container, not anything LVM teaches.

Validation

Run these in the shell that still holds LAB.

diff -q "$LAB/ledger-0900.txt" "$LAB/restored.txt"
echo "restored digest identical to 09:00 -> exit $?"

grep -c 'snapshot used' "$LAB/cow-growth.txt"
grep -q 'snapshot used 100.00%' "$LAB/cow-growth.txt"
echo "store reached 100.00% -> exit $?"

grep -q 'swi-I-s---' "$LAB/invalidation.txt"
echo "attribute shows I -> exit $?"

grep -q 'Invalidating snapshot: Unable to allocate exception' "$LAB/invalidation.txt"
echo "kernel logged the invalidation -> exit $?"

grep -q "can't read superblock" "$LAB/invalidation.txt"
echo "mount refused the invalidated device -> exit $?"

grep -A2 '^vgs:' "$LAB/shared-fate.txt"

Expected results, exactly:

A non-zero exit on the invalidation checks usually means the origin was not rewritten far enough; add another 100 MiB at seek=200 and read lvs again.

Expected Outcome

The 09:00 ledger was restored from the snapshot, digest 9eb4e2ad8e08e1dcaaf87ababab964b0 on both sides. The exception store filled from 0.01% to 100.00% under 100 MiB of origin rewrite with nothing written to the snapshot; the kernel then invalidated it rather than blocking the origin; and destroying the one physical volume removed both volumes.

Record these from your own run:

• Actual restore time: the value in timings.txt. One small file completes in milliseconds; the number matters because it is the only part of this procedure that scales with the data you lost.
• Actual RPO observed: the value in timings.txt — the gap between SNAP_TAKEN and ACCIDENT. Everything written to the origin in that window was not recoverable from the snapshot.

Troubleshooting

Aborting. Failed to wipe start of new LV., or a message ending not found: device not cleared. LVM asked udev to present and clear the new device node and timed out waiting for a daemon the container does not run. Write /etc/lvm/lvmlocal.conf as in Task 2, lvremove the half-created volume, retry. It names neither udev nor the container.

modprobe: FATAL: Module dm-snapshot not found in directory /lib/modules/... /lib/modules is not bind-mounted from the host, or belongs to a different kernel. Compare uname -r against the directory names under /lib/modules.

device-mapper: reload ioctl ... failed although dm_mod is loaded. The snapshot target itself is missing; dm_mod alone is not enough. Confirm dm_snapshot appears in /proc/modules.

Operation not permitted, or /dev/mapper/control does not exist. The container is not privileged, or /dev is its own rather than the host’s. Both are needed.

losetup: cannot find an unused loop device. /dev is not the host’s, or every loop device is in use; losetup -a lists what is attached.

mount: ... can't read superblock on the snapshot. Invalidation, not corruption — check lvs -o lv_attr for I in the fifth position and dmesg for Unable to allocate exception. Neither fsck nor lvextend helps.

Volume group "rbdrvg" not found after Task 8. Expected: that is the result being measured, not a fault.

Cleanup

set -u
LAB="$HOME/rbdr-lab-06"

umount /mnt/snap 2>/dev/null || true
umount /mnt/rbdr-origin 2>/dev/null || true
vgchange -an rbdrvg 2>/dev/null || true
vgremove -f rbdrvg 2>/dev/null || true

for L in $(losetup -O NAME -n -j /srv/rbdr-lab06/rbdr-pv.img 2>/dev/null); do
  losetup -d "$L" || true
done

rm -f /srv/rbdr-lab06/rbdr-pv.img
rmdir /srv/rbdr-lab06 /mnt/rbdr-origin /mnt/snap 2>/dev/null || true

{
  echo "== loop devices =="; losetup -a
  echo "== volume groups =="; vgs --noheadings 2>&1
  echo "== logical volumes =="; lvs --noheadings 2>&1
  echo "== device-mapper =="; dmsetup ls 2>&1
} > "$LAB/post-state.txt"

diff "$LAB/pre-state.txt" "$LAB/post-state.txt"
echo "environment restored to pre-lab state -> exit $?"

The diff is the point. Exit 0 means the loop, volume group, logical volume and device-mapper inventories match Task 1’s; any output names exactly what the lab left behind.

Production notes

• Size the exception store from the origin’s write volume during the snapshot’s lifetime, plus margin for the bookkeeping the 10.12% figure exposes. The origin’s capacity is only a ceiling.
• Alert on data_percent and the fifth lv_attr character together. A monitor reading only the percentage reports a calm 100.00 on a volume dead for an hour.
• Alert on the copy’s byte and object counts against the previous run too. A job reading a snapshot invalidated mid-stream can end with a partial archive of plausible size.
• Create, copy, remove — from a trap, so an abandoned snapshot cannot go on taxing origin writes after the job has failed.
• A snapshot is not a backup: Task 8 removed both with one device. It becomes part of one only once its contents reach a repository with an independent failure domain.

What You Learned

• The exception store is charged on origin writes. Nothing in Task 6 wrote to data_snap, and it still went from 0.01% to 100.00%.
• Overflow destroys rather than saturates. The kernel chose the live service, logged Invalidating snapshot: Unable to allocate exception., and the origin kept serving writes as owi-aos--- throughout.
• Only one of the three signals is the fill level. Data% looks identical at 100% full and 100% dead.
• can't read superblock on a snapshot is not corruption. The filesystem is intact and the device below it stopped answering.
• A restore has two numbers: the time it took, and the recovery point set by how long ago the snapshot was taken.
• One volume group on one physical volume is one failure domain. vgs and lvs returned nothing, because the snapshot was never an independent copy.