LVM thin provisioning and thin-pool monitoring

A normal logical volume reserves its extents when you create it. A 200 GB LV consumes 200 GB of the volume group immediately, whether or not anything is written to it.

A thin logical volume does not. It is carved out of a thin pool and consumes pool space only as blocks are written. That lets you create ten 200 GB volumes on a 500 GB pool — useful for virtual machine images, container storage, and per-tenant volumes where most of the allocation is never touched.

It also means the guarantee is gone. The sum of the volumes exceeds the pool, and if the writes ever arrive, the pool runs out.

Creating a thin pool

$ sudo lvcreate --type thin-pool -L 500G --poolmetadatasize 1G -n tpool vg0

  Logical volume tpool created.

$ sudo lvcreate --type thin -V 200G --thinpool vg0/tpool -n vm-web01

  Logical volume vm-web01 created.

The two percentages

lvs reports both, and they fail in different ways.

$ sudo lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent --select 'lv_layout=thin,pool'

  LV     VG   LSize   Data%  Meta%
tpool  vg0  500.00g 82.31  41.07

Data exhaustion (Data% reaches 100). Writes to any thin volume in the pool fail. The filesystems on top see I/O errors and typically remount read-only; ext4 may go further and flag the filesystem for fsck. It is ugly, and it hits every tenant of the pool at once, but it is recoverable: extend the pool, and the volumes come back.

Metadata exhaustion (Meta% reaches 100). The pool can no longer record where blocks live. Adding data space does not help, because data space is not what ran out. The pool is deactivated and needs lvconvert --repair, which needs a spare volume of at least the metadata size and does not always succeed. This is the one that loses pools.

What snapshots do to metadata

Thin snapshots are the usual reason metadata runs out ahead of data. Every snapshot adds mappings, and a host taking hourly snapshots of thirty volumes accumulates them quickly. Data usage may look comfortable at 40% while metadata climbs past 90%.

1. Count the snapshots on the pool: lvs -o lv_name,origin,data_percent --select "lv_attr=~^V"
2. Check whether the retention policy is actually deleting them - an expiry job that fails silently is the common root cause
3. Size metadata for the snapshot count you intend to keep, not the count you have today
4. Re-check metadata_percent after any change to snapshot frequency or retention

Autoextend: a safety net, not a strategy

LVM can grow a pool automatically when it crosses a threshold.

# /etc/lvm/lvm.conf
activation {
    thin_pool_autoextend_threshold = 80
    thin_pool_autoextend_percent   = 20
}

With these settings, lvm2-monitor.service extends the pool by 20% whenever it passes 80% full. Confirm the service is running — without it, the settings do nothing:

systemctl is-active lvm2-monitor.service

Exporting the metrics

There is no node_exporter collector for thin pools. Use the textfile collector with a small script on a timer:

#!/usr/bin/env bash
# /usr/local/sbin/lvm-thin-metrics -> /var/lib/node_exporter/textfile/lvm_thin.prom
set -euo pipefail
out=$(mktemp)
lvs --noheadings --nosuffix --units b \
    -o vg_name,lv_name,data_percent,metadata_percent \
    --select 'lv_layout=thin,pool' |
while read -r vg lv data meta; do
  printf 'lvm_thinpool_data_percent{vg="%s",pool="%s"} %s\n'     "$vg" "$lv" "$data"
  printf 'lvm_thinpool_metadata_percent{vg="%s",pool="%s"} %s\n' "$vg" "$lv" "$meta"
done > "$out"
mv "$out" /var/lib/node_exporter/textfile/lvm_thin.prom

- alert: ThinPoolDataHigh
  expr: lvm_thinpool_data_percent > 80
  for: 10m
  labels: { severity: warning }

- alert: ThinPoolDataCritical
  expr: lvm_thinpool_data_percent > 90
  for: 5m
  labels: { severity: critical }

- alert: ThinPoolMetadataHigh
  expr: lvm_thinpool_metadata_percent > 70
  for: 5m
  labels: { severity: critical }

Writing to a temporary file and renaming it into place matters: the textfile collector may read the file mid-write otherwise and parse a truncated line.

Recovering a full pool

1. Stop the writers first. Every additional write makes the situation worse and adds metadata entries. Stop the VMs or services using the thin volumes
2. Identify which resource ran out. lvs -o lv_name,data_percent,metadata_percent vg0/tpool - data and metadata have different fixes
3. If data is full: lvextend -L +NNG vg0/tpool from free VG extents. If the VG has none, add a PV first with pvcreate and vgextend
4. If metadata is full: lvextend --poolmetadatasize +NNG vg0/tpool. If the pool has already been deactivated, lvconvert --repair vg0/tpool rebuilds the metadata into a spare volume - take a copy of the metadata device first if you can
5. Bring the filesystems back deliberately. They will have gone read-only or errored; remount and check each one rather than assuming they recovered
6. Then fix the cause. Deleted snapshots, extended pool, corrected retention job - and confirm the alerts you should have had are now in place

Knowledge check