LVM monitoring and capacity planning · Linux

LVM capacity exhaustion is one of the more confusing Linux failures. The filesystem says “no space left on device” but the LV still has free space. The reason is that the filesystem’s view of the disk and LVM’s view of the disk are different layers.

The capacity layers

Read-only / SafeLVM capacity layers— Four capacity layers: PV (physical disk), VG (pool), LV (allocation), filesystem (inside LV). Each can be a bottleneck. Check that they reconcile: the two PVs total 599.00g and the VG reports exactly that, with 4.00g + 205.00g = 209.00g free. The 90% filesystem usage is the most common production concern; the 209.00g VFree is the pool available for growing LVs.

$ pvs -o+pv_used; echo ---; vgs -o+vg_used,vg_free; echo ---; lvs -o+lv_used; echo ---; df -h /data

  PV         VG   Fmt  Attr PSize   PFree   Used
/dev/sda2  vg0  lvm2 a--   99.00g   4.00g   95.00g
/dev/sdb1  vg0  lvm2 a--  500.00g 205.00g  295.00g
--- VG   #PV #LV #SN Attr   VSize   VFree
vg0    2   3   0 wz--n- 599.00g 209.00g
--- LV   VG  Attr       LSize   Used
data vg0 -wi-ao---- 295.00g 265.00g
--- Filesystem Size  Used Avail Use% Mounted on
/dev/mapper/vg0-data  295G  265G   30G  90% /data

Read the LV Attr column - it is not decoration

Reading this output is the point of the lesson, and the attribute string is where the state actually lives. A VG attribute is six characters (wz--n-); an LV attribute is ten (-wi-ao----). If you see a six-character string in an LV row, you are looking at the wrong column or the wrong command.

The positions that matter day to day:

Position	Values	Meaning
1	`-` plain, `o` origin (has snapshots), `s` snapshot, `I` invalid snapshot, `V` thin volume, `t` thin pool	Volume type
2	`w` writable, `r` read-only	Permissions
5	`a` active, `-` inactive	Activation state
6	`o` open (mounted or in use), `-` not open	In-use state

So:

-wi-ao---- - an ordinary writable LV, active and mounted. This is what a healthy volume looks like.
owi-aos--- - an origin that currently has a live snapshot. Every write to it is now copy-on-write.
swi-a-s--- - a snapshot, active.
swi-I-s--- - a snapshot that has filled its COW space and is unusable. The data behind it is gone; the only action left is lvremove. Position 1 changing from s to I is the single most important state change in this output, and it happens silently.

An invalid snapshot is not recoverable

Once position 1 reads I, the snapshot has been dropped by the kernel because writes to the origin exhausted its allocated COW space. Nothing restores it - not extending it afterwards, not remounting. A backup job that was reading from it produced a truncated or unusable copy, and it very likely exited 0.

Alert on snapshot fill before it happens:

lvs -o lv_name,lv_attr,origin,data_percent --select 'lv_attr =~ ^s'

Anything above 80% Data% needs lvextend now, and any I in the attribute is an incident.

Adding a PV to a VG

Configuration changeadd PV— pvcreate initializes the new disk. vgextend adds the new PV to the existing VG. The VG capacity increases immediately. The new PV's space is now part of the VG's free pool for LV extension.

$ pvcreate /dev/sdc1; vgextend vg0 /dev/sdc1; vgs -o+vg_free

Physical volume '/dev/sdc1' successfully created.
Volume group 'vg0' successfully extended
--- VG   Attr VSize  VFree
vg0  wz--n- 1.50t 1.20t

Extending an LV across multiple PVs

Data-loss riskextend across PVs— lvextend extends the LV by 500G. LVM allocates from any PV that has free extents, so the LV now spans /dev/sda2, /dev/sdb1 and /dev/sdc1. resize2fs grows the filesystem to match. This extension is LINEAR, not striped: LVM appends whole runs of extents from one PV at a time rather than interleaving blocks, so a sequential read still hits one PV at a time and throughput does not increase.

$ lvextend -L +500G /dev/vg0/data; resize2fs /dev/vg0/data

Size of logical volume vg0/data changed from 200.00g to 700.00g
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/vg0/data to 183500800 (4k) blocks.

Striping is not the default and you do not get it by accident. LVM’s default segment type is linear; man lvextend lists -i/--stripes and -I/--stripesize as explicit options, and --type enumerates linear first. To actually stripe, ask for it and give it enough PVs:

lvextend -L +500G -i 3 -I 256k /dev/vg0/data

Check what you really built rather than assuming:

Read-only / Safeverify segment type— Type linear with #Str 1 on each segment is a spanned volume, not a striped one. A striped LV reports Type striped with #Str equal to the number of PVs and a non-zero Stripe size. Run this after every extend so the layout in your head matches the layout on disk.

$ sudo lvs -o lv_name,seg_type,stripes,stripe_size,devices vg0

  LV   Type   #Str Stripe Devices
data linear    1     0  /dev/sda2(0)
data linear    1     0  /dev/sdb1(0)
data linear    1     0  /dev/sdc1(0)

Moving data off a failing disk

Data-loss riskpvmove— pvmove moves all data off one PV to other PVs in the VG. The LV stays mounted throughout. Use this when a disk is showing errors - move data off before it dies, then replace the disk and add it back to the VG.

$ pvmove /dev/sdb1 /dev/sdc1

Detected pvmove in progress, continuing
/dev/sdb1: Moved: 100.00%

Detecting deleted-but-open files

Configuration changelsof +L1— lsof +L1 shows files that have been deleted but are still open by a process. The disk space is still consumed by the open file handle. The space is freed when the process closes the file or restarts.

$ lsof +L1 /data 2>/dev/null | head

COMMAND  PID  USER  FD  TYPE DEVICE  SIZE/OFF  NLINK  NODE NAME
myapp   12345  myapp  42u  REG  253,1   50000000  0  12345 /data/large.log (deleted)

Two layers this lesson does not cover

The four-layer model (PV, VG, LV, filesystem) is complete only for thick volumes on plain disks. Two other layers fill up or fail on their own schedule, and neither shows in df, vgs or lvs -o+lv_used.

Thin pools. A thin pool over-commits: the sum of the thin volumes can exceed the pool. df inside a thin volume reports the volume’s virtual size, so it can read 40% full while the pool underneath is at 100% and writes are already failing. A thin pool has two usage figures, data_percent and metadata_percent, and metadata exhaustion is the worse of the two:

sudo lvs -o lv_name,lv_size,data_percent,metadata_percent vg0

Alert on both. linux-lvm-thin-provisioning covers the thresholds, autoextend and the recovery path.

md RAID underneath the PVs. LVM is an allocator, not a resilience layer, so a degraded mirror below a PV is invisible to every command in this lesson. A VG sitting on a RAID 5 array that has already lost a member looks perfectly healthy in vgs right up to the second failure:

cat /proc/mdstat
sudo mdadm --detail /dev/md0

linux-md-monitoring-and-failure-detection covers mdadm --monitor, the mail alerting and the rebuild window.