LVM troubleshooting - common failures and how to recover

LVM failures are rare but high-impact. The most common cases are a missing PV (disk disappeared), a full VG (no room to grow an LV), and a corrupted metadata header. Each has a clear recovery path.

Missing PV

$ pvs

  PV         VG   Fmt  Attr PSize  PFree
/dev/sda2  vg0 lvm2 a--  99.00g 5.00g
/dev/sdb1  vg0 lvm2 a--  500.00g 200.00g
unknown    vg0 lvm2 a--  500.00g 500.00g

Recovery is a staged procedure, not a single command. Work down it in order and stop as soon as the problem is solved.

1. Establish whether the disk is actually gone. A flapping SAN path, a rebooted iSCSI target or a loose cable is not a dead disk, and the PV comes back on its own once the transport recovers.
2. Snapshot the current metadata to a file you control, before touching anything.
3. Run the removal as a dry run to see exactly what it would do.
4. If no logical volume depends on the missing PV, remove it without --force.
5. Only if logical volumes do depend on it, and only after you have a restorable backup, escalate to --force - which deletes those logical volumes.

# 1. Is the disk really gone?
dmesg -T | tail -50
lsblk
sudo iscsiadm -m session          # iSCSI: is the session up?
sudo multipath -ll                # SAN: are any paths still active?

# 2. Snapshot the current metadata BEFORE touching anything.
sudo vgcfgbackup -f /root/vg0-before-recovery.vg vg0

# 3. See exactly what would be removed. --test changes nothing.
sudo vgreduce --removemissing --test vg0

$ vgreduce --removemissing vg0; pvs

  PV         VG   Fmt  Attr PSize  PFree
/dev/sda2  vg0 lvm2 a--  99.00g 5.00g
/dev/sdb1  vg0 lvm2 a--  500.00g 200.00g

# vgreduce --removemissing --force vg0

  WARNING: Partial LV data needs to be repaired or removed.
Removing LV vg0/data from VG vg0
Wrote out consistent volume group vg0.

Undo a bad structural change

ls -lt /etc/lvm/archive/vg0_*.vg        # timestamped, pre-change metadata
sudo vgcfgrestore --list vg0            # pick the seqno from before the mistake
sudo vgcfgrestore -f /etc/lvm/archive/vg0_00042-abcdefgh.vg vg0

# If the PV header itself was wiped, the disk no longer has its UUID.
# Recreate it with the ORIGINAL uuid from the archive file, or the
# restore will not match the metadata:
sudo pvcreate --uuid "kVxi3T-...-original-uuid" \
  --restorefile /etc/lvm/archive/vg0_00042-abcdefgh.vg /dev/sdb1
sudo vgcfgrestore -f /etc/lvm/archive/vg0_00042-abcdefgh.vg vg0
sudo vgchange -ay vg0

A plain pvcreate /dev/sdb1 here would be the second disaster: it writes a new UUID, the archived metadata no longer refers to any disk you have, and the restore becomes impossible. --uuid with --restorefile is what makes the disk answer to its old identity again.

Full VG

$ lvextend -L +100G /dev/vg0/data; lvextend -l +100%FREE /dev/vg0/data

  Insufficient free space: 256 extents needed, but only 0 available
Size of logical volume vg0/data changed from 200.00g to 300.00g

Corrupted metadata

$ vgcfgrestore vg0

  Restored volume group vg0 with metadata sequence 42

$ ls /etc/lvm/backup/vg0*

/etc/lvm/backup/vg0

Knowledge check