Break/Fix: the volume goes read-only and the thin pool is only 62% full

Reported symptoms

• At 03:12 the application on /srv/data began failing every write with Read-only file system. No writes succeed; reads are fine.
• df -h shows the filesystem 41% used with 3.4T available.
• The storage team checked SMART on all twelve disks and the RAID controller. Everything is optimal.
• The nightly snapshot job failed with a message about the thin pool reaching a threshold, but the pool reports 62% data usage.
• Nobody changed anything. The last deployment was nine days ago.

Evidence provided

$ df -h /srv/data
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vg0-appdata    8.0T  3.2T  4.8T  41% /srv/data

$ sudo lvs -o lv_name,lv_size,data_percent,metadata_percent,lv_attr vg0
  LV       LSize   Data%  Meta%  Attr
  appdata    8.00t  62.11         Vwi-aotz--
  data       8.00t  62.11  100.00 twi-aotzM-

$ sudo dmsetup status vg0-data-tpool
0 17179869184 thin-pool 32 32768/32768 6710886/10800000 - ro needs_check

$ sudo dmesg -T | grep -i thin | tail -4
[Tue Aug 11 03:11:58 2026] device-mapper: thin: 253:6: reached low water mark for metadata device: sending event.
[Tue Aug 11 03:12:04 2026] device-mapper: thin: 253:6: metadata operation 'dm_pool_alloc_data_block' failed: error = -28
[Tue Aug 11 03:12:04 2026] device-mapper: thin: 253:6: aborting current metadata transaction
[Tue Aug 11 03:12:04 2026] device-mapper: thin: 253:6: switching pool to read-only mode

$ sudo lvs -a vg0 -o lv_name,lv_size
  LV              LSize
  appdata           8.00t
  data              8.00t
  [data_tdata]      8.00t
  [data_tmeta]    128.00m
  [lvol0_pmspare] 128.00m

$ sudo lvs -o lv_name,origin vg0 --noheadings | grep -c appdata
341

$ sudo vgs vg0
  VG  #PV #LV #SN Attr   VSize  VFree
  vg0   4 342   0 wz--n- 10.90t 1.20t

Work the evidence before reading on

Four numbers in that transcript disagree with each other, and the disagreement is the whole scenario:

1. df says 41%. lvs says Data% 62.11. Those two can differ legitimately — the filesystem sits inside a thin volume, and blocks the filesystem has freed are still mapped in the pool until they are discarded. Neither is anywhere near full.
2. Meta% says 100.00, and it is in a column that only appears because the command asked for it by name.
3. dmsetup status shows two fractions. One of them is 32768/32768.
4. vgs says 1.2T of the volume group is not allocated to anything.

Before reading on, answer one question: which of the two fractions in dmsetup status is the data usage, and which is the metadata usage? Getting that the wrong way round is what sends most operators to the wrong subsystem.

Root cause

1. A thin pool is two volumes wearing one name

vg0/data is not a volume. It is a device-mapper target assembled from two hidden volumes that lvs only shows with -a:

• [data_tdata] — 8 TiB. This holds the actual blocks. It is the number everybody monitors.
• [data_tmeta] — 128 MiB. This holds the block map: which chunk of which thin volume lives at which offset in _tdata, plus the reference counts that let snapshots share blocks.

Data% reports the first. Meta% reports the second. They are independent, and they are consumed by different things.

In dmsetup status the two fractions appear in that order, metadata first:

0 17179869184 thin-pool 32 32768/32768 6710886/10800000 - ro needs_check
                             ^metadata      ^data

32768/32768 is 128 MiB of 512-byte metadata blocks, entirely consumed. 6710886/10800000 is the data, at 62%.

2. Snapshots cost metadata, not data

This is the part that makes the failure counter-intuitive. Creating a snapshot of an 8 TiB thin volume copies no data at all — that is the point of thin snapshots. What it does create is a second set of mappings referencing the same blocks, plus reference counts on every shared chunk. Each subsequent write to either volume splits a shared chunk into two mappings.

The snapshot job here ran nightly and never expired anything. 341 snapshots later, the data volume has grown by whatever the application actually wrote, and the metadata volume has grown by the mapping cost of 341 overlapping views of the same 8 TiB.

3. Read-only is the correct behaviour

When device-mapper cannot allocate a metadata block it cannot record where a new data block went. Accepting the write anyway would mean acknowledging data it could not later find. So it aborts the transaction and switches the pool read-only, which the filesystem above it sees as EROFS.

The pool is not corrupt. It is refusing to become corrupt.

4. Why every first-line check said “healthy”

• df measures the filesystem, which never sees the pool.
• lvs with no -o does not print Meta% at all on most distributions’ default field set.
• SMART and the RAID controller measure the physical layer, three abstractions below the fault.
• The application logged EROFS, which reads like disk failure or a filesystem that was remounted read-only after an error — so the first hour goes into dmesg | grep -i error, xfs_repair planning, and a controller firmware check.

The single command that would have identified this in ten seconds is the one that names the field explicitly.

Resolution

1. Stop the writers first. Every retry against a read-only pool is noise in the journal and, if the application buffers, memory pressure you do not need. Stop the application unit and quiesce the snapshot timer
2. Confirm the volume group has free extents. sudo vgs vg0 — you need unallocated space to grow into. If VFree is 0, this becomes a "add a PV first" problem before anything else can proceed
3. Extend the metadata volume. This is a metadata volume operation, not a metadata write through the pool, so it works while the pool is read-only:
4. `` sudo lvextend --poolmetadatasize +512M vg0/data ``
5. If lvextend refuses because the pool needs a check, deactivate and repair. lvconvert --repair rebuilds the metadata into the _pmspare volume and swaps it in:
6. `` sudo lvchange -an vg0/appdata sudo lvchange -an vg0/data sudo lvconvert --repair vg0/data sudo lvextend --poolmetadatasize +512M vg0/data ``
7. Reactivate and confirm the pool came back read-write before touching the filesystem. sudo lvchange -ay vg0/data then sudo lvchange -ay vg0/appdata, then check dmsetup status for rw
8. Remount the filesystem read-write. sudo mount -o remount,rw /srv/data. If the filesystem itself was flagged, run its own repair path — xfs_repair on an unmounted XFS, fsck on ext4 — before remounting
9. Now delete the accumulated snapshots, oldest first, checking Meta% as you go. Each deletion returns metadata blocks
10. Start the application last, after a write test has succeeded by hand

Verification

1. Metadata usage is off the ceiling. sudo lvs -o lv_name,data_percent,metadata_percent,lv_attr vg0 shows Meta% with real headroom, and the ninth character of lv_attr is - rather than M
2. The pool is read-write and needs no check. sudo dmsetup status vg0-data-tpool ends in rw with no needs_check flag
3. A write actually succeeds. sudo touch /srv/data/.rwtest && sudo rm /srv/data/.rwtest — this is the test that can fail, and it is the only one that proves the application will work
4. The snapshot count is bounded. sudo lvs -o lv_name,origin vg0 --noheadings | grep -c appdata returns the number your retention policy allows, not 341
5. It survives a reactivation. Deactivate and reactivate the pool, or reboot a spare node in the same configuration, and re-check Meta%. A pool that is only healthy until the next activation has not been fixed
6. Monitoring now sees the metadata volume. Force a test alert or lower the threshold temporarily and confirm it fires

Prevention

• Alert on metadata_percent with the same seriousness as data_percent. Most shipped monitoring templates graph only the latter, which is precisely why this class of incident reaches production.
• Enable dmeventd monitoring so the kernel’s low-water-mark event reaches something that can act: sudo lvchange --monitor y vg0/data. The reached low water mark for metadata device line in dmesg fired six seconds before the failure and nothing was listening.
• Set thin_pool_autoextend_threshold and thin_pool_autoextend_percent in /etc/lvm/lvm.conf so the pool grows itself while the volume group still has free extents.
• Give every snapshot job an expiry and alert on snapshot count. The unbounded retention is the real defect; metadata exhaustion is only the symptom it chose.
• Keep free extents in the volume group. A thin pool in a fully allocated VG has no recovery path that does not involve adding hardware.