Failure: filesystem full, inodes exhausted, filesystem read-only

Three distinct failures present with the identical symptom: the application logs ENOSPC, or “No space left on device”, or simply stops writing. They need three different fixes, and one of them is a hardware incident. Two commands separate them.

The two-command split

$ df -h /var; df -i /var

Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg0-var  20G   19G     0 100% /var
Filesystem       Inodes  IUsed IFree IUse% Mounted on
/dev/mapper/vg0-var 1.3M   214K  1.1M   17% /var

• df -h at 100% with df -i low: the filesystem is full.
• df -h with space free and df -i at 100%: inodes are exhausted. Every write fails with ENOSPC even though gigabytes are free, because there is no free inode to allocate a new file. This is the one that wastes hours, because the operator keeps looking at df -h, sees free space, and disbelieves the error.
• Both showing free space, writes still failing: the filesystem has been remounted read-only. See below.

Inode exhaustion has a characteristic shape: millions of tiny files, usually a spool, a session directory, a mail queue, or a cache with no eviction. Find the directory holding them by counting entries, not bytes:

# the directories with the most entries, not the most bytes
find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head

-xdev keeps the search on one filesystem. Without it you walk the whole tree and misattribute the count.

Full: find the space, and check for the invisible kind

Start with the obvious:

du -x -h -d1 /var 2>/dev/null | sort -h | tail

-x again stays on one filesystem. If du accounts for far less than df reports as used, the missing space is held by deleted but still open files - a log the application still has an open descriptor to, unlinked by a rotation that never signalled the process:

# lsof -nP +L1 /var | head

COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NODE NAME
java     3412 app   14w   REG  253,2 8589934592     0  918 /var/log/app/app.log (deleted)

NLINK 0 with a live descriptor is the signature. The space returns when the descriptor closes - so signal the process to reopen its log (kill -HUP, or the service reload), or restart it. Deleting more files will not help, because the file is already deleted.

Read-only: this is not a filesystem problem

A filesystem that remounts itself read-only did not run out of anything. The kernel took it read-only because a write failed and it will not risk corrupting more:

$ dmesg -T | grep -iE 'remount|read-only|I/O error|medium error' | tail; mount | grep ' / '

[Mon Aug 10 04:12:19 2026] blk_update_request: critical medium error, dev sda, sector 190312448
[Mon Aug 10 04:12:19 2026] EXT4-fs (dm-1): Remounting filesystem read-only
/dev/mapper/vg0-root on / type ext4 (ro,relatime,errors=remount-ro)

The order of the two lines is the diagnosis: an I/O or medium error preceded the remount. That is a failing disk, a failing controller, or a dropped storage path - not a filesystem you should immediately remount read-write.

1. Preserve the evidence: capture dmesg and the journal now, because a reboot loses the ring buffer.
2. Check the hardware: smartctl -a on the device, and the controller or SAN path status.
3. Decide whether the data is still trusted. If the array is degraded, fix redundancy before touching the filesystem.
4. Only then run the filesystem check, offline, on a device you have confirmed is healthy.
5. Remounting read-write to "get the service back" without any of the above is how a recoverable disk failure becomes a restore from backup.

A read-only remount with no I/O error before it is a different animal: usually a filesystem metadata inconsistency, or a deliberate errors=remount-ro triggered by a corruption the kernel detected. Same discipline, different cause.

LVM-backed: check the layer below

On LVM the filesystem can be full while the volume group has free extents, which makes recovery cheap:

vgs                      # VFree on the group
lvextend -r -L +5G /dev/vg0/var    # -r grows the filesystem too

Thin pools invert the risk. A thin pool at 100% data usage gives every filesystem on it write errors at once, and the filesystems themselves may still report free space. Always check lvs -o +data_percent,metadata_percent before believing df.

Knowledge check