Break/Fix: the filesystem is full and deleting the big file did not help

A filesystem full drill: identify the cause (logs, archive, or just full), decide on the fix, and apply it without losing data.

Tasks

Task 1: Inject the failure

Blast radius: a disposable host or VM, and only the scratch mounts you create below. The injection is size-capped, so the root filesystem is never touched. The Cleanup section at the end of this lesson restores the host exactly.

Create a 100 MiB tmpfs and fill it:

$ sudo mkdir -p /tmp/lab
sudo mount -t tmpfs tmpfs /tmp/lab -o size=100M
sudo dd if=/dev/zero of=/tmp/lab/bigfile bs=1M count=200
# Fails by design: "No space left on device"
df -h /tmp/lab

dd stops at the size cap and exits non-zero. The mount is full; the host is not. Applications writing under /tmp/lab now fail with “No space left on device”.

Optionally, use a real filesystem that is still bounded. A tmpfs has no journal and no reserved blocks, so it cannot reproduce every disk-full failure mode. Build an ext4 filesystem inside a file and loop-mount it instead. The blast radius is one image file you delete afterwards.

$ sudo fallocate -l 256M /var/tmp/fslab.img
sudo mkfs.ext4 -q /var/tmp/fslab.img
sudo mkdir -p /mnt/fslab
sudo mount -o loop /var/tmp/fslab.img /mnt/fslab
sudo dd if=/dev/zero of=/mnt/fslab/bigfile bs=1M count=512
df -h /mnt/fslab
df -i /mnt/fslab

mke2fs asks for confirmation because the target is not a block device. That prompt is your last checkpoint, so read the path it quotes before answering. Never add -F to silence it.

Task 2: Diagnose

df -h                       # which mount is full?
df -i                       # or has it run out of inodes?
du -sh /tmp/lab/* 2>/dev/null | sort -h

Run df before du. df names the full mount in one line; du on the wrong subtree wastes minutes. Then identify the largest consumer. Common causes:

• Log files (especially without log rotation).
• Application data.
• Cache directories.
• Orphan files (deleted but held open).

Task 3: Decide on the fix

The fix depends on the cause:

• Log files: rotate. Configure logrotate.
• Application data: archive old data.
• Cache: clear the cache.
• Disk too small: expand the volume.

The fix must be safe: do not delete data without confirmation.

Task 4: Apply the fix

For log files:

$ # Manual rotation
sudo logrotate -f /etc/logrotate.conf

# Or specific log
sudo logrotate -f /etc/logrotate.d/nginx

# Or manual truncation of a named, identified file
sudo truncate -s 0 /var/log/nginx/access.log

Truncate an active log; never rm it. The writing process holds the file descriptor, so rm unlinks the name but the blocks stay allocated until the process closes or restarts. truncate -s 0 frees the blocks immediately and the process keeps writing to the same inode.

Remove the lab filler the same way you created it, by name:

sudo rm -f /tmp/lab/bigfile
df -h /tmp/lab

For orphan files:

# Link-count filter. Prints an NLINK column, never the text "(deleted)".
sudo lsof -nP +L1

# Full listing, filtered on the path annotation instead.
sudo lsof -nP | awk '/\(deleted\)/ { print $1, $2, $7, $9 }'

The two forms are not interchangeable, and mixing them is the usual reason this step appears to find nothing. +L1 selects files whose link count is below 1 — exactly the unlinked-but-open set — and reports that count in an NLINK column. It does not emit the string (deleted). So lsof +L1 | grep deleted returns no output on every host, healthy or not, and reads as “no orphan files” when the filesystem is full of them. The (deleted) annotation comes from plain lsof, which lists every open file.

Once you have the holder, you do not have to restart it:

# PID 1842, descriptor 9, from the lsof output above
ls -l /proc/1842/fd/9              # -> /var/log/app/old.log (deleted)
sudo truncate -s 0 /proc/1842/fd/9 # blocks freed, process keeps running
df -h /var/log

# Restarting the writer also works, at the cost of a service blip
sudo systemctl restart nginx.service

For disk expansion (LVM):

$ # Extend the logical volume
sudo lvextend -L +10G /dev/vg0/root

# Resize the filesystem
sudo resize2fs /dev/vg0/root    # ext4
sudo xfs_growfs /               # xfs - grow only, XFS cannot shrink

Task 5: Document

TROUBLESHOOTING REPORT: Filesystem Full
Date: 2026-08-09
Symptom: Application failed with "No space left on device"
Impact: Application unavailable
Root cause: Log files grew without rotation
Fix: Implemented log rotation; cleared existing logs
Prevention: Configured logrotate for all services
         Added monitoring for disk usage > 80%

Cleanup

Undo the injection. Run this even if the drill ended early.

$ # tmpfs injection
sudo rm -f /tmp/lab/bigfile
sudo umount /tmp/lab
sudo rmdir /tmp/lab

# Loop-backed injection, only if you ran the optional variant
sudo umount /mnt/fslab
sudo rmdir /mnt/fslab
sudo rm -f /var/tmp/fslab.img

Confirm the host is back to its starting state:

findmnt /tmp/lab      # no output
findmnt /mnt/fslab    # no output
losetup -a            # no loop device on fslab.img
df -h /               # free space unchanged from before the drill

umount releases the loop device automatically, so losetup -d is not needed. If umount reports “target is busy”, find the process holding the mount with sudo lsof +D /mnt/fslab, stop it, then unmount again. Do not use umount -l here: a lazy unmount hides the mount but leaves the loop device and the image file attached.

Knowledge check