Lab: Investigate inodes, deleted-but-open files, and permission drift

This lab trains the muscle memory of the disk-full / inode-full / permission-drift diagnostic. Three failure modes production hosts exhibit, three commands that catch them.

Objective

By the end of this lab, you will be able to take a Linux host, diagnose “the disk is full but I cannot see why”, and lock down a configuration file with an attribute that survives chmod.

Architecture

flowchart LR
  FS["Filesystem<br/>inodes + blocks"]
  DF["df -h / df -i<br/>asks the filesystem"]
  DU["du<br/>walks the directory tree"]
  GAP{"df &gt; du?"}
  LSOF["lsof -nP +L1<br/>find the holder"]
  PROC["/proc/PID/fd/N<br/>recover or truncate"]
  PERM["find -perm<br/>permission audit"]
  CHATTR["chattr +i<br/>tamper-proof files"]
  FS --> DF
  FS --> DU
  DF --> GAP
  DU --> GAP
  GAP -->|"yes: space is unlinked but open"| LSOF
  LSOF --> PROC
  FS --> PERM
  PERM --> CHATTR

Requirements

A Linux host (Ubuntu 24.04 LTS or Debian 12 preferred).
Root or sudo access.
lsof installed (apt install lsof).

Scenario

You are handed a host that is “almost full” and a config file that “should not have changed”. You have one hour. Determine:

Whether the filesystem is byte-full, inode-full, or both.
Whether any disk space is held by deleted-but-open files, how much, and how to get it back without restarting the service.
Whether any file or directory has drifted to a world-writable permission.
Demonstrate that you can lock down /etc/ssh/sshd_config so that even root cannot modify it.

Tasks

Task 1: Read both df reports

df -h
df -i

Identify the mounts at risk. Note any where IUse% exceeds 70% — these are the inode-exhaustion candidates.

Task 2: Create a deleted-but-open file on purpose

A healthy host has nothing to find, so the lab would teach you nothing. Inject the failure first, then diagnose it.

mkdir -p /var/tmp/holdtest
( exec 9>/var/tmp/holdtest/big.log
  dd if=/dev/zero of=/var/tmp/holdtest/big.log bs=1M count=512
  rm /var/tmp/holdtest/big.log
  sleep 900 ) &
HOLDER=$!
echo "holder PID: $HOLDER"

The subshell opens file descriptor 9 on the file, writes 512 MB into it, then unlinks it. The directory entry is gone but the inode is not freed, because a process still holds it open.

Now watch df and du disagree - the single most useful signal in a disk-full incident:

df -h /var/tmp
du -sh /var/tmp/holdtest

du walks the directory tree and reports almost nothing. df asks the filesystem and still counts the 512 MB. That gap is the deleted-but-open space.

Task 3: Find the holder and reclaim the space

sudo lsof -nP +L1

+L1 filters to files with a link count below 1 - exactly the unlinked-but-open set - and prints an NLINK column. Note that it does not print the literal string (deleted), so lsof +L1 | grep deleted returns nothing on every host, healthy or not. The (deleted) annotation belongs to plain lsof, which lists every open file. The two forms are not interchangeable:

# Link-count filter. NLINK column, no "(deleted)" text.
sudo lsof -nP +L1

# Full listing, filtered on the path annotation instead.
sudo lsof -nP | awk '/\(deleted\)/ { print $1, $2, $7, $9 }'

Confirm your injected holder appears, then look at the descriptor directly:

sudo lsof -nP +L1 | grep "$HOLDER"
ls -l /proc/$HOLDER/fd/9

/proc/PID/fd/9 is a symlink to the unlinked path, suffixed (deleted). That symlink is still a working handle on the data, which gives you two options that do not require a restart:

# Recover the contents first if the file mattered (a lost log)
sudo cp /proc/$HOLDER/fd/9 /var/tmp/recovered.log

# Reclaim the space in place, without killing the process
sudo truncate -s 0 /proc/$HOLDER/fd/9
df -h /var/tmp

The space returns immediately and the process keeps running. For a log file still being appended to, truncating the descriptor is the safe emergency move; restarting the service is the alternative, and it costs an outage.

Clean up the holder:

kill $HOLDER
rmdir /var/tmp/holdtest

Expected result: df and du disagree by roughly 512 MB before the truncate, and agree after it. Common real offenders: log files rotated but held by a process without copytruncate, databases with active file handles, container runtimes holding overlay layers.

Task 4: Audit world-writable files

sudo find / -xdev -type f -perm -o+w -not -path '/proc/*' -not -path '/sys/*' -not -path '/dev/*' 2>/dev/null | head

Anything returned is a file every user can modify. Most production hosts return a few entries (some in /var/tmp or application scratch directories) and the rest deserve investigation.

sudo find / -xdev -type d -perm -o+w -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null | head

World-writable directories deserve the same audit. The expected result is a short list including /tmp and possibly a few application directories; the rest need the sticky bit set or a permission fix.

Task 5: Audit setuid binaries

sudo find / -xdev -type f -perm /4000 -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null | sort > /tmp/setuid-audit.txt
wc -l /tmp/setuid-audit.txt
cat /tmp/setuid-audit.txt

Confirm every entry is a vendor package. Use dpkg -S or rpm -qf on any unexpected entry to identify its origin.

Task 6: Lock down sshd_config

sudo chattr +i /etc/ssh/sshd_config
lsattr /etc/ssh/sshd_config
sudo tee -a /etc/ssh/sshd_config <<<'# lock-down test' >/dev/null
sudo chattr -i /etc/ssh/sshd_config
sudo tee -a /etc/ssh/sshd_config <<<'# lock-down test' >/dev/null

The first append should fail (EPERM). The second should succeed. Confirm with:

tail -3 /etc/ssh/sshd_config

Restore the file by removing the test line:

sudo sed -i '$d' /etc/ssh/sshd_config
tail -3 /etc/ssh/sshd_config

Validation

You have successfully completed this lab when you can show:

The df -h and df -i outputs for every mount with > 50% used.
A df -h /var/tmp and du -sh /var/tmp/holdtest pair taken while the holder was running, showing the ~512 MB gap.
The complete sudo lsof -nP +L1 output for the lab host, with the holder PID identified.
A df -h /var/tmp taken after truncate -s 0 /proc/PID/fd/9, showing the space returned while the process was still running.
A list of world-writable files and directories found.
A list of setuid binaries with their package origin.
A successful demonstration of chattr +i blocking a write even by root.

Expected outcome

A written summary explaining:

Which mounts are healthy on bytes and inodes.
Which mounts are at risk of inode exhaustion.
The size of the df-versus-du gap, and what caused it.
Why truncating /proc/PID/fd/N reclaimed the space without a service restart.
Which files are world-writable, and whether each is intentional.
Which setuid binaries exist and which are vendor-supplied.
A demonstration that chattr +i provides tamper resistance even against root.

Troubleshooting

lsof: command not found — apt install lsof (Debian) or dnf install lsof (RHEL).
chattr: Operation not supported while setting flags — your filesystem does not support the immutable flag. Confirm with findmnt -o FSTYPE /. ext4, XFS, and Btrfs support chattr +i; some virtual filesystems do not.
The first append succeeded despite chattr +i — you are in a container with read-only root, or the file is on a filesystem mounted with no_user_xattr or nouser_xattr. Check mount.
lsof +L1 output is empty even with the holder running — you are almost certainly filtering it. lsof +L1 | grep deleted returns nothing on any host: +L1 prints an NLINK column and never emits the string (deleted). Run sudo lsof -nP +L1 bare, or use sudo lsof -nP | awk '/$deleted$/' instead.
The holder subshell exited early — dd may have failed for lack of space in /var/tmp. Check df -h /var/tmp and lower the count=512 to something the mount can hold.
No deleted-but-open files found beyond the injected one — that is the expected result on a healthy host. Document it; the point of Task 2 is that you no longer have to take an empty result on trust.

Cleanup

Confirm the injected holder is gone and its space returned:

sudo lsof -nP +L1 | grep holdtest    # expect no output
ls -d /var/tmp/holdtest 2>/dev/null  # expect no output
rm -f /var/tmp/recovered.log
df -h /var/tmp

If the holder is still listed, kill the PID from Task 2. Space held by a deleted-but-open file is only released when the last descriptor on it is closed.

The chattr flag was removed at the end of Task 6. Verify:

lsattr /etc/ssh/sshd_config

The file should no longer have the i flag. The test line was removed by sed -i '$d' in Task 6; if you skipped that step, remove it manually.

What you learned

You can now diagnose the three common disk-full failure modes (byte exhaustion, inode exhaustion, deleted-but-open) and audit the two common permission-drift failure modes (world-writable files, unexpected setuid binaries). You can also lock down critical files with chattr +i, surviving even root-driven misconfiguration.

Specifically for deleted-but-open space you can now: spot it by comparing df against du, find the holding process with sudo lsof -nP +L1, recover the contents with cp /proc/PID/fd/N, and reclaim the space with truncate -s 0 /proc/PID/fd/N while the service keeps running. That last step is the difference between a five-second fix and an unplanned restart.

Objectives

Prerequisites