File descriptors and /proc/fd - understanding open files

A file descriptor is a handle to an open file, socket, or pipe. Every open resource uses one. When the limit is hit, the application fails. This lesson covers FD management.

What is a file descriptor?

A file descriptor (FD) is an integer the kernel uses to track an open resource:

• Files on disk.
• Network sockets (TCP, UDP).
• Pipes (anonymous, named).
• Unix domain sockets.
• Special files (PTYs, devices).

The FD table is per-process. The kernel has a system-wide limit too.

Inspect FDs

# Substitute your own values before running:
PID=1234

# Per-process FDs
ls /proc/$$/fd | head
ls /proc/"$PID"/fd | wc -l     # count

# lsof for a process
sudo lsof -p "$PID" | head

# FDs by type
sudo lsof -p "$PID" | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Open files in a directory
sudo lsof +D /var/log | head

Deleted-but-open files

A descriptor keeps a file alive after its last name is removed. rm unlinks the directory entry; the inode and every block it owns stay allocated until the last descriptor closes. This is why df and du disagree: du walks names, df asks the filesystem.

df -h /var/log      # 100% used
du -sh /var/log     # 2 GB accounted for on a 50 GB filesystem

The gap is deleted-but-open space. Find the holder:

# Link-count filter. Prints an NLINK column, never the text "(deleted)".
sudo lsof -nP +L1

# Full listing, filtered on the path annotation instead.
sudo lsof -nP | awk '/\(deleted\)/ { print $1, $2, $7, $9 }'

The two forms are not interchangeable. +L1 selects files whose link count is below 1 and reports that count in an NLINK column; it does not emit the string (deleted). So lsof +L1 | grep deleted returns nothing on any host, healthy or not, and reads as “no orphan files” when the filesystem is full of them. The (deleted) annotation belongs to plain lsof.

/proc/PID/fd is what makes this recoverable. The entry is a working handle on data that has no name left:

ls -l /proc/1842/fd/9              # -> /var/log/app/old.log (deleted)
sudo cat /proc/1842/fd/9 > /var/tmp/recovered.log   # read it back
sudo truncate -s 0 /proc/1842/fd/9 # free the blocks, process keeps running
df -h /var/log

“Too many open files” error

java.net.SocketException: Too many open files
accept: Too many open files

Cause: the process has hit its FD limit (ulimit -n or cgroup limit). The kernel rejects new open() calls.

Fix:

1. Check the limit: cat /proc/<pid>/limits | grep "open files"
2. Check usage: ls /proc/<pid>/fd | wc -l
3. Raise the limit (systemd directive or limits.conf)
4. Find and fix the leak in the application

FD leaks

An FD leak is when a process opens FDs without closing them. Symptoms:

• FD count grows over time.
• Eventually hits the limit.
• Application errors with “too many open files”.

Common causes:

• Files opened without close().
• Network connections not closed (no keep-alive cleanup).
• File handle caching without bounds.
• Database connections not returned to pool.

Detect with monitoring:

# Watch FD count over time
watch -n 60 'ls /proc/<pid>/fd | wc -l'

Alert if FD count grows.

Raise FD limits

For systemd services:

[Service]
LimitNOFILE=65536

For users (via limits.conf):

*       soft    nofile     65536
*       hard    nofile     131072

For containers:

docker run --ulimit nofile=65536:131072 myapp

Apply and restart the service.

Best practices

• Always close FDs (use try-with-resources in Java, with in Python, defer in Go).
• Set FD limits to match expected usage (not unlimited).
• Monitor FD count per process.
• Alert on growth, not just absolute count.
• Test the limit: trigger an FD storm and verify the application handles it gracefully.

Knowledge check