iotop and pidstat - per-process I/O statistics

iostat shows the device. iotop and pidstat show the process. After identifying a storage bottleneck, these tools find the cause.

iotop

sudo iotop

Output:

Total DISK READ: 50.00 M/s | Total DISK WRITE: 20.00 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO>    COMMAND
 1234  be/4  www-data  45.00 M/s    0.00 B/s  0.00 %  85.00 %  nginx
 5678  be/4  postgres   5.00 M/s   15.00 M/s  0.00 %  12.00 %  postgres
 1234  be/4  www-data   0.00 B/s   20.00 M/s  0.00 %   3.00 %  nginx

Reading:

• Total DISK READ / WRITE: aggregate throughput.
• TID: thread ID. Same PID may have multiple TIDs.
• DISK READ / WRITE: per-thread I/O rates.
• IO>: percent of time the thread is waiting for I/O.
• SWAPIN: percent of time waiting for swap in.
• COMMAND: the executable.

The thread at the top is doing the most I/O. Investigate that process.

iotop -o shows only processes actually doing I/O, filtering out idle ones.

pidstat -d

pidstat -d 1

Output:

14:30:00  UID    PID   kB_rd/s  kB_wr/s  kB_ccwr/s  iodelay  Command
14:30:01  1000    1234      200.00    0.00      0.00        0  nginx
14:30:01  1000    5678       50.00  150.00      0.00       12  postgres

• kB_rd/s: read KB/s.
• kB_wr/s: write KB/s.
• kB_ccwr/s: cancelled write KB/s (writes that were cancelled, e.g. truncated files).
• iodelay: time this task spent blocked waiting on I/O, in clock ticks. This is the per-process counterpart of system-wide %iowait, and usually the fastest way to attribute iowait to a specific process. A process with modest kB/s and a high iodelay is waiting on slow storage, not generating load.

Find the offending process

# Live view
sudo iotop -o

# Capture for analysis
sudo iotop -o -b -n 5 > /tmp/iotop.log

Identify the top process. Investigate:

• What is the process doing?
• Is it a known workload (backup, log rotation)?
• Is the process misbehaving (infinite loop, runaway query)?

Distinguish read vs write pressure

There is no pidstat flag that filters I/O by direction. The -d report already contains both directions side by side, which is what you compare:

pidstat -d 1

Do not reach for -dr or -dw expecting “reads” and “writes”. In pidstat, -r selects page-fault and memory statistics and -w selects context-switch statistics. Combining them with -d is legitimate, but know what you are asking for:

pidstat -dr 1     # -d I/O  AND  -r page faults / memory
pidstat -dw 1     # -d I/O  AND  -w context switches

If reads dominate: the application is reading a lot, and cache may help. If writes dominate: it is generating write pressure — batch or coalesce the writes in the application, or move the workload to storage with more write IOPS.

biosnoop and biolatency

For more detail, use bcc tools (if installed):

# Per-I/O tracing
sudo biosnoop

# Latency histogram
sudo biolatency

These show every I/O with its latency. Powerful but require bcc.

Common patterns

Knowledge check