Disk Metrics

A database reports slow queries. CPU is at 18%. Memory is fine. The disk panel shows throughput at 80 MB/s on a SATA SSD rated for 500 MB/s. The disk is not the bottleneck. The bottleneck is IOPS: the workload is doing 90 000 random reads per second on a device rated for 70 000. node_disk_* metrics tell you the throughput but the IOPS are the real number, and they are not on the same panel by default. This lesson is about reading node_disk_* so that you do not mistake throughput for capacity.

What it is

The diskstats collector reads /proc/diskstats and emits metric families per block device. The families in node_exporter 1.8.x are:

# Bytes read and written.
node_disk_read_bytes_total{device="sda"}  4.5e+10
node_disk_written_bytes_total{device="sda"}  1.2e+10

# Reads and writes completed (count of I/O operations).
node_disk_reads_completed_total{device="sda"}  2.3e+06
node_disk_writes_completed_total{device="sda"}  8.9e+05

# Time spent reading and writing, milliseconds.
node_disk_read_time_seconds_total{device="sda"}  1.2e+03
node_disk_write_time_seconds_total{device="sda"}  5.6e+02

# IO in flight right now.
node_disk_io_now{device="sda"}  4

# IO time weighted by queue length, milliseconds per second.
node_disk_io_time_seconds_total{device="sda"}  1.2e+04

# Time spent in queue waiting.
node_disk_writes_merged_total{device="sda"}  1.2e+05
node_disk_reads_merged_total{device="sda"}  1.8e+05

The four labels are device, mode (some metrics), and a few others. The cardinalities scale with the number of block devices — typically 1–10 per host, more on database servers.

Per-device vs per-mode

Three of the four families above are per-device only (read_bytes_total, reads_completed_total, read_time_seconds_total). The fourth is the time-weighted metric node_disk_io_time_seconds_total, which is per-device but its rate gives a 0..1 saturation gauge.

The three operations the on-call derives from these metrics:

1. Throughput — bytes per second. Compute from rate(node_disk_read_bytes_total[5m]) + rate(node_disk_written_bytes_total[5m]).
2. IOPS — operations per second. Compute from rate(node_disk_reads_completed_total[5m]) + rate(node_disk_writes_completed_total[5m]).
3. Saturation — the fraction of time the disk was busy. Compute from rate(node_disk_io_time_seconds_total[5m]).

The third is the most operationally useful: a value of 0.8 means the disk was busy 80% of the time. Above 0.95, the disk is the bottleneck.

node_disk_io_now

node_disk_io_now is a gauge that returns the current number of I/O operations in flight on the device. It is the direct read of /proc/diskstats field 9.

A healthy disk shows io_now oscillating between 0 and the device’s queue depth (typically 32 for SATA, 1–64 for NVMe). Sustained io_now near the queue depth is a saturation signal. A value of io_now that stays near zero while throughput is high means the workload is sequential; a value that spikes to dozens while throughput is low means the workload is random and latency-bound.

# Per-device in-flight IOs.
node_disk_io_now

# Devices near their queue depth (rough heuristic).
node_disk_io_now > 16

The kernel also reports per-queue statistics (/sys/block/sda/queue/nr_requests), but node_disk_io_now is the simpler signal.

await vs svctm

The kernel reports per-IO timings: node_disk_read_time_seconds_total (sum of time spent in each read IO) and node_disk_writes_completed_total (number of reads completed). The ratio gives average milliseconds per IO.

# Average read latency, milliseconds.
rate(node_disk_read_time_seconds_total[5m])
  / rate(node_disk_reads_completed_total[5m])
  * 1000

# Average write latency.
rate(node_disk_write_time_seconds_total[5m])
  / rate(node_disk_writes_completed_total[5m])
  * 1000

The historical names for these are await and svctm. svctm is no longer reported by the kernel (since 4.18) because it is unreliable on modern devices; node_exporter does not emit it. await survives as the kernel’s “average per-request wait + service time” but is not directly exposed — the computation above is the modern equivalent.

A healthy NVMe device returns 0.1–0.5 ms per IO. A SATA SSD returns 0.5–5 ms. A spinning disk returns 5–20 ms. Anything consistently above the device’s expected range is a saturation signal.

The limits for SSDs and NVMe

node_disk_* was designed for spinning disks. On SSDs and NVMe devices the metrics are still correct but not sufficient:

• Throughput is rarely the bottleneck. SSDs saturate IOPS before throughput. A SATA SSD at 80% throughput rated capacity may be at 5% IOPS capacity. The dashboard says “disk is fine” while the IOPS saturate.
• rotational=1 is not exposed. The kernel reports rotational status in /sys/block/sda/queue/rotational, but node_exporter’s diskstats collector does not emit it as a label. There is no easy way to filter “spinning” vs “SSD” devices in the metric set.
• Per-device queue depth is bounded. A single NVMe device has many queues (one per CPU core). The node_disk_io_now is per-device, not per-queue. It underreports contention on NVMe.
• Latency distributions are not exposed. A histogram of IO latency would tell you whether the slow tail is the problem; node_disk_* is averages only.

The complementary tools for SSD/NVMe:

• iostat -x 1 for live investigation (not for scraping).
• Per-device SMART counters via smartctl or a dedicated exporter.
• The node_exporter textfile collector for cron-driven SMART data.
• A dedicated storage exporter (e.g. node_exporter + custom collector) for histogram-style latency metrics.

How to configure it

The diskstats collector is enabled by default. The flag to tune is --collector.diskstats.device-exclude, which strips unwanted devices:

--collector.diskstats.device-exclude=^(loop.*|ram.*|dm-.*|md.*|sr.*)$

The regex drops loop devices, ramdisks, device-mapper volumes, software RAID members, and SCSI CD-ROMs — all of which produce noise without operational value. LVM volumes (dm-*) are dropped because the real metrics are on the underlying device; the dm- layer is a thin abstraction.

The recording rules:

# /etc/prometheus/rules/disk.yml
groups:
  - name: disk
    interval: 30s
    rules:
      - record: instance:disk_throughput_bytes:rate5m
        expr: sum by (instance, device) (
            rate(node_disk_read_bytes_total[5m])
          + rate(node_disk_written_bytes_total[5m])
        )

      - record: instance:disk_iops:rate5m
        expr: sum by (instance, device) (
            rate(node_disk_reads_completed_total[5m])
          + rate(node_disk_writes_completed_total[5m])
        )

      - record: instance:disk_busy_ratio:rate5m
        expr: rate(node_disk_io_time_seconds_total[5m])

      - record: instance:disk_read_latency_ms:rate5m
        expr: rate(node_disk_read_time_seconds_total[5m])
          / rate(node_disk_reads_completed_total[5m])
          * 1000

      - record: instance:disk_write_latency_ms:rate5m
        expr: rate(node_disk_write_time_seconds_total[5m])
          / rate(node_disk_writes_completed_total[5m])
          * 1000

The PromQL patterns the on-call uses:

# Devices saturated (>0.95 busy).
instance:disk_busy_ratio:rate5m > 0.95

# Devices with high IOPS (above 50k is usually saturated on SSD).
instance:disk_iops:rate5m > 50000

# Devices with high read latency (>10ms sustained).
instance:disk_read_latency_ms:rate5m > 10

# Devices with sustained in-flight queue.
node_disk_io_now > 16

How to validate it

# READ-ONLY
# 1. The collector is enabled.
curl -sf http://localhost:9100/metrics | grep '^# HELP node_disk_io_time_seconds_total'

# 2. Real devices are exposed.
curl -sf http://localhost:9100/metrics \
  | grep '^node_disk_read_bytes_total' \
  | awk -F'device="' '{print $2}' | awk -F'"' '{print $1}'

# 3. Cross-check against /proc/diskstats.
awk '{print $3}' /proc/diskstats | tail -n +2

# 4. The busy-ratio recording rule returns a sensible value.
# In PromQL:
# rate(node_disk_io_time_seconds_total{device="sda"}[5m])
# A healthy idle host returns ~0.01-0.05.
# A saturated host returns >0.9.

Expected for a healthy database host:

$ curl -sf http://localhost:9100/metrics | grep '^node_disk_io_time_seconds_total'
node_disk_io_time_seconds_total{device="nvme0n1"} 1.23e+05
$ curl -sf http://localhost:9100/metrics | grep '^node_disk_io_now'
node_disk_io_now{device="nvme0n1"} 2

If node_disk_io_time_seconds_total is missing or shows zero, either the device is excluded by the regex or /proc/diskstats is empty (rare, but possible on a host with no block devices, like a memory-only container host).

How it can fail

1. LVM volumes not visible. Symptom: the application is on /dev/mapper/vg0-data, but the panel shows the underlying device /dev/sda. Cause: dm-* excluded. Fix: the metrics are on the underlying device; the panel looks at it. Document the mapping in the runbook.
2. NVMe IOPS saturation invisible. Symptom: database throughput is dropping; the throughput panel says the disk is at 20% capacity. Cause: IOPS are saturated, not throughput. Fix: panel the IOPS gauge separately; alert on it.
3. Loop devices clutter the panel. Symptom: 30 devices in the panel, only 2 are real. Cause: container hosts create loop devices for image mounts. Fix: add loop.* to the device-exclude regex.
4. ZFS volumes look like multiple devices. Symptom: a ZFS pool shows up as several sd* devices. Cause: ZFS uses whole disks and exports a virtual device. Fix: panel the zfs collector (separate metrics under node_zfs_arc_* and similar).
5. Counter reset on device hot-swap. Symptom: throughput gauge drops to zero at the moment of a hot-swap. Cause: the counter resets. Fix: rate() handles counter resets correctly; the dashboard may still show a “drop” that isn’t real. Annotate.
6. Spinning disk and SSD look the same. Symptom: saturation alerts fire on a SATA SSD that is not actually saturated. Cause: throughput and IOPS are conflated. Fix: alert on the busy ratio and on the read/write latency ratio, not on throughput.

Security implications

Disk metrics are low-risk: device names, byte counts, IO counts. They do not reveal file contents or user data. The risk surfaces are:

• The /metrics endpoint reveals device names. On a shared host that is reconnaissance.
• node_disk_io_now and queue depths reveal workload shape (sequential vs random). Low operational risk; minor reconnaissance value.
• node_exporter does not write disk metrics. The textfile collector (opt-in) does write; restrict its directory.

The deeper risk is that some operators turn off device-level metrics because the labels are “ugly.” That decision discards the saturation signal. The fix is the canonical device-exclude regex, not disabling the collector.

Performance implications

The diskstats collector reads /proc/diskstats once per scrape. It is cheap — single milliseconds. The cost is in the dashboard side:

• A panel that expands to thousands of series across a fleet with hundreds of devices is slow to load.
• The recording rules above reduce the per-panel cost.
• Per-device labels on a fleet of 1000 hosts with 5 disks each produce 5000 series per metric family. Negligible cardinality overall.

The bigger performance consideration is device exposure on modern hosts. A host with NVMe namespaces, LVM, ZFS, loop devices, and device-mapper can have 30+ devices. The exclude regex cuts this down; without it, the panel is unreadable.

Production guidance

• Canonical --collector.diskstats.device-exclude regex in version control.
• Recording rules: busy ratio, IOPS, throughput, latency. Five rules per device.
• Alerts:
  • disk_busy_ratio > 0.9 for 5m — early warning.
  • disk_busy_ratio > 0.95 for 1m — page.
  • disk_read_latency_ms > 10 for 5m on devices expected to be sub-millisecond (NVMe).
  • disk_iops > device_ceiling * 0.8 — IOPS-specific.
• Dashboard: busy ratio at top, then IOPS, then throughput, then latency. Sorted by device. The on-call should be able to identify the saturated device in under five seconds.

Verification

You should now be able to answer:

• What does node_disk_io_time_seconds_total measure, and why is its rate the right saturation gauge?
• Why does io_now alone understate saturation on NVMe?
• What is the difference between bytes throughput and IOPS, and which is the bottleneck on a SATA SSD?
• Why does the kernel no longer expose svctm, and what is the modern way to compute per-IO latency?

Quiz