Disk Latency Observability · Observability

The database is slow. The application is timing out. The network panels are green. The CPU panel is green. The memory panel is green. The disk panel reports “30% used” and the on-call engineer closes the dashboard. Twenty minutes later the same engineer opens it again, having learned nothing new, and the user-facing latency has tripled.

The dashboard asked the wrong question. The relevant question is not “how much disk is used” but “how long does the disk take to service a request.” Capacity is the wrong axis for performance; latency is the right axis. The metrics that answer the latency question are a different family on node_exporter, derived from /proc/diskstats, and they are easy to filter into uselessness if the operator does not know which device to watch.

What it is

Disk latency observability on Linux is the discipline of measuring four quantities per block device:

Per-request time - node_disk_read_seconds_total and node_disk_write_seconds_total, divided by the corresponding read/write count, give the average service time.
Average wait - node_disk_io_time_seconds_total divided by the I/O count gives the average including queue time. The kernel reports both as cumulative counters; the rate is the metric.
Queue depth - node_disk_queue_length (read from /sys/class/block/<dev>/queue/nr_requests) is the standing queue of requests waiting for the device.
Throughput - node_disk_read_bytes_total and node_disk_written_bytes_total, divided by the scrape interval, give the I/O bandwidth.

The “average” the dashboard reports is misleading on its own; per-request percentiles require histograms, which the kernel and the standard --collector.diskstats do not emit. For production SLOs, the rule is to alert on utilisation and queue depth, and to use the application-side histogram for percentiles.

Why a sysadmin cares

Disk latency is the silent failure mode of production. The filesystem has space; the disk is not “full”; the metric “disk space used” is green. The I/O subsystem is, however, at queue depth 32 and the average wait is 200 ms. The application is slow because every request is waiting on the disk. The operator who looks at “disk used” finds nothing; the operator who looks at node_disk_io_time_seconds_total finds the answer.

A second reason: SSDs and HDDs have different latency profiles. A 10 ms average wait on an HDD is normal; a 10 ms average wait on a NVMe SSD is a serious problem. The same threshold applied to both is wrong. The right discipline is per-host-class thresholds.

A third reason: per-device filtering is the difference between a dashboard that answers questions and a dashboard that returns the average over a thousand devices. The node_disk_* metrics are emitted per block device; the device label is the axis on which the operator pivots. Without filtering, the metric is useless at fleet scale.

How it works

The Linux kernel publishes per-block-device statistics in /proc/diskstats. The fields and the node_exporter mappings:

  /proc/diskstats field      node_exporter metric
  -----------------------    ----------------------------------------
  reads_completed            node_disk_reads_completed_total
  reads_merged               node_disk_reads_merged_total
  sectors_read               node_disk_read_bytes_total (sectors * 512)
  read_time (ms)             node_disk_read_seconds_total (ms / 1000)
  writes_completed            node_disk_writes_completed_total
  writes_merged              node_disk_writes_merged_total
  sectors_written            node_disk_written_bytes_total
  write_time (ms)            node_disk_write_seconds_total
  io_in_progress             node_disk_io_now
  io_time (ms)               node_disk_io_time_seconds_total
  weighted_io_time (ms)      node_disk_io_weighted_seconds_total
  [discards, flush, ...]     node_disk_discards_*
                             node_disk_flush_*

The kernel-side iostat tool reports the same fields. The dashboard equivalent is:

                  Disk Latency (USE: saturation)
                  ===============================

   util %    = 100 * rate(node_disk_io_time_seconds_total[1m])
             - the fraction of time the device was busy during
               the interval. A queue depth of 1 means 100% util
               is the saturation point.

   await (ms)= 1000 * rate(node_disk_read_seconds_total[1m])
               /
               rate(node_disk_reads_completed_total[1m])
             - the average time per read, including queue time.
               On a saturated device, await grows much faster
               than service time.

   svctm (ms)= 1000 * rate(node_disk_read_seconds_total[1m])
               /
               rate(node_disk_reads_completed_total[1m])
             - DEPRECATED. The kernel field this is derived from
               is unreliable; do not use it. The replacement is
               the histogram from the application side.

   avgrq-sz  = (rate(node_disk_read_bytes_total[1m])
               + rate(node_disk_written_bytes_total[1m]))
               /
               (rate(node_disk_reads_completed_total[1m])
                + rate(node_disk_writes_completed_total[1m]))
             - the average request size, in bytes.

   iops      = rate(node_disk_reads_completed_total[1m])
               + rate(node_disk_writes_completed_total[1m])
             - the operations per second.

   bps       = rate(node_disk_read_bytes_total[1m])
               + rate(node_disk_written_bytes_total[1m])
             - the bytes per second.

   queue     = node_disk_queue_length
             - the maximum number of requests the device can
               have in flight (read from sysfs).

   inflight  = node_disk_io_now
             - the number of requests currently in flight.

The svctm field is deprecated. The kernel source removed it because the measurement is unreliable on devices with command queueing. Use await and the application-side histogram.

Under the hood

The kernel updates /proc/diskstats on every I/O completion. The fields are cumulative counters and increase monotonically through the device’s lifetime. The io_in_progress field is a gauge, not a counter; the io_time and weighted_io_time fields are cumulative counters that wrap via the standard Prometheus counter-reset handling.

The node_disk_io_time_seconds_total counter is the sum of milliseconds during which the device had at least one request in flight, divided by 1000. Saturation on a queue of depth 1 (single-platter HDD, no NCQ) means 100% util is the saturation point. On a queue of depth 32 (NVMe), 100% util is one thirty-second of the saturation headroom.

The weighted_io_time field is the modern equivalent of the deprecated svctm; it accounts for the time each request spent in the queue. The Prometheus node_exporter mapper does not currently distinguish this from the raw io_time, so the practical production metric is the application-side histogram plus the node_disk_io_time_seconds_total rate.

The node_disk_queue_length value is read from /sys/class/block/<dev>/queue/nr_requests. The value is the maximum number of requests the kernel will queue, not the current depth. The current depth is node_disk_io_now. The two together are the saturation picture.

The --collector.diskstats.ignored-devices flag is the cardinality control. The default ignores ram, loop, fd, md, and dm- devices. On a docker host, the exclusion should also catch veth and overlay if the disk collector is configured to walk them.

How to configure it

The disk collector is enabled by default. The flags that matter are the device ignore regex and the mount point exclude:

# /etc/systemd/system/node_exporter.service.d/override.conf
[Service]
ExecStart=
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.diskstats.ignored-devices=^(ram|loop|fd|md|dm-).*$ \
  --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run|var/lib/docker/.+)($|/) \
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

The exclude regex above is the default plus the docker overlay. On a host with many block devices (Ceph, MD-RAID, LVM), the regex may need to be tuned to expose the production disks and suppress the virtual devices.

Reload the unit:

# SEVERITY: SERVICE-IMPACT
sudo systemctl daemon-reload
sudo systemctl restart node_exporter

For databases, the production path is to add a dedicated exporter that emits histograms. The node_exporter --collector.diskstats does not emit per-latency histograms; the database-side or application-side exporter fills the gap. The lesson on histograms expands this.

How to validate it

The first check is that the metrics are present per device:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep '^node_disk_reads_completed_total' | head -5

Expected output (illustrative):

node_disk_reads_completed_total{device="nvme0n1"} 1.2e+06
node_disk_reads_completed_total{device="sda"} 8.2e+05
node_disk_reads_completed_total{device="sdb"} 6.4e+04

The label device is the pivot. If the host reports only device="dm-0" and not the underlying physical disks, the device ignore regex is filtering too aggressively.

The second check is the dashboard query. The production-grade disk latency panel is:

100 * rate(node_disk_io_time_seconds_total{device=~"nvme.*|sda|sdb"}[1m])

The label selector is the per-device filter. The expression returns the percentage of the last minute the device was busy. A value above 80% on a single-platter HDD is the saturation warning; above 80% on a NVMe is one thirty-second of the saturation headroom.

The third check is the await query:

1000 * (
  rate(node_disk_read_seconds_total{device=~"nvme.*|sda"}[1m])
  /
  rate(node_disk_reads_completed_total{device=~"nvme.*|sda"}[1m])
)

The expression returns the average read latency in milliseconds over the last minute. The threshold is per host class: HDD database host should be below 15 ms; NVMe database host below 2 ms.

The fourth check is the alert rule:

# /etc/prometheus/rules/disk.rules.yml
groups:
- name: disk.latency
  interval: 30s
  rules:
  - alert: DiskAwaitHigh
    expr: |
      1000 * (
        rate(node_disk_read_seconds_total{device=~"nvme.*|sda"}[1m])
        /
        rate(node_disk_reads_completed_total{device=~"nvme.*|sda"}[1m])
      ) > 25
    for: 10m
    labels:
      severity: ticket
      team: platform
      resource: disk
    annotations:
      summary: 'Average read wait on {{ $labels.instance }}/{{ $labels.device }} above 25 ms for 10m'
      description: 'Device is saturated. Check queue depth and IOPS.'
      runbook_url: 'https://runbooks.example.com/host/disk-latency'

The threshold of 25 ms is the HDD baseline. For NVMe, the threshold is 2 ms; the alert file should be split per host class rather than reused.

How it can fail

Six failure modes appear repeatedly in production.

The disk latency metric is ignored because the dashboard reports “disk used.” The dashboard’s headline metric is node_filesystem_avail_bytes / node_filesystem_size_bytes; the latency panel is buried. Symptom: the on-call engineer closes the dashboard at “30% used” and misses the 100 ms await. The fix is to give the latency panel equal weight on the host overview.
The await metric includes virtual devices. The formula above averages over loop, dm-, and ram devices that report zero latency. The aggregate is dominated by the low-latency devices and the actual problem is invisible. The fix is to filter the device label.
The svctm metric is used. The metric is deprecated and unreliable on devices with command queueing. Symptom: the value is computed but disagrees with the application-side histogram. The fix is to remove the dashboard panel and use the application-side histogram.
The threshold is wrong for the device class. A 25 ms threshold is fine for an HDD; the same threshold for a NVMe is false positive territory. Symptom: alerts fire on healthy NVMe hosts. The fix is per-host-class thresholds.
The kernel block device is not the bottleneck. A user report says “disk is slow,” but the metric is green; the bottleneck is the network filesystem (NFS, Ceph, S3) that fronts the block device. Symptom: the host metrics are healthy, the application is slow. The fix is to inspect the network filesystem path independently.
The disk is failing. The kernel sets node_disk_io_now to a high value and node_disk_io_time_seconds_total rate spikes. The kernel ring buffer reports I/O error. Symptom: the latency metric is high and the log says so. The fix is to replace the device.

How to troubleshoot it

The diagnostic order when a host is slow and disk is the suspect:

Inspect node_disk_io_time_seconds_total rate. Is the device above 80% busy?
Inspect node_disk_io_now. Is the in-flight queue near the maximum queue depth?
Inspect node_disk_read_seconds_total and write equivalent divided by the count. Is the await above the per-host-class threshold?
Inspect node_disk_reads_completed_total rate. Is the IOPS rate unusually high or low?
Inspect node_disk_read_bytes_total rate. Is the throughput at the device’s nominal limit?
Inspect dmesg and journalctl -k. Is the kernel logging I/O errors?
Inspect the application-side histogram. Is the request latency matching the disk latency?

Each step confirms or rules out a layer. The first three answer the saturation question; the next two answer the throughput question; the last two answer the error and the correlation question.

What if a NVMe database host sees 100 ms await on reads?

A database host runs on a NVMe SSD and sees 100 ms await on reads. The threshold is configured for HDDs (25 ms) and the alert fires. The on-call engineer assumes the disk is failing and starts a maintenance window. The actual cause is a background vacuum on the database that is saturating the device. The fix is to inspect the database’s I/O scheduler and the vacuum process, not to replace the disk.

A host runs on a Ceph volume. The Ceph OSD is failing on a different physical host. The latency on the application host is 200 ms; the local disk metrics are green. The fix is to inspect the Ceph cluster and the network path between the application host and the OSD. The local disk metrics are the wrong layer for the question.

A host has a SATA SSD but the kernel reports it as rotational because the SSD does not set the non-rotational bit in the ATA IDENTIFY. The kernel scheduler uses the CFQ elevator; the SSD latency is 50 ms instead of 1 ms. The fix is to set /sys/class/block/<dev>/queue/rotational=0 or to replace the SSD with one that reports correctly.

Security implications

/proc/diskstats is world-readable. The metrics node_exporter emits do not contain file names, file contents, or user data; they are aggregate device counters. The PII surface is low.

The kernel ring buffer may contain I/O error lines that include the device serial number. The serial number is a low-sensitivity identifier but is considered operational metadata. The journal forwarding should be configured to ship the kernel line without alteration.

The --collector.diskstats.ignored-devices regex is a configuration control; an attacker who can modify the systemd unit can suppress metrics for a target device. The fix is to detect drift on the systemd unit and to require code review for changes to the configuration file.

Performance implications

The disk metrics on node_exporter are per-device counters and gauges. The cardinality is the number of devices per host. A host with two disks and a docker overlay has three relevant devices; the metric is cheap. A host with MD-RAID, LVM, Ceph, and docker may have dozens; the metric is acceptable.

The cost of reading /proc/diskstats is a few hundred microseconds per scrape. The scrape interval is not a performance concern.

The bottleneck is the absolute size of the per-device label set. A host with a thousand loop devices (containers, snap) reports a thousand devices; the cardinality is the issue. The fix is the device ignore regex.

Production guidance

Filter the device label. The default ignore regex is a starting point; production fleets tune it per host class.
Set per-host-class thresholds. HDD database host 25 ms await; NVMe database host 2 ms; cache host 5 ms.
Do not use the svctm metric. The kernel field is deprecated and the value is unreliable.
Treat the disk latency metric as the primary performance metric, not “disk used.” The two are different questions.
Alert on the saturation metric (io_time_seconds_total) and the await metric; the throughput metric is the planning question, not the real-time question.
For percentile-based SLOs, use the application-side histogram. The kernel counters do not emit per-request latency.

Verification

You should now be able to answer:

What is the difference between node_disk_io_time_seconds_total rate and node_disk_read_seconds_total rate, and what question does each answer?
Why is the svctm metric deprecated, and what is the replacement?
How does the SSD vs HDD difference change the right await threshold?
What is the right per-device filter for a host with local NVMe, Ceph, and a docker overlay?
What is the diagnostic order when a host is slow and disk is the suspect?