USE methodology - the framework for performance investigation

The USE method is a discipline for performance investigation. For every resource, check three things: utilisation, saturation, and errors. This catches most issues quickly.

What USE checks

For every resource (CPU, memory, disk, network):

• Utilisation: how much of the resource is being used. E.g. CPU at 80%, disk at 70%.
• Saturation: how much work is queued waiting for the resource. E.g. run queue length, disk queue depth.
• Errors: error counts. E.g. network errors, disk errors.

A healthy resource has low utilisation, no saturation, no errors. Problems show up as high utilisation, high saturation, or growing error counts.

Why USE works

USE is fast and complete. For every resource, three questions. The answers either:

• Identify the bottleneck (high utilisation or saturation).
• Rule out the resource (low utilisation, no saturation).
• Surface a bug (errors).

Without USE, performance investigation wanders. With USE, the investigation is structured.

Resource checklist

For each resource, USE gives specific metrics:

CPU

• Utilisation: %user, %system, %iowait, %steal (from top or mpstat).
• Saturation: run queue length (the r column in vmstat 1) and cat /proc/pressure/cpu (some avg10). Load average is not a CPU saturation metric on Linux — it counts tasks in uninterruptible sleep as well as runnable ones, so a host blocked on a dead NFS mount shows a load average of 40 with an idle CPU. Use it as a “something is queuing” smoke alarm, then go to r and PSI for the actual answer.
• Errors: hardware errors (from mcelog or rasdaemon).

Memory

• Utilisation: free vs used (from free or /proc/meminfo).
• Saturation: page scan rate, swap activity.
• Errors: allocation failures (from dmesg).

Disk

• Utilisation: %util from iostat -x — with a caveat. %util is the fraction of wall time with at least one request in flight. On a single-queue spinning disk that is a real utilisation figure. On NVMe and SSDs, which serve many requests concurrently, it pins at 100% while the device is nowhere near its throughput or IOPS ceiling. Compare throughput and latency against the device’s rated figures instead of trusting %util.

• Saturation: aqu-sz (average queue size) and r_await/w_await from iostat -x, plus cat /proc/pressure/io.

• Errors: not from iostat — it has no error column in any output mode. I/O errors surface in the kernel log, in SMART data, and in per-device sysfs counters:

  journalctl -k | grep -iE 'I/O error|medium error|blk_update_request'
  sudo smartctl -A /dev/nvme0n1
  DEV=sda   # substitute your device
  cat "/sys/block/$DEV/device/ioerr_cnt" 2>/dev/null

Network

• Utilisation: sar -n DEV throughput against the link rate in /sys/class/net/<iface>/speed. (nicstat prints a %ifutil column directly, but it is not packaged for Debian, Ubuntu or RHEL, so do not build a runbook around it.)
• Saturation: dropped packets, retransmits.
• Errors: CRC errors, frame errors.

How to apply

For each suspected problem:

1. List every resource.
2. For each resource, check utilisation, saturation, errors.
3. Identify the bottleneck.
4. Drill down into the bottleneck.

Example: “the application is slow”.

CPU: 80% user - utilisation high. Is that the cause?
  Saturation: vmstat r column averaging 11 on 4 cores - runnable
  tasks are queuing three deep. /proc/pressure/cpu some avg10=42.
  That is saturation.
Memory: 60% used - utilisation moderate.
  /proc/pressure/memory some avg10=0.00 - not saturated.
Disk: %util 5%, aqu-sz 0.1, w_await 0.4ms - not the cause.
Network: 0% errors - not the cause.

Conclusion: CPU is saturated. Drill into CPU usage:
which processes are using CPU?

Note what settled the CPU question. Load average was not used: 4 runnable tasks on 4 cores is 100% utilised, which the %user figure already told us, and load average would have said the same thing about a host stuck in uninterruptible I/O sleep with an idle CPU. The r column and the CPU pressure figure distinguish “busy” from “over-subscribed”, and only the second one is saturation.

The structure: list resources, check USE, identify the bottleneck, drill in.

Limitations

USE does not catch every problem. It catches resource bottlenecks. Other performance issues (lock contention, CPU cache misses, NUMA imbalance) need additional diagnostics.

For end-to-end latency, also check RED (Rate, Errors, Duration) at the application level.

Knowledge check