Application USE

A checkout service is reporting p95 latency above its SLO. The RED dashboard shows the duration is high, the rate is normal, the error rate is near zero. The user-visible degradation is real. The RED panels cannot tell the on-call why the service is slow. The expected next move is to look at the host, the container, and the process — and the methodology that walks a resource through three dimensions is the USE method.

USE is the host layer’s counterpart to the application’s RED. While RED says “the service is slow for users,” USE says “the resource the service depends on is saturated,” which is the next step in the investigation loop.

What it is

USE stands for Utilisation, Saturation, and Errors, applied to every resource the application uses. The method was popularised by Brendan Gregg in 2012 and is the canonical checklist for the host layer:

• Utilisation — the fraction of time the resource is busy. For CPU, it is the fraction of a wall-clock second the cores are running. For memory, it is the working set over the limit. For disk, it is the fraction of wall-clock time the device is processing a request.
• Saturation — the degree of queueing. The amount of work queued behind the resource. CPU has a run queue; disks have an I/O queue; networks have a socket buffer.
• Errors — error events on the resource. Disk errors, network interface errors, memory allocation failures.

The four canonical resources the application uses are CPU, memory, disk, and network. The methodology applies to each one independently. A healthy panel has low utilisation, low saturation, and zero errors on each resource for the application’s workload shape.

USE is the opposite of RED in the sense that RED is the user-visible layer and USE is the cause layer. A checkout-latency incident typically starts with RED panels and ends with USE panels: the service is slow, and the host has run out of CPU or the disk is thrashing.

Why a sysadmin cares

The RED panel tells the on-call that the service is slow. The USE panel tells the on-call why the service is slow. The two are complementary; the on-call that has only RED panels has to fall back to logs and traces to find the host-level cause, which costs five to ten minutes per incident.

Three operational questions disappear when USE is in place:

1. “Is this host over-provisioned or under-provisioned?” CPU utilisation at 8% over the last 30 days means the host can be downsized. CPU saturation at 95% means the host is the bottleneck.
2. “Is this service I/O-bound or CPU-bound?” A service with 3% CPU utilisation and 95% disk utilisation is I/O-bound. Resizing CPU does not help; the disk is the bottleneck.
3. “Is the container limit the bottleneck or the host?” A container with cpu.limit=500m and cpu.usage=495m but host cpu.usage=30% is throttled by the cgroup limit, not the host. The fix is the limit, not the host.

How it works

The mental model is: every resource is a queue with a service rate. Utilisation is how busy the server is; saturation is how full the queue is; errors are the events the server reports when it cannot serve the request.

  resource (CPU / memory / disk / network)
       |
       |--- utilisation = % time busy
       |
       |--- saturation  = queue depth (or run-queue length)
       |
       |--- errors      = error events (oom_kill, ENOSPC, etc.)
       |
       v
  application request

The interaction between the three dimensions is what makes the methodology useful. A 95% utilisation does not mean the resource is failing — it means the resource is busy. A 95% saturation means the queue is backing up; the upstream is the bottleneck. A non-zero error rate means the resource is failing requests, regardless of the other two.

The trap is to read utilisation alone. A 5% CPU utilisation with a 95% saturation on the disk I/O queue is a system that is bottlenecked on disk, not on CPU. The dashboard that shows only CPU utilisation tells the on-call the wrong story.

Under the hood

How to configure it

A USE dashboard row needs three panels per resource: utilisation, saturation, and errors. The four resources produce twelve panels; a single Grafana row is the correct display.

1. The Prometheus scrape configs

The cAdvisor and node_exporter scrapes are configured as separate jobs so the labels are distinct:

# /etc/prometheus/prometheus.yml
scrape_configs:
  # SEVERITY: CONFIGURATION -- host-level USE
  - job_name: node
    static_configs:
      - targets:
          - node-01.internal:9100
          - node-02.internal:9100
          - node-03.internal:9100
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: "${1}"
      - target_label: cluster
        replacement: prod-eu-west-1

  # SEVERITY: CONFIGURATION -- container-level USE
  - job_name: cadvisor
    metrics_path: /metrics/cadvisor
    static_configs:
      - targets:
          - node-01.internal:10250
          - node-02.internal:10250
          - node-03.internal:10250
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - target_label: job
        replacement: cadvisor

The cadvisor job uses the kubelet’s /metrics/cadvisor endpoint on port 10250; the on-prem node_exporter runs on port 9100. The two jobs give the host view and the container view from the same node.

2. The recording rules

Twelve panels per row is expensive to evaluate on every dashboard load. The PromQL for the four resources is promoted to a recording rule:

# /etc/prometheus/rules/use.rules
groups:
- name: use
  interval: 30s
  rules:
  # CPU utilisation (host)
  - record: host:cpu_utilisation:rate5m
    expr: |
      1 - avg by (instance) (
        rate(node_cpu_seconds_total{mode="idle"}[5m])
      )

  # CPU saturation (host run queue)
  - record: host:cpu_saturation:rate5m
    expr: |
      avg by (instance) (
        rate(node_load5{}[5m])
      )

  # Memory pressure (host)
  - record: host:mem_pressure:ratio
    expr: |
      1 - (
        node_memory_MemAvailable_bytes /
        node_memory_MemTotal_bytes
      )

  # Disk IO utilisation (host)
  - record: host:disk_io_utilisation:rate5m
    expr: |
      avg by (instance, device) (
        rate(node_disk_io_time_seconds_total[5m])
      )

  # Disk saturation (host queue depth)
  - record: host:disk_io_in_progress:current
    expr: |
      avg by (instance, device) (
        node_disk_io_now
      )

  # Network errors (host)
  - record: host:net_errors:rate5m
    expr: |
      sum by (instance, device) (
        rate(node_network_receive_errs_total[5m])
      ) + sum by (instance, device) (
        rate(node_network_transmit_errs_total[5m])
      )

  # Container CPU throttling
  - record: container:cpu_throttle:rate5m
    expr: |
      rate(container_cpu_cfs_throttled_seconds_total[5m])

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/use.rules

Expected output:

SUCCESS: /etc/prometheus/rules/use.rules

3. The Grafana dashboard row

A single row of twelve panels, organised by resource:

  Host: checkout-prod-01
  +----------------+----------------+----------------+
  | CPU util       | CPU sat (load) | CPU errors     |
  +----------------+----------------+----------------+
  | 32%            | 1.4            | 0              |
  +----------------+----------------+----------------+
  | Mem util       | Mem sat (swap) | OOM kills      |
  +----------------+----------------+----------------+
  | 41%            | 0              | 0              |
  +----------------+----------------+----------------+
  | Disk IO util   | Disk IO queue  | Disk errors    |
  +----------------+----------------+----------------+
  | 8%             | 0.2            | 0              |
  +----------------+----------------+----------------+
  | Net util       | Net drops      | Net errors     |
  +----------------+----------------+----------------+
  | 12%            | 0              | 0              |
  +----------------+----------------+----------------+

The row is one host at a time. The dashboard variable $instance selects the host. The on-call that needs to compare across hosts uses the multi-host row, which is a separate dashboard.

How to validate it

Validate that USE is live and correct with four checks, one per resource.

1. CPU utilisation is present.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=host:cpu_utilisation:rate5m{instance="node-01.internal"}'

Expected output:

{"value":[1755000030,"0.32"]}

A value of 0.32 is 32% — a healthy non-saturated host.

2. Memory pressure is bounded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=host:mem_pressure:ratio{instance="node-01.internal"}'

Expected output:

{"value":[1755000030,"0.41"]}

A value above 0.85 is a signal that the host is close to its memory limit; the page should fire on this metric with a 5-minute for:.

3. Disk IO queue is reported.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=host:disk_io_in_progress:current{instance="node-01.internal"}'

Expected output:

{"value":[1755000030,"0"]}

A value above 1 on a non-RAID device is a signal that disk I/O is queueing. The page should fire on a value above 5 for 5 minutes.

4. Network errors are recorded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=host:net_errors:rate5m{instance="node-01.internal"}'

Expected output:

{"value":[1755000030,"0"]}

A non-zero value is a signal of physical layer issues or buffer pressure. The page should fire on a value above 0.1 for 10 minutes.

How it can fail

Four specific failure shapes appear regularly in USE deployments:

1. Reading utilisation in isolation. The host CPU utilisation is 30%. The on-call concludes the host is healthy. The disk IO queue is at 12. The host is bottlenecked on disk; the metric just does not show it. The checkout service is slow because the disk is slow, not because the CPU is busy. Symptom: increasing checkout latency with 30% CPU utilisation.
2. Memory pressure measured as MemFree. The dashboard uses 1 - (node_memory_MemFree_bytes / node_memory_MemTotal_bytes) as the pressure indicator. The host has 8% MemFree but 60% MemAvailable. The pressure is 40%, not 92%. The page is over-firing; the on-call mutes the channel. Symptom: “memory pressure” page fires every shift on a healthy host.
3. Container CPU throttling not recorded. The KDE --cpu-cfs-quota enforcement is on, but the container_cpu_cfs_throttled_seconds_total series is not in the dashboard. The application is slow because the cgroup is throttling it; the host is idle. The on-call sees the host CPU at 15% and concludes the host is fine. Symptom: application latencies with host CPU below 20%.
4. The cAdvisor job is missing the name label. The scrape config does not apply the name!="" filter, and the dashboard shows the empty container name. The on-call sees the metrics but cannot tell which container is the bottleneck. Symptom: dashboard shows name="" across every panel.

How to troubleshoot it

When a USE panel says something the operator does not believe, the diagnostic order is:

1. Confirm the metric is the host metric, not the container metric. The two have different label sets and live in different jobs. A label mix-up produces a panel that drifts from the host’s actual state.
2. Confirm the resource is the bottleneck. Compare utilisation across the four resources. The highest utilisation is the most likely bottleneck, but the highest saturation is the actual bottleneck. The two can disagree.
3. Confirm the cgroup limit is not the bottleneck. A container with low host CPU utilisation but high container_cpu_cfs_throttled_seconds_total is throttled, not limited. The fix is the limit.
4. Confirm the metric is fresh. A scrape that has fallen behind by more than 2 minutes is a stale reading. The dashboard may show a green panel while the host is on fire.
5. Confirm the alert fires on saturation, not utilisation. A saturated disk has a queue, not a high utilisation. A page on the queue is the right page; a page on utilisation alone misses the incident.

Security implications

The USE metric set is bounded and the labels are structural (host, device, mode). The risk is the process metric: process.executable.path and process.command_line may contain command-line arguments that include credentials or paths to secrets. The OpenTelemetry SDK allows the operator to drop these attributes via the view API; the configured view should drop them by default.

The node_exporter’s textfile collector is another risk: the operator can write arbitrary metrics to a file and the exporter will scrape them. The path should be owned by a non-root user with a strict allowlist.

Restrict the USE query to operators with a recorded purpose. The metrics are not sensitive on their own, but the combination of node identity, network interface, and process list is enough to identify the host’s role and may be useful to an attacker.

Performance implications

node_exporter runs as a single process on each host and reads from /proc and /sys. The cost is a few millicores per host and under 50 MB of RAM. The scrape is every 15 seconds by default; the cardinality is bounded by the number of CPUs, network interfaces, disks, and filesystems — typically under 1000 series per host.

cAdvisor is heavier. It watches the cgroup hierarchy and emits one series per container per metric. A host with 100 containers emits roughly 100,000 series per scrape. The retention budget is the limit; the scrape interval is the lever.

The on-call cost is the dashboard. The twelve-panel row is a few hundred millicores of Grafana query time per minute. The recording rule resolves this.

Production guidance

• Use USE for the host, RED for the application. The two are complementary. The team that has both resolves an incident in twelve minutes; the team that has neither resolves it in sixty.
• Read saturation, not utilisation, as the bottleneck signal. A 30% CPU with a saturated disk is bottlenecked on disk.
• Use MemAvailable, not MemFree, for memory pressure. The kernel reclaims page cache under pressure; MemFree is the wrong signal.
• Bound the cardinality of the cAdvisor scrape. The name!="" filter is mandatory. The id!="/" filter removes the root container.
• Match the dashboard row to the SLO. The RED panel answers “is the service healthy for users?” The USE panel answers “is the host healthy?” The two panels are read together.

Verification

You should now be able to answer:

• What is each of the three USE dimensions, and which kernel resource provides the data for each one?
• Why is saturation the more reliable bottleneck signal than utilisation, and what is the trap in reading utilisation alone?
• What is the difference between MemFree and MemAvailable, and which one is the right pressure metric?
• Why does the cAdvisor metric container_cpu_cfs_throttled_seconds_total matter for application performance, and what does it tell the on-call?
• How does the on-call layer the host, container, and process metrics to isolate the layer that is the bottleneck?

Quiz