Linux Metrics Overview

A 03:00 page says “host db-prod-07 is slow.” The on-call engineer opens Grafana. The CPU panel is green. The memory panel is green. The disk panel is green. The network panel is green. The user complaint is real. The dashboard is not telling the truth because the dashboard is asking the wrong question.

The USE method is the discipline that turns a wall of green panels into a checklist you can run in your head when the page fires. For every resource on a host, you ask four questions: how busy is it, how queued is it, how full has it ever been, and what errors has it produced. The standard Linux host exposes four resource classes: CPU, memory, disk, network. node_exporter reads the kernel and publishes the metrics that answer the questions.

What it is

Linux metrics observability is the discipline of measuring the four resource classes on a Linux host (CPU, memory, disk, network) using the kernel’s own counters, and exporting them to a Prometheus-format scraper. node_exporter is the canonical exposer. It is a single Go binary that reads /proc, /sys, and the filesystems mount table, and serves plain-text metrics on a local HTTP port (9100 by convention). The metrics are the standard kernel counters, not synthetic values from custom scripts.

The alternative is to invent metrics from inside the application or from a wrapper around iostat, vmstat, and mpstat. That re-implements what the kernel already publishes, breaks the consistent time-series naming Prometheus expects, and burns engineering time on metrics that already exist.

Why a sysadmin cares

The first five minutes of a production incident are spent looking at the host. The on-call engineer answers: which host, which resource, which saturation, which error. The faster they have that answer, the faster they have the mitigation. The USE method collapses four questions per resource into a checklist; the node_exporter standard names make the queries portable across hosts, regions, and clouds.

A second reason: every alert that fires on a host is, ultimately, an alert on a resource metric. Latency, error rate, and queue depth on the host are the data points that connect a user-visible symptom to a root cause. Without them, the on-call engineer has no pivot between the symptom and the cause.

How it works

The USE method asks four questions per resource:

                USE Checklist (per resource)
                ============================

   U  Utilisation      - how busy is the resource?
                        CPU time, memory used, disk occupied,
                        link bandwidth used.

   S  Saturation       - how queued is the resource?
                        runqueue depth, page-steal pressure,
                        I/O queue depth, socket backlog.

   E  Errors           - how many operations have failed?
                        CPU is rarely the source; disk and
                        network carry the error counters.

   +  Pressure / Health - psi metrics, soft/hard limits,
                          OOM, drops, carrier.

For each Linux resource class, the standard metrics are:

     Resource   Utilisation metric          Saturation metric
     --------   -----------------------     --------------------
     CPU        node_cpu_seconds_total      node_load5
                                          node_pressure_cpu_waiting_seconds_total

     Memory     (1 - MemAvailable/MemTotal) node_vmstat_pswpin / pswpout
                                          node_memory_pressure_stall_seconds_total

     Disk       (1 - node_filesystem_avail) node_disk_io_time_seconds_total
                byte / node_filesystem_size  node_disk_queue_length
                                          node_disk_await

     Network    node_network_receive_bytes  node_network_up == 0
                / node_network_speed        node_network_*: carrier down
                                          drops, errs, fifo, colls

node_exporter 1.8.x exposes around 1,000 metric series across roughly 30 collectors. The full list is long; the canonical references are the upstream README and the live /metrics endpoint on a deployed instance. The lesson on each resource class (CPU, memory, disk, network) deepens the corresponding row.

Under the hood

How to configure it

The canonical install on Ubuntu 24.04, Debian 12, and RHEL 9 uses the upstream tarball plus a systemd unit. The package repos distribute older versions; the tarball is the supported path.

# SEVERITY: READ-ONLY - inspect the latest release tag first
VERSION=1.8.2
curl -fsSL "https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-amd64.tar.gz" \
  -o /tmp/node_exporter.tgz

# SEVERITY: CONFIGURATION - extracts to /opt
sudo tar -xzf /tmp/node_exporter.tgz -C /opt
sudo mv /opt/node_exporter-${VERSION}.linux-amd64 /opt/node_exporter
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

The systemd unit controls the runtime flags. The flags below are the production baseline; enable only the collectors your fleet needs.

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node_exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/opt/node_exporter/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.systemd \
  --collector.processes \
  --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run|var/lib/docker/.+)($|/) \
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ \
  --collector.netclass.ignored-devices=^(veth.*|docker.*|br-.*)$ \
  --collector.diskstats.ignored-devices=^(ram|loop|fd|md|dm-).*$ \
  --collector.cpu.info
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

The flags worth memorising:

• --web.listen-address=0.0.0.0:9100 - the local Prometheus scrapes one of these. The port is not internet-exposed; the firewall is the boundary.
• --collector.systemd - enables the node_systemd_* metrics. The default is off in 1.8.x; turn it on for the service-availability lesson.
• --collector.processes - exposes selected process counts and states; default is on.
• --collector.filesystem.mount-points-exclude and --collector.filesystem.fs-types-exclude - suppress the dozens of pseudo-filesystems the kernel exposes but production does not care about.
• --collector.netclass.ignored-devices and --collector.diskstats.ignored-devices - exclude virtual interfaces and loop devices that produce high-cardinality noise.

Activate the unit:

# SEVERITY: SERVICE-IMPACT
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

How to validate it

The first check is that the service is up and the port is bound:

# SEVERITY: READ-ONLY
systemctl is-active node_exporter
ss -lntp | grep :9100

ss should report a listener on :9100 owned by the node_exporter user. If the port is missing, the unit failed; inspect journalctl -u node_exporter -n 50 for the cause.

The second check is that the metrics endpoint is exposing the expected collectors:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | head -20

Expected output (illustrative):

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.3e-05
...
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 9023.41
node_cpu_seconds_total{cpu="0",mode="system"} 12.18

The third check is that Prometheus is scraping the target. From the Prometheus host:

# SEVERITY: READ-ONLY
curl -s http://prometheus:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="node") | {url, health, lastScrape}'

Expected: health: "up" and a recent lastScrape timestamp. The next lesson on Prometheus scrape configuration covers the scrape_configs: block.

The fourth check is the cardinality. Count the series per host:

# SEVERITY: READ-ONLY
curl -s http://localhost:9100/metrics | grep -v '^#' | wc -l

A bare-metal host with the flags above reports roughly 800 to 1,200 series. A value above 2,000 indicates a collector that is publishing flood (often filesystem without exclusions on a host with many overlay mounts).

How it can fail

Six failure modes appear repeatedly in production deployments. Each has an observable symptom.

1. The collector is not enabled. The service is up, the port is bound, the metrics endpoint returns 200, but the specific collector is missing. Symptom: node_systemd_* is absent even though the user is on a systemd host. The flag is missing from ExecStart=; restart the unit with the flag and reload.
2. The scrape job is mis-scoped. Prometheus is configured against the wrong port or the wrong IP. Symptom: the target shows health: down in the Prometheus UI with a transport error. The fix is a scrape_configs: edit; never the node.
3. The exclude regex is too aggressive. A team excludes ^/var/lib/docker to suppress container overlays, then adds a database on /data and finds it missing from the metrics. Symptom: node_filesystem_* does not include the expected mount. The fix is to make the exclusion list less greedy.
4. A high-cardinality label appears. A team enables a collector that walks /proc/<pid>/ and emits a label per process. Symptom: Prometheus OOMs; the rule evaluator stalls; every alert goes silent. The fix is to disable the offending collector and bound the label fan-out.
5. The version is drifting. Different hosts run different node_exporter versions. Symptom: query in Grafana returns partial results; some metrics names only exist on the newer version. The fix is a single configuration baseline (Ansible, Salt, Puppet, etc.) pinned to a tag.
6. The kernel is not exporting the source. PSI metrics are missing because the kernel is older than 4.20, or the source is noexec. Symptom: node_pressure_* is absent. The fix is either a kernel upgrade or substituting the older equivalent metrics (node_load5, node_vmstat_pswpin).

How to troubleshoot it

The diagnostic order is from the host outward:

1. Is the service running? systemctl is-active node_exporter.
2. Is the port bound? ss -lntp | grep :9100.
3. Does the local endpoint serve metrics? curl localhost:9100/metrics.
4. Does the network path allow the Prometheus server to reach the port? nc -vz prometheus 9100 from the Prometheus host.
5. Is Prometheus scraping successfully? curl prometheus:9090/api/v1/targets.
6. Is the right collector enabled? curl localhost:9100/metrics | grep <name>.
7. Is the cardinality sane? curl -s localhost:9100/metrics | grep -v '^#' | wc -l.

Each step confirms or rules out a layer. Step 1 is is the service running? Step 5 is is the service doing what I want it to do? The distinction is the one the rubric on troubleshooting emphasises.

Security implications

node_exporter reads the kernel and emits metric values. It does not require root; the node_exporter user is enough for the default collectors. The hardened unit above uses ProtectSystem=, ProtectHome=, PrivateTmp=, and ProtectKernelTunables= to contain the process if it is exploited.

The port is the boundary. Recommendations:

• Bind the listener to 0.0.0.0:9100 and rely on a host firewall (iptables / nftables) to restrict source IPs to the Prometheus server. Do not rely on bind-address alone.
• In clouds (AWS, GCP, Azure), put the host in a security group that allows :9100 only from the scraper subnet.
• If the network between Prometheus and the host is untrusted, front node_exporter with a reverse proxy that requires mTLS via --web.config.file. The cost is operational; the alternative is an internet-exposed kernel surface.
• Do not enable --collector.systemd and --collector.processes on a host that is shared with untrusted workloads. The first exposes service names; the second exposes process names.

Performance implications

Cardinality is the dominant cost. A node_exporter with all collectors enabled and no exclusions produces 8,000 to 12,000 series per host. At 1,000 hosts, that is 8 to 12 million series of Prometheus work. The exclusions cut this by 60 to 80 percent. The disk and network lessons return to per-device filtering.

The scrape interval is the second cost. The default 15 seconds is fine for production; ten seconds doubles the cost in the rule evaluator and the storage; 30 seconds halves it but loses the ability to alert on sub-minute spikes. The lesson on recording rules covers how to keep the granularity and the cost.

The third cost is the metric lifetime. node_exporter counters monotonically increase; they are safe across restarts. But the fleet lifetime and the metric lifetime are different. A counter that resets on restart (the host being reimaged) must be handled by rate() and increase() in PromQL, which handle counter resets natively. The lesson on rate functions expands this.

Production guidance

• Pin the version. node_exporter 1.8.x is the current stable; pin the tag in the configuration management.
• Exclude what you do not need. The mount-points-exclude and fs-types-exclude flags are the two biggest cardinality controls.
• Scrape from the same network region. Cross-region scraping adds latency to the scrape and ties the metrics pipeline to the WAN.
• Bind on the firewall, not the application. node_exporter has no auth; the host firewall is the security boundary.
• Verify the baseline. Run the validation commands above on a fresh host and again after every configuration change. The difference between 800 and 8,000 series is a configuration error, not a feature.

Verification

You should now be able to answer:

• What are the four questions USE asks per resource, and which metrics on node_exporter answer each?
• Why is node_exporter reading /proc and /sys preferable to running iostat and vmstat from a script?
• Which two node_exporter flags most directly bound the cardinality of a host’s scrape?
• What is the diagnostic order when a host’s metrics stop appearing in Prometheus?
• How does the host firewall fit into the security boundary for port 9100?

Quiz