node_exporter Overview

A host reports 95% disk full. The on-call opens the dashboard for the host, sees three panels — CPU, memory, network — and nothing about disk. The cause is that node_exporter was deployed without the filesystem collector enabled, or it was deployed behind a firewall rule that blocked Prometheus from reaching it. The incident was not the disk filling up. The incident was the missing telemetry.

node_exporter is the agent that turns a Linux host into a Prometheus scrape target. It exposes host-level metrics (CPU, memory, disk, network, filesystem, kernel VM statistics, hardware sensors, systemd units) over an HTTP endpoint on port 9100. Prometheus pulls the endpoint on a schedule. Everything the rest of this module teaches — node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, node_disk_io_now, node_network_receive_bytes_total — comes from this one binary.

What it is, precisely

node_exporter is a single static Go binary. It does not write a database. It does not push metrics anywhere. It listens on TCP port 9100 by default and serves a GET /metrics endpoint that returns Prometheus exposition format text. The structure is:

   ┌──────────────────────────────┐
   │  Linux host                  │
   │                              │
   │   /proc, /sys, /sys/fs/...   │   <- kernel surface
   │           │                  │
   │           ▼                  │
   │   ┌───────────────────────┐  │
   │   │   collectors          │  │   <- one Go file each,
   │   │   (cpu, mem, disk,    │  │      each reads one surface
   │   │    net, fs, systemd,  │  │
   │   │    textfile, ethtool) │  │
   │   └─────────┬─────────────┘  │
   │             │                │
   │             ▼                │
   │   ┌───────────────────────┐  │
   │   │   HTTP listener :9100 │  │
   │   │   GET /metrics        │  │
   │   └───────────────────────┘  │
   └──────────────────────────────┘
                 ▲
                 │  scrape every 15s
                 │
            Prometheus

A collector is a small Go package that reads one part of the kernel (or one part of the userspace, like systemd) and emits metrics. node_exporter 1.8.x ships around thirty collectors. Most are enabled by default. A handful — filesystem, diskstats, ethtool, wifi, systemd, processes, ntp, perf — are opt-in for cost or correctness reasons. The selection is made on the command line.

Why a sysadmin cares

The host-level metrics that node_exporter emits are the bottom of every USE-method investigation. When a service is slow, the investigation walks down: service metric, dependency metric, host metric. node_exporter is what makes the third leg possible.

Three operational pain points are common without it:

1. Unknown host pressure. A database host shows high query latency. The application panels say nothing useful. The operator does not know whether the host is CPU-bound, IO- bound, or memory-pressure-bound. Without node_cpu_*, node_disk_*, node_memory_* the investigation stalls.
2. Capacity planning with no baseline. “Should we buy another box?” is a question answered by six months of node_cpu_seconds_total and node_memory_* history. Without that history, the answer is guesswork.
3. Alert on saturation. “CPU above 80% for 10 minutes” is a useful alert only if the metric that backs it is real. Without node_exporter, the alert is impossible.

How it works — under the hood

How to deploy it

The canonical deployment is a systemd unit. The official Prometheus documentation recommends running node_exporter as a non-root user with explicit --collector flags and a controlled listener address.

/etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus node_exporter
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=0.0.0.0:9100 \
  --collector.systemd \
  --collector.processes \
  --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/lib/docker/.+)($|/) \
  --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ \
  --collector.netclass.ignored-devices=^(veth.*|docker.*|br-.*)$ \
  --collector.netdev.device-exclude=^(veth.*|docker.*|lo)$ \
  --collector.diskstats.device-exclude=^(loop.*|ram.*|dm-.*|md.*|sr.*)$ \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector
Restart=on-failure
RestartSec=5s
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ReadWritePaths=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

Notes on the choices:

• --collector.systemd and --collector.processes are opt-in. They produce useful alerts (“systemd unit is failed”, “process count grew by 200 in 5 minutes”) but cost extra scrape time.
• --collector.filesystem.mount-points-exclude removes pseudo-filesystems that are not interesting and would otherwise dominate the filesystem panel.
• --collector.netclass.ignored-devices and --collector.netdev.device-exclude remove the noise of virtual interfaces (veth, docker, lo) from network metrics.
• --collector.textfile.directory enables the textfile collector for cron-style metrics (e.g. backup age, cert expiry). See below.
• The hardening lines (NoNewPrivileges, ProtectSystem, ProtectHome) restrict the process so a node_exporter compromise cannot rewrite the host.

The official Prometheus docs use the --collector.disable- defaults flag pattern when a fleet wants to opt-in to each collector explicitly. That is safer at scale (every new collector in a new node_exporter version requires a deliberate decision) but more verbose. A small fleet usually starts with all defaults and adds excludes; a large fleet usually disables defaults and opts in deliberately.

The textfile collector

The textfile collector is the escape hatch for “metrics that no collector knows about.” A cron job drops a Prometheus-format text file into a directory; node_exporter exposes it on the next scrape.

# /etc/cron.d/backup_age_metrics
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

*/5 * * * * root /usr/local/bin/backup_age.sh

#!/bin/bash
# /usr/local/bin/backup_age.sh
# Emits backup_age_seconds gauge to the textfile directory.
OUT=/var/lib/node_exporter/textfile_collector
mkdir -p "$OUT"
TMP=$(mktemp "$OUT/backup_age.XXXXXX")
LATEST=$(stat -c %Y /var/backups/last-good.tar.gz 2>/dev/null || echo 0)
NOW=$(date +%s)
printf '# HELP backup_age_seconds Seconds since the last good backup.\n' > "$TMP"
printf '# TYPE backup_age_seconds gauge\n' >> "$TMP"
printf 'backup_age_seconds %d\n' "$((NOW - LATEST))" >> "$TMP"
mv "$TMP" "$OUT/backup_age.prom"

The atomic rename pattern (mktemp then mv) prevents Prometheus from reading a half-written file mid-rotation.

What is safe vs expensive

Not all collectors cost the same. The cheap ones read /proc files once per scrape: cpu, memory, loadavg, netstat. The expensive ones read more:

• filesystem walks the mount table and stats every mount on every scrape. On a host with hundreds of mounts (containers, bind mounts) this is measurable. The mount-points-exclude flag bounds it.
• ethtool shells out to ethtool -S and ethtool -i for every NIC. This is fine on a few 10G NICs but can take seconds on a host with many virtual functions.
• perf reads kernel perf counters and requires --cap-add=SYS_ADMIN or similar. Most operators leave it off.
• processes enumerates /proc/[0-9]*. Cheap on a host with a few hundred processes; expensive on a database host with thousands of connections.
• systemd talks to systemd’s D-Bus API on every scrape. Cheap on modern systemd; slow on hosts with many transient units.

How to validate it

After the systemd unit starts, the validation chain is:

# READ-ONLY
# 1. Service is running.
systemctl status node_exporter.service

# 2. Listening on the expected address.
ss -ltnp | grep ':9100'

# 3. Endpoint responds and contains the expected metric families.
curl -sf http://localhost:9100/metrics | grep -E '^node_(cpu|memory|disk|network|filesystem)' | head

# 4. Prometheus sees it as up.
# In Prometheus, the Targets UI shows the endpoint.
# Programmatic equivalent:
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=up{job="node_exporter"}' | jq '.data.result[0]'

Expected response from the curl /metrics snippet:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 1.234e+05
node_cpu_seconds_total{cpu="0",mode="iowait"} 12
node_cpu_seconds_total{cpu="0",mode="system"} 4.5e+03
node_cpu_seconds_total{cpu="0",mode="user"} 9.8e+03
node_cpu_seconds_total{cpu="1",mode="idle"} 1.231e+05
...
# HELP node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 4.5e+09
...

Each metric family should be present. If node_filesystem_* is missing, the filesystem collector was disabled or no filesystem passed the fs-types-exclude filter. If node_network_* is missing, all interfaces were excluded.

The validation also includes the up job:

# READ-ONLY
# The scrape duration should be well below the scrape interval.
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=scrape_duration_seconds{job="node_exporter"}' \
  | jq '.data.result[].value'

A scrape duration above 1 second on a default install means a collector is misbehaving (almost always filesystem or ethtool). The fix is an exclude.

How it can fail

The high-frequency failure modes, each tied to a recognisable symptom:

1. node_exporter not running. Symptom: up{job= "node_exporter"} == 0 for the host. The host panels go dark. Fix: systemctl status, journalctl -u node_exporter, fix the unit. This is loud, not silent.
2. Port blocked at the firewall. Symptom: same up == 0 on the Prometheus side, but node_exporter is healthy locally. Fix: open TCP 9100 between Prometheus and the host. The lesson’s most common cause is a cloud security group.
3. Scrape exceeds the scrape interval. Symptom: scrape_ duration_seconds rising toward scrape_interval_seconds. Prometheus begins to drop samples. Fix: identify the expensive collector with the per-collector metrics (node_scrape_collector_duration_seconds), add an exclude.
4. High cardinality from unintended labels. Symptom: Prometheus TSDB head grows faster than expected. node_exporter labels are usually small. The exception is the processes collector: process_exporter style labels can blow up cardinality if the process name contains a request ID. Fix: drop the collector or fix the process name.
5. Conflicting port. Symptom: node_exporter refuses to start with “address already in use”. Fix: check ss -ltnp for 9100. Often a previous instance or a test container.
6. Stale binary. Symptom: a new collector flag is accepted by --help but rejected on start. Cause: the systemd unit was updated but the binary was not. Fix: install the matching binary version.

Security implications

node_exporter exposes process-level and host-level data. Anyone who can read /metrics can read:

• Filesystem mount points and labels.
• Network interface names and addresses.
• The list of systemd units and their states.
• Hardware details (CPU model, memory size, NIC firmware).

In a hostile network, this is reconnaissance for an attacker. Two operational defaults mitigate this:

1. Bind to a private address. --web.listen-address= 10.0.0.10:9100 rather than 0.0.0.0. Prometheus reaches it through the management network; nothing else does.
2. Restrict with a firewall. Even on the management network, restrict TCP 9100 to the Prometheus source IPs.
3. Drop unnecessary collectors. The systemd and processes collectors leak more than the kernel ones. Disable them if the operational value is not used.

The --web.config flag enables mutual TLS and basic auth on the listener. For most internal deployments a network ACL is sufficient; for any deployment that crosses a trust boundary, enable auth.

node_exporter does not write to the filesystem under normal operation. It writes only via the textfile collector, into the directory you specify, and it does not follow symlinks. The directory should be owned by the node_exporter user and have mode 0755.

Performance implications

The node_exporter process is small but not negligible. Typical numbers on a 2025-era Linux host with the recommended enable set:

• Resident memory: 15–30 MiB.
• CPU during scrape: 20–80 ms of one core.
• Scrape duration: 50–400 ms depending on collector set.

These numbers are bounded by the scrape interval, not by time-of-day. A 15-second scrape interval on a single host means roughly 1% CPU spent in node_exporter on average.

The levers are:

• Scrape interval. 30s is fine for most hosts; 15s is the common default. Going below 10s on a large fleet starts to matter in the Prometheus TSDB.
• Collector set. The excludes shown above cut 60–80% of the scrape duration on hosts with many mounts or interfaces.
• Number of instances per Prometheus. A single Prometheus server can scrape 10 000 node_exporters at a 15s interval without breaking a sweat on modern hardware, assuming a bounded collector set.

Production guidance

• One canonical systemd unit per OS family. Distribute with Ansible, Salt, or your config-management tool of choice.
• One canonical Prometheus scrape job, with relabel_config stripping the host label down to a stable identifier.
• Exclude virtual interfaces and pseudo-filesystems by default. The dashboard that ships with these excludes is the dashboard operators learn to read.
• Set an alert on up{job="node_exporter"} == 0 for 2 minutes. This is the loud failure path.
• Set an alert on count by (__name__) ({__name__== "node_filesystem_avail_bytes"}) < 5 for 10 minutes on hosts that should have filesystems. This is the silent failure path.
• Pin the binary version. node_exporter 1.8.x is the current production line as of this lesson; upgrades go through the normal canary discipline.

Verification

You should now be able to answer:

• Where does node_exporter get its data, and what is a collector?
• Which collectors are enabled by default, and which are opt-in?
• What does the textfile collector do, and when would you use it?
• Why does a Kubernetes node need device-exclude=^(veth.*)$?
• What is the difference between loud and silent failure of node_exporter, and which alert catches which?

Quiz