Docker Host Metrics · Observability

A 03:00 page lands in the queue. Checkout is throwing HTTP 503 for roughly one in three attempts. The container panel in Grafana shows green: every working set is below its limit, CPU is comfortable, no restarts. The host panel shows green too, because there is no host panel. An engineer opens a shell, runs df, and finds the root filesystem at 100 percent. The container dashboards never showed this. They cannot. They show what the cgroup sees, not what the kernel sees.

This lesson is about the signals that only the host view exposes, and how to deploy node_exporter so those signals reach Prometheus on a Docker 28.x host.

What it is

node_exporter is the Prometheus exporter that publishes the host view. It is a single static Go binary that reads from /proc and /sys on the host and serves a Prometheus exposition endpoint on port 9100 by default. It exposes the kind of metrics that cAdvisor cannot: host-level CPU, memory (with separate counters for the page cache, slab, kernel and userspace), filesystem usage across every mounted device, disk I/O latency, network interface counters, entropy, and kernel OOM events.

The right pattern on a Docker host is to run both node_exporter and cAdvisor. The two views do not overlap. They are complementary.

    Host kernel view                    Container view
    +---------------------+             +---------------------+
    | node_exporter       |             | cAdvisor            |
    | /proc, /sys         |             | /sys/fs/cgroup/*    |
    | Host filesystem     |             | cgroup-constrained  |
    | Host network        |             | per-container I/O   |
    | Host OOM events     |             | cgroup OOM events   |
    +---------------------+             +---------------------+
              |                                  |
              +------------+---------------------+
                           |
                       Prometheus

Why a sysadmin cares

The failure modes that only the host view catches are the ones that happen at 03:00: the disk fills, the kernel picks a victim, the entropy pool drains, the network interface starts dropping packets. Every one of these events is invisible to a cgroup-scoped view of the world.

A team that runs cAdvisor only finds out about host disk pressure when their database stops writing. The container metrics keep reporting green because the cgroup still has block budget. The host is the limit, and only the host view reports it.

How it works

The default build of node_exporter enables a curated set of collectors. Each collector owns one slice of the /proc and /sys filesystems:

    Collector            Source               Signal class
    -----------------    --------------       -------------------------
    cpu                  /proc/stat           utilisation, saturation
    meminfo              /proc/meminfo        utilisation, saturation
    loadavg              /proc/loadavg        saturation
    filesystem           /proc/mounts,        utilisation (free bytes,
                         statfs()             inode count, read-only)
    diskstats            /proc/diskstats      utilisation, I/O latency
    netdev               /proc/net/dev        utilisation, errors
    vmstat               /proc/vminfo         saturation (paging,
                                                context switches)
    meminfo_numa         /sys/devices/...     utilisation per NUMA node
    systemd              systemd D-Bus        unit state, restart count
    processes            /proc/[0-9]*         per-PID rss, cpu, fds
    entropy              /proc/sys/kernel/    saturation
                         random/entropy_avail
    textfile             *.prom in a dir      pull-based custom metrics
    uname                uname(2)             identity

The USE method maps cleanly onto this list. Utilisation comes from node_cpu_seconds_total, node_memory_MemAvailable_bytes, and node_filesystem_avail_bytes. Saturation comes from node_load5, node_vmstat_pgpgin, node_network_mtu_bytes, and node_entropy_available_bits. Errors come from node_network_receive_errs_total.

A small number of collectors ship disabled by default and are not free to enable: wifi, hwmon, and the experimental systemd collector. They are empty on most servers and exist for specific hardware; leave them off unless you have the hardware.

Under the hood

node_exporter is a single Go binary that binds one TCP port (default 9100). It serves /metrics for Prometheus and / for a human-readable index of the collectors it has enabled. There is no authentication and no encryption; the security model assumes network isolation. A versioned binary speaks Prometheus exposition format natively; a textfile collector reads *.prom files from a directory and serves them verbatim, which is how a cron job on the host can ship PostgreSQL lag, certificate expiry, or backup-age counters without giving the exporter any credentials.

On a Docker host the binary must be reachable to the same paths the host kernel exposes. That means the container must run with the host PID namespace, the host network namespace, and read-only access to /proc, /sys, and /. The classic invocation binds all three and adds --pid=host.

How to configure it

The minimal production-grade invocation for Docker 28.x:

NODE_EXPORTER_VERSION=v1.8.2

docker run -d \
  --name node-exporter \
  --restart=unless-stopped \
  --net=host \
  --pid=host \
  --read-only \
  --cap-drop=ALL \
  --security-opt=no-new-privileges:true \
  -v "/proc:/host/proc:ro" \
  -v "/sys:/host/sys:ro" \
  -v "/:/rootfs:ro,rslave" \
  -v "/var/lib/node_exporter/textfile_collector:/textfile_collector" \
  "quay.io/prometheus/node-exporter:${NODE_EXPORTER_VERSION}" \
  --path.procfs=/host/proc \
  --path.sysfs=/host/sys \
  --path.rootfs=/rootfs \
  --collector.textfile.directory=/textfile_collector \
  --collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($|/)' \
  --no-collector.wifi \
  --no-collector.hwmon

Severity: CONFIGURATION. A restart of the container is required to apply changes.

Walk through the important flags:

--net=host binds to host port 9100 so Prometheus can scrape without translation. It also means the exporter does not need -p 9100:9100 and there is no bridge network in the way.
--pid=host is required for the processes collector. Without it, the container sees only its own PID namespace and reports a load average of 0.00.
-v "/proc:/host/proc:ro" and the matching /sys bind give the exporter read-only access to the host’s proc and sys. --path.procfs and --path.sysfs point the exporter at the bind mount, not at the container’s own /proc.
-v "/:/rootfs:ro,rslave" is required for the filesystem collector. The rslave propagation ensures that filesystems mounted on the host after the container started appear in node_filesystem_*.
--collector.filesystem.mount-points-exclude strips the bind mounts themselves from the export. Without this the cgroup, proc, sys, and overlay filesystems show up as fake full disks and the alert fires on the wrong target.
--collector.textfile.directory exposes a stable directory for cron-style custom metrics. The directory should be owned by a user the exporter can read; ownership problems are the most common reason this collector appears empty.
--no-collector.wifi and --no-collector.hwmon are explicit opt-outs for collectors that are empty on a server and create noise in the output.
--read-only, --cap-drop=ALL, and --security-opt=no-new-privileges:true reduce the exporter’s privilege. The --cap-drop=ALL is safe because the exporter does not need any Linux capabilities to read /proc and /sys; the --pid=host and bind mounts are enough.

For containerised environments running cgroup v2 (the Docker 28.x default on recent kernels), no extra flag is required for the host metrics themselves; cgroup v2 only affects the per-container collectors in cAdvisor.

How to validate it

Four checks, cheapest first.

# READ-ONLY: the container is running.
docker ps --filter name=node-exporter \
  --format '{{.Names}} {{.Status}} {{.Image}}'
# node-exporter   Up 14 minutes   quay.io/prometheus/node-exporter:v1.8.2

# READ-ONLY: the port is bound on the host network.
ss -ltn | grep 9100
# LISTEN 0  128  0.0.0.0:9100  0.0.0.0:*  users:(("node-exporter",pid=8421,fd=12))

# READ-ONLY: the /metrics endpoint answers.
curl -fsS http://localhost:9100/metrics | head -5
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
# go_gc_duration_seconds{quantile="0"} 2.45e-05

# READ-ONLY: the expected host series exist.
curl -fsS http://localhost:9100/metrics \
  | grep -E '^node_(filesystem_avail_bytes|cpu_seconds_total|memory_MemAvailable_bytes)' \
  | head
# node_cpu_seconds_total{cpu="0",mode="idle"} 52431.72
# node_memory_MemAvailable_bytes 8.43e+09
# node_filesystem_avail_bytes{device="/dev/sda1",mountpoint="/",fstype="ext4"} 1.12e+11

# READ-ONLY: Prometheus sees the target.
curl -fsS http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=up{job="node-exporter"}'
# {"status":"success","data":{"resultType":"vector","result":[{"value":[1734259200,"1"]}]}}

A clean validation: the container is Up, the host port 9100 is bound, the /metrics endpoint returns the Go runtime metrics plus the host series, and up{job="node-exporter"} is 1. Each failure mode below maps to one of these signals failing.

How it can fail

These are the failure shapes that appear most often in production.

node_filesystem_avail_bytes reports only the cgroup filesystems. The host root filesystem never appears in Grafana. Cause: -v "/:/rootfs:ro" was omitted, the --path.rootfs flag was not set, or the bind mount path is wrong. Detection: curl -fsS /metrics | grep node_filesystem_avail_bytes returns only overlay, tmpfs, and the proc mounts.
node_load5 is stuck at zero. Cause: --pid=host is missing. The exporter sees only its own PID namespace, sees only itself, and reports a load average of 0.00. Detection: node_load5 is exactly 0 while the host is obviously loaded.
A disk-usage alert fires on /proc, /sys, or /host. Cause: the --collector.filesystem.mount-points-exclude regex was not set, or it does not match the bind-mount paths in use. The bind mounts appear as real filesystems and the alert fires on the overlay layer. Detection: node_filesystem_avail_bytes{mountpoint="/proc"} exists.
Memory metrics reflect the container cgroup, not the host. Cause: --path.sysfs was not pointed at the bind mount, or the exporter is reading its own container cgroup. The numbers look healthy because the container is small; the host is at 92 percent. Detection: node_memory_MemTotal_bytes is roughly the container memory limit, not the host physical memory.
No metrics at all after docker run. Cause: the daemon was started with seccomp or apparmor profiles that block setns(CLONE_NEWNS); the --pid=host invocation fails silently and the container exits. Detection: docker logs node-exporter shows an operation not permitted error from the runtime.
The textfile collector files never appear. Cause: the bind mount of the host directory was not done, the files are owned by nobody and the exporter cannot read them, or the files have a non-.prom extension. Detection: node_textfile_mtime_seconds is absent.

How to troubleshoot it

The diagnostic order matters. Do not skip the cheap steps.

Is the container running?
```
docker ps --filter name=node-exporter
docker logs --tail 20 node-exporter
```
A clean start logs the listening address and the enabled collectors. Errors here mean a flag, a path, or a capability is wrong.
Is /metrics reachable?
```
curl -sI http://localhost:9100/metrics
```
A connection refused means the port is bound inside the container, not on the host; the --net=host flag is missing or the bridge network is publishing the wrong port.
Are the host series present?
```
curl -s http://localhost:9100/metrics \
  | grep -c '^node_'
```
A low count (under 100 series) means a collector is disabled or a bind mount is wrong. A count of zero means the exporter’s --path.procfs and --path.sysfs are pointing at empty bind mounts.
Is Prometheus scraping it? Confirm up{job="node-exporter"} is 1. If up is 0, the Prometheus configuration is wrong; the exporter is fine.
Are the right collectors enabled? Compare curl -s /metrics | grep '# HELP' against the collector list. A missing node_entropy_* line means --collector.entropy is disabled (default on the unprivileged build) and the host entropy pool is invisible.

Security implications

node_exporter exposes everything readable from /proc and /sys to anyone who can reach port 9100. That includes process command lines, which on a busy host may include database connection strings or session tokens passed on the command line. The default model is network isolation. The bare minimum production hardening:

Bind the listener to the host network only (--net=host).
Restrict the listener to the Prometheus subnet with an inbound firewall rule on port 9100. Prometheus does not authenticate, so the network is the authentication.
Run the container with --read-only and a tmpfs for /tmp.
Drop all capabilities with --cap-drop=ALL; the exporter needs none.
Disable the processes collector on a multi-tenant host if process command lines might carry secrets.
Do not enable the systemd collector on a host whose systemd metadata you do not want scraped.

Do not put node_exporter behind a TLS termination proxy. Prometheus does not speak TLS natively, and adding a proxy in the middle complicates the scrape with no real gain when the network is already isolated.

Performance implications

node_exporter is cheap. The default collector set produces roughly 700 time series and the scrape takes 30 ms to 100 ms on a modern host. The cost is dominated by the filesystem and processes collectors, both of which walk large directories.

Set --collector.processes.max-names=10 if the host has tens of thousands of processes; the collector walks /proc linearly.
Scrape interval stays at 15 s for normal hosts. Drop to 60 s for hosts where scrape cost is measurable in CPU.
The textfile collector is single-threaded and reads sequentially; keep the directory small (under 100 files).
The systemd collector walks the entire unit tree on every scrape. Disable it unless you actually need it.

Production guidance

The single best production discipline is to verify what the host sees and what the container sees are reported separately. On a production host with cAdvisor and node_exporter both running, a Grafana panel that overlays node_memory_MemAvailable_bytes against sum(container_memory_working_set_bytes) shows the host view and the container view side by side. The gap between the two is the memory the kernel is using for page cache, slab, and other host state. A wide gap is normal. A vanishing gap means the host is running out of working memory before the containers are.

Verification

You should now be able to answer:

What signals does node_exporter expose that cAdvisor cannot?
Why is the --pid=host flag non-optional for the processes collector?
What is the symptom of a missing -v "/:/rootfs:ro" bind mount?
How do you confirm Prometheus is scraping the exporter?
Which collectors are off by default and should stay off on a server?