Prometheus + node_exporter + cAdvisor + Grafana · Docker & Containers

Prometheus scrapes, Grafana draws, and three exporters produce the numbers. That much fits on a slide. What does not fit on a slide is which exporter owns which layer, where two of them report the same thing differently, and the two networking details that make a correctly assembled stack scrape nothing at all on the first attempt.

Who publishes what

flowchart LR
  Kernel[Kernel: /proc, /sys] --> NodeExp[node_exporter :9100]
  Cgroups[cgroup v2 tree] --> CAdv[cAdvisor :8080]
  Daemon[dockerd] --> DMetrics[metrics-addr :9323]
  NodeExp --> Prom[Prometheus]
  CAdv --> Prom
  DMetrics --> Prom
  AppExp[Application /metrics] --> Prom
  Prom --> Grafana
  Prom --> AlertMgr[Alertmanager]

Component	Reads	Answers	Does not answer
node_exporter	`/proc`, `/sys`	Is the host healthy	Anything per-container
cAdvisor	the cgroup tree	Is this container healthy	Whether the app inside works
Docker daemon metrics	dockerd internals	Is the engine healthy	Per-container resource usage
Application exporter	the app	Are users affected	Why

The four are not interchangeable and the overlaps are small but real: node_exporter’s node_memory_MemAvailable_bytes and the sum of every container’s working set describe the same machine from different ends, and they will not add up, because the host runs things that are not containers and because page cache belongs to nobody in particular.

node_exporter

Configuration changenode_exporter— Host metrics. The rootfs bind mount plus --path.rootfs is what makes filesystem metrics describe the host rather than the container.

NODE_EXPORTER_VERSION=v1.9.1

docker run -d \
--name node-exporter \
--restart unless-stopped \
--network host \
--pid host \
--read-only \
--cap-drop ALL \
--security-opt no-new-privileges=true \
-v '/:/host:ro,rslave' \
"quay.io/prometheus/node-exporter:${NODE_EXPORTER_VERSION}" \
--path.rootfs=/host

Three flags carry the whole design:

-v '/:/host:ro,rslave' mounts the host root read-only. rslave propagates mounts made on the host after the container started into the container, so a filesystem mounted next Tuesday still appears in node_filesystem_*. Without it, the exporter reports on the set of filesystems that existed at container start and silently stops noticing new ones.
--path.rootfs=/host tells the exporter to prefix host paths with that mount point. Omit it and you get metrics about the container’s own tiny filesystem, which look plausible and are wrong.
--network host and --pid host let network and process metrics describe the host. In a network namespace of its own, node_network_* reports on a veth pair.

The collectors you get by default include cpu, meminfo, filesystem, diskstats, netdev, pressure, loadavg, stat and vmstat — around fifty of them. The vmstat collector is the one worth knowing about here: by default it exports only fields matching ^(oom_kill|pgpg|pswp|pg.*fault).*, which means node_vmstat_oom_kill is available out of the box. That counter is host-wide OOM kills, and it catches the case where the host ran out of memory and the kernel picked a victim, which is a different failure from a container exceeding its own cgroup limit.

cAdvisor

Configuration changecAdvisor— Per-container metrics read from the cgroup tree. The upstream invocation, with the privilege cost stated rather than hidden.

CADVISOR_VERSION=v0.55.1

docker run -d \
--name cadvisor \
--restart unless-stopped \
--publish 127.0.0.1:8080:8080 \
--volume /:/rootfs:ro \
--volume /var/run:/var/run:ro \
--volume /sys:/sys:ro \
--volume /var/lib/docker/:/var/lib/docker:ro \
--volume /dev/disk/:/dev/disk:ro \
--device /dev/kmsg \
--privileged \
"ghcr.io/google/cadvisor:${CADVISOR_VERSION}"

cAdvisor is a privileged container, and you should know why before you run it

That --privileged is the upstream-documented invocation, and this course does not pretend otherwise — but it is a large grant and it deserves an explicit decision rather than a copy-paste.

cAdvisor needs it because it reads /dev/kmsg to observe kernel OOM messages, walks /sys/fs/cgroup across every cgroup on the host, and reads /var/lib/docker to map cgroup IDs to container names. Several of those are blocked by the default seccomp and capability sets.

What that means in practice: a compromise of the cAdvisor container is a compromise of the host. Weigh that honestly.

It is mounted read-only everywhere except /dev/kmsg, which limits the damage from a bug but not from a deliberate escape.
Publish it on 127.0.0.1 only, as above, and let Prometheus reach it over the loopback interface or a dedicated network. A cAdvisor listening on 0.0.0.0:8080 hands the whole container inventory of the host to anyone who can reach the port, with no authentication whatsoever.
Pin the version. :latest on a privileged container means an unreviewed image gains host-level access on the next docker pull.

If that trade is unacceptable in your environment, the alternative is not “run it unprivileged and hope” — the metrics simply will not be collected. It is to drop cAdvisor and take container metrics from the Docker daemon’s own endpoint plus application instrumentation, accepting a much thinner container layer.

The metric names you will actually use, all of them verified against cAdvisor’s published metric list:

Metric	Type	Use
`container_cpu_usage_seconds_total`	counter	`rate()` gives cores consumed
`container_cpu_cfs_throttled_periods_total`	counter	numerator of the throttle ratio
`container_cpu_cfs_periods_total`	counter	denominator of the throttle ratio
`container_spec_cpu_quota`, `container_spec_cpu_period`	gauge	quota divided by period equals the limit in cores
`container_memory_working_set_bytes`	gauge	memory that is not reclaimable cache — alert on this
`container_memory_usage_bytes`	gauge	includes page cache — do not alert on this
`container_spec_memory_limit_bytes`	gauge	the cgroup limit
`container_oom_events_total`	counter	OOM kills inside the cgroup, including children
`container_fs_usage_bytes`, `container_fs_limit_bytes`	gauge	writable-layer and filesystem usage
`container_network_receive_bytes_total`, `container_network_transmit_bytes_total`	counter	per-interface throughput
`container_last_seen`	gauge	timestamp of last observation — the basis for absence alerts
`container_start_time_seconds`	gauge	detects a restart without needing `RestartCount`

The empty-name series that doubles your storage bill

cAdvisor emits a series for every cgroup it can see, not only Docker containers. That includes the root cgroup, every systemd slice, and every .scope on the host, all of which arrive with an empty name label and an id label like /system.slice/ssh.service.

On a busy host this is the majority of the series cAdvisor produces. They are not useless, but they are not what you asked for, and they multiply your cardinality by the number of systemd units on the machine.

Drop them at scrape time. Prometheus relabelling uses RE2, which has no negative lookahead, so express this as “the name label is empty” rather than as “the id label is not a container”:

metric_relabel_configs:
  - source_labels: [name]
    regex: ''
    action: drop

The symptom of not doing this is not an error. It is a Prometheus that grows its TSDB three times faster than you predicted, and a container_* query in Grafana that returns hundreds of rows you have to squint past.

The Docker daemon’s own metrics

The daemon publishes Prometheus metrics when you tell it where to listen. The key is metrics-addr in /etc/docker/daemon.json, and the conventional port is 9323.

metrics-addr on 127.0.0.1 cannot be scraped by a containerised Prometheus

The Docker documentation’s own example is:

{
  "metrics-addr": "127.0.0.1:9323"
}

That is the right default for a host where Prometheus runs natively, and it is the wrong one — silently — when Prometheus runs in a container. 127.0.0.1 inside the Prometheus container’s network namespace is the container’s own loopback, not the host’s. The target goes down, the error says “connection refused”, and the usual first diagnosis is that the daemon setting did not take effect. It did. It is simply unreachable.

Three fixes, in order of preference:

Run Prometheus with --network host. Now 127.0.0.1:9323 means what you meant, and the daemon endpoint is never exposed off-box.
Bind to the bridge gateway, typically 172.17.0.1:9323, so containers on the default bridge can reach it and the wider network cannot. Verify the address on your host — it is the docker0 address, not a constant.
Bind to a management address, if you are scraping from a central Prometheus. The endpoint is unauthenticated and reveals your engine version, daemon ID and container counts, so firewall it to the scraper.

Never 0.0.0.0:9323 on an internet-facing host. It is not a control plane, but it is free reconnaissance.

Service impact possibleenable and verify daemon metrics— Editing daemon.json requires a daemon reload. Verify against the endpoint, not against the file.

# 1. Add the key, keeping any existing content in the file
sudo python3 - <<'PY'
import json, pathlib
p = pathlib.Path('/etc/docker/daemon.json')
cfg = json.loads(p.read_text()) if p.exists() and p.read_text().strip() else {}
cfg['metrics-addr'] = '127.0.0.1:9323'
p.write_text(json.dumps(cfg, indent=2) + '\n')
PY

# 2. Reload. SIGHUP is enough for metrics-addr; it does not stop containers.
sudo systemctl reload docker

# 3. Verify the endpoint answers and carries the expected namespace
curl -fsS http://127.0.0.1:9323/metrics | grep -c '^engine_daemon_'

A non-zero count from step 3 is the evidence. A zero means the daemon reloaded without the key — most often because the JSON was malformed and dockerd kept its previous configuration.

What the engine namespace actually contains

The daemon registers its metrics under the namespace engine, subsystem daemon, so every name begins engine_daemon_. The set is small and specific:

Metric	What it is
`engine_daemon_container_actions_seconds`	histogram of container operations, labelled `action` (`create`, `start`, `delete`, `commit`, `changes`)
`engine_daemon_image_actions_seconds`	histogram of image operations, labelled `action`
`engine_daemon_network_actions_seconds`	histogram of network operations, labelled `action`
`engine_daemon_host_info_functions_seconds`	histogram of host-info gathering
`engine_daemon_container_states_containers`	gauge, labelled `state` = `running` / `paused` / `stopped`
`engine_daemon_health_checks_total`	counter of healthchecks run
`engine_daemon_health_checks_failed_total`	counter of healthchecks that failed
`engine_daemon_health_check_start_duration_seconds`	histogram of healthcheck setup time
`engine_daemon_events_total`	counter of events logged
`engine_daemon_events_subscribers_total`	gauge of current event-stream subscribers
`engine_daemon_engine_info`	info gauge, always 1, labelled `version`, `commit`, `architecture`, `graphdriver`, `kernel`, `os`, `os_type`, `os_version`, `daemon_id`
`engine_daemon_engine_cpus_cpus`, `engine_daemon_engine_memory_bytes`	host CPU count and memory as the engine sees them

The builder registers a second, separate namespace: builder_builds_triggered_total and builder_builds_failed_total, the latter labelled reason with values including dockerfile_syntax_error, build_canceled, unknown_instruction_error and build_target_not_reachable_error.

Two things are conspicuously absent, and it is worth knowing rather than hunting for them: there is no per-container resource metric — that is cAdvisor’s job — and there is no image-pull-failure counter. A failed pull shows up in the daemon journal and nowhere in this endpoint.

Because the histograms are labelled by action, the genuinely useful query here is not a resource graph but a latency one:

histogram_quantile(0.99,
  sum by (le, action) (rate(engine_daemon_container_actions_seconds_bucket[5m])))

A create action whose p99 climbs from 200 ms to 8 s is a storage driver in trouble, and it is visible in this metric long before any container-level graph moves.

Docker’s documentation warns that “the available metrics and the names of those metrics are in active development and may change at any time”. Treat alert rules built on this namespace as needing a check at each engine upgrade, and prefer absent() guards so a renamed metric fails loudly rather than silently never firing.

The scrape config, annotated

Configuration changeprometheus.yml— Every non-default line here exists for a reason stated below it.

global:
scrape_interval: 15s        # container lifetimes are short; 60s misses them
scrape_timeout: 10s         # must be < scrape_interval, and cAdvisor is slow
evaluation_interval: 15s
external_labels:
  host: docker-prod-01      # survives federation and remote_write

scrape_configs:
# ---- host layer -------------------------------------------------------
- job_name: node
  static_configs:
    - targets: ['127.0.0.1:9100']
      labels:
        layer: host

# ---- container layer --------------------------------------------------
- job_name: cadvisor
  scrape_interval: 30s      # cAdvisor walks the whole cgroup tree; 15s hurts
  scrape_timeout: 25s
  static_configs:
    - targets: ['127.0.0.1:8080']
      labels:
        layer: container
  metric_relabel_configs:
    # Drop every cgroup that is not a container: systemd slices, the root
    # cgroup, and machine.slice all arrive with an empty name label.
    - source_labels: [name]
      regex: ''
      action: drop
    # Drop the per-interface network series for the loopback interface.
    - source_labels: [interface]
      regex: 'lo'
      action: drop

# ---- engine layer -----------------------------------------------------
- job_name: docker
  static_configs:
    - targets: ['127.0.0.1:9323']
      labels:
        layer: engine

# ---- application layer ------------------------------------------------
- job_name: apps
  metrics_path: /metrics
  static_configs:
    - targets: ['api:8000', 'worker:8000']
      labels:
        layer: app

The lines that are not defaults, and why:

scrape_interval: 15s globally. Prometheus defaults to 1 minute. A container that crash-loops with a five-second restart backoff is born and dies entirely between two 60-second scrapes, so its CPU and memory series never exist and the incident is invisible in the container layer. Fifteen seconds is the smallest interval most people can afford in storage.
scrape_timeout: 25s on cAdvisor, with a 30-second interval. cAdvisor’s response time scales with the number of cgroups on the host, not the number of containers, and on a host with many systemd units it can take several seconds. The default 10-second timeout produces intermittent scrape failures that look exactly like a flapping network. The timeout must always be less than the interval; Prometheus refuses to start otherwise.
external_labels. The one label that identifies which host an alert came from once metrics leave this Prometheus. Adding it later means every historical series lacks it.
labels: {layer: ...}. Cheap, and it lets a single alert rule or dashboard variable select a whole layer.
metric_relabel_configs rather than relabel_configs. Relabelling runs against targets before the scrape; metric relabelling runs against samples after it. Dropping cAdvisor’s non-container series is a per-sample decision, so it belongs in the second one. Putting it in the first drops the whole target and you get nothing.

When static_configs stops scaling

The config above hardcodes targets, which is correct for one host and tedious for twenty. Prometheus ships a docker_sd_config service discovery that talks to the Docker daemon and creates a target for each network IP and port a container exposes, refreshed every 60 seconds by default.

It solves a real problem and introduces one: Prometheus needs access to the Docker socket to use it, which — as the hardening part of this course covers at length — is equivalent to root on that host. Reaching for docker_sd is a decision to trust your monitoring stack with root, and it is often the right call inside a single trust boundary and the wrong call across one.

The middle path most teams settle on is file-based service discovery, where a configuration-management run writes the target list to a JSON file that Prometheus watches. It keeps the inventory declarative without handing the socket to a scraper.

Alert rules against verified metrics

Configuration changerules/docker.yml— Six rules. Each threshold has a justification in the lesson text; each is written so a renamed metric fails loudly.

groups:
- name: docker-containers
  rules:
    - alert: ContainerRestartLoop
      expr: changes(container_start_time_seconds{name!=""}[15m]) >= 3
      for: 0m
      labels: { severity: ticket }
      annotations:
        summary: 'Container restarted 3+ times in 15 minutes'

    - alert: ContainerOOMKilled
      expr: increase(container_oom_events_total{name!=""}[10m]) > 0
      for: 0m
      labels: { severity: ticket }
      annotations:
        summary: 'OOM kill inside container cgroup'

    - alert: ContainerMemoryNearLimit
      expr: |
        container_memory_working_set_bytes{name!=""}
          / container_spec_memory_limit_bytes{name!=""} > 0.9
        and container_spec_memory_limit_bytes{name!=""} > 0
      for: 10m
      labels: { severity: ticket }
      annotations:
        summary: 'Working set above 90 percent of the cgroup limit'

    - alert: ContainerCPUThrottled
      expr: |
        rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m])
          / rate(container_cpu_cfs_periods_total{name!=""}[5m]) > 0.25
      for: 10m
      labels: { severity: ticket }
      annotations:
        summary: 'Over a quarter of scheduling periods throttled'

- name: docker-engine
  rules:
    - alert: DockerDaemonDown
      expr: up{job="docker"} == 0
      for: 2m
      labels: { severity: page }
      annotations:
        summary: 'Docker daemon metrics endpoint unreachable'

    - alert: DockerMetricsMissing
      expr: absent(engine_daemon_container_states_containers)
      for: 15m
      labels: { severity: ticket }
      annotations:
        summary: 'Engine metric name changed or endpoint reconfigured'

The last rule is the one people leave out. Docker’s engine metric names are explicitly not stable, and a rule whose metric no longer exists does not error — it evaluates to an empty vector and never fires again. absent() turns silent blindness into a ticket.

The and container_spec_memory_limit_bytes > 0 clause in the memory rule is the guard against the false positive from the previous lesson: containers with no limit report a value derived from host memory, and without the guard they would dominate the alert.

Grafana

Add Prometheus as a data source pointing at the Prometheus container, then import the community dashboards as a starting point:

Node Exporter Full, dashboard ID 1860 — the reference host dashboard, and genuinely good.
cAdvisor / Docker, dashboard ID 893 — dated, but a reasonable skeleton for container panels.

Import them, then delete two thirds of the panels. An imported dashboard is a catalogue of everything an exporter can produce, which is the opposite of what you want during an incident. The dashboard that earns its place has four rows in the order of the previous lesson’s failure map: user impact, container health, engine health, host health — so that scrolling down is the same motion as narrowing the cause.

What happens if you run Prometheus on the host it monitors?

It is the normal single-host setup and it has one property worth naming before you rely on it: the monitoring dies with the thing it monitors.

The host’s disk fills. Prometheus cannot write its TSDB, the alert about the disk filling cannot be evaluated, and Alertmanager cannot deliver it because it is on the same host.
The daemon wedges. Prometheus is a container, so it wedges too.
The host reboots. The gap in your data is exactly the interval you most wanted to look at.

This is not an argument against the local stack — it is cheap, it has no network dependency, and it is far better than nothing. It is an argument for one external check that does not share fate: a remote blackbox probe of the application URL, or remote_write to a Prometheus somewhere else. One external signal converts “the monitoring went quiet” from an ambiguity into an alert.

Give the local Prometheus a bounded retention (--storage.tsdb.retention.time and --storage.tsdb.retention.size) as well. A monitoring stack that fills the disk it is monitoring is a genuinely common self-inflicted outage.

Verifying the stack, in a way that can fail

Read-only / Safeevery target, with its last error

$ curl -fsS http://127.0.0.1:9090/api/v1/targets \
| python3 -c 'import json,sys
for t in json.load(sys.stdin)["data"]["activeTargets"]:
  print(t["labels"]["job"], t["health"], t.get("lastError", ""))'

node up
cadvisor up
docker down connection refused
apps up

Illustrative output

That output is the real first-run experience: three targets up and the daemon down, because metrics-addr is bound to a loopback the Prometheus container cannot see. “Open Grafana and see if there are graphs” would not have told you which target failed or why.