What the daemon can tell you — stats, events, healthchecks, /metrics

Before you deploy an exporter, know what the daemon already emits. There are exactly four channels, and the useful distinction between them is not what they contain — it is whether you can build an alert on them.

Three of the four are things you look at during an incident. One of them is monitoring.

docker stats

docker stats streams live resource usage, refreshing about once a second. --no-stream takes a single sample and exits, which is the form you want in a script.

$ docker stats --no-stream \
--format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.PIDs}}'

NAME         CPU %     MEM USAGE / LIMIT   MEM %     PIDS
otel         0.02%     140MiB / 512MiB     27.34%    12
grafana      0.89%     194.6MiB / 512MiB   38.01%    17
prometheus   1.56%     98.98MiB / 1GiB     9.67%     12
loki         0.36%     114.3MiB / 512MiB   22.33%    17
postgres     0.00%     36.63MiB / 1.5GiB   2.38%     6
redis        0.41%     17.51MiB / 512MiB   3.42%     6

Two columns mislead often enough to be worth memorising.

CPU % is not a share of the host. It is scaled by the number of CPUs, so 100% means one fully consumed CPU and a container using four cores reads 400%. On a 12-CPU host, 1200% is the ceiling, not 100%. An alert written as “CPU % above 90” fires constantly on a busy two-core workload and never fires on a saturated one-core workload.

MEM USAGE / LIMIT shows the host’s total memory as the limit when the container has no memory limit set. MEM % is then that container’s usage as a fraction of the whole host, which is not a number anyone wants. If your MEM LIMIT column shows the same large figure for every container, nothing has a limit — that is the finding, and it matters more than any of the numbers beside it.

docker events

The daemon publishes an event for every lifecycle transition it performs: containers created, started, killed, dying, OOM-killed; images pulled; volumes mounted; networks connected; health status changing.

$ docker events --since 30m --until 0m \
--format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}}'

1786462730 container exec_create: /bin/sh -c wget --quiet --spider http://127.0.0.1:3000/api/health || exit 1 grafana
1786462730 container exec_start: /bin/sh -c wget --quiet --spider http://127.0.0.1:3000/api/health || exit 1 grafana
1786462730 container exec_die grafana
1786462731 container exec_create: /bin/sh -c wget --quiet --spider http://127.0.0.1:8222/healthz || exit 1 nats
1786462731 container exec_start: /bin/sh -c wget --quiet --spider http://127.0.0.1:8222/healthz || exit 1 nats
1786462731 container exec_die nats

--since and --until accept timestamps or durations, and --until with a value makes the command exit instead of streaming — which is what you want in a script. Filter aggressively:

docker events --since 24h --until 0m \
--filter 'event=die' \
--filter 'event=oom' \
--filter 'event=health_status' \
--format '{{.Time}} {{.Actor.Attributes.name}} {{.Action}} exit={{.Actor.Attributes.exitCode}}'

Healthcheck state

A container’s health is exposed on .State.Health, and Docker keeps the last five probe results:

$ docker inspect loki \
--format '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}} probes={{len .State.Health.Log}}'

healthy streak=0 probes=5

Two properties matter for monitoring design:

• health_status events fire on transition, not on every probe. A container that has been healthy for four days emits no health_status events at all. Absence of the event means “no change”, not “no data”.
• The probe log is five deep. A container that flapped unhealthy → healthy → unhealthy → healthy overnight shows a clean status and a FailingStreak of 0 in the morning. The flapping is only recoverable from the event stream, if something captured it.

The daemon metrics endpoint

This is the only one of the four that is a real monitoring interface. Enable it in /etc/docker/daemon.json:

{
  "metrics-addr": "127.0.0.1:9323"
}

Restart the daemon, then scrape it:

curl -s http://127.0.0.1:9323/metrics | grep -E '^engine_daemon' | head

# prometheus.yml
scrape_configs:
  - job_name: docker
    static_configs:
      - targets: ["127.0.0.1:9323"]

Bind it to 127.0.0.1 unless you have a specific reason not to. It is an unauthenticated endpoint that describes your infrastructure.

Sanity check