What to monitor — host, Docker, container, application

Most Docker monitoring advice stops at “collect CPU, memory, disk and network”. That advice is not wrong, it is just not a monitoring strategy — it tells you what a machine is doing without telling you which of the six ways a containerised service fails is currently happening.

This lesson runs the other direction. Start from the failure, then name the signal that distinguishes it from every other failure.

The four layers

flowchart TB
  subgraph App[Application]
    A1[Request rate]
    A2[Error rate]
    A3[Latency p50/p95/p99]
    A4[Saturation]
  end
  subgraph Container
    C1[CPU throttling vs limit]
    C2[Memory working set vs limit]
    C3[Restart count]
    C4[Healthcheck state]
    C5[Block and network I/O]
  end
  subgraph Docker
    D1[Daemon reachable]
    D2[Image pull failures]
    D3["/var/lib/docker by subtree"]
    D4[Container state counts]
  end
  subgraph Host
    H1[CPU and run queue]
    H2[Memory and OOM kills]
    H3[Disk and inodes]
    H4[Network throughput]
  end

The layers exist for attribution, not for completeness. When latency rises, the question is which layer: the app got slower, the container hit its CPU quota, the daemon is wedged, or the host is swapping. One layer of metrics cannot answer that. Four can.

Failure-to-signal map

This is the table to keep. Everything after it is detail.

Read the right-hand column twice. Each entry is a signal that a reasonable person watches and that stays reassuringly green through the failure.

Container layer, in detail

Restart count

A crash-looping container is the most common container failure and the easiest to miss, because a container that restarts in two seconds is “running” every time you look at it.

$ docker ps -a --format '{{.Names}}' | while read -r c; do
echo "$c $(docker inspect -f '{{.RestartCount}}' "$c") $(docker inspect -f '{{.State.Status}}' "$c")"
done

web 0 running
worker 47 running
redis 0 running
legacy-batch 3 exited

worker has restarted 47 times and is currently up. Nothing in docker ps says so, and nothing in a CPU graph says so either — each incarnation is young, uses little, and dies before it registers on a 15-second scrape.

Two properties of RestartCount matter for alerting:

• It counts restarts performed by the restart policy, not restarts you performed. A container you docker restart by hand does not necessarily advance it.
• It is not monotonic across a container’s whole life the way a Prometheus counter is. Treat it as a gauge and alert on the increase over a window, not on the absolute value — otherwise a container that crash-looped once six months ago alerts forever.

The alert: restart count increased by 3 or more in 15 minutes. Three, because Docker’s own on-failure default retry behaviour and a single bad deploy both produce one or two; a third is a pattern. Fifteen minutes, because Docker’s restart policy backs off between attempts, so a tighter window can miss a slow crash loop entirely.

The false positive: a rolling deploy. Every container is replaced, and if your alert keys on container name rather than container ID it can read a replacement as a restart. Silence container-restart alerts for the duration of a deploy, or key the alert on the container ID label, which changes when the container is replaced and stays constant when it restarts.

OOM kills

$ docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' worker

137 true 

There is a second trap underneath the first. .State.OOMKilled is set when the container’s main process was the OOM victim. A container running a supervisor, a shell wrapper, or any process that forks workers can have a child killed by the cgroup OOM killer while PID 1 survives. The container stays up, the application quietly loses a worker, and .State.OOMKilled remains false.

For that case, watch the cgroup counter directly. The kernel keeps a running tally per cgroup, and cAdvisor exports it as container_oom_events_total. Alert on any increase in that counter — there is no benign OOM kill — and treat it as separate from the container-died alert, because it fires in cases where nothing died.

The alert: container_memory_working_set_bytes over container_spec_memory_limit_bytes above 0.9 for 10 minutes. Ninety percent, because the kernel begins reclaiming aggressively well before the limit and the last ten percent buys you roughly one alert cycle to act. Ten minutes, because a JVM or Go heap routinely touches its ceiling for seconds before a garbage collection returns it.

The false positive: a container with no memory limit set. With no limit, container_spec_memory_limit_bytes reports a sentinel value derived from host memory rather than a meaningful ceiling, and the ratio becomes nonsense. Exclude unlimited containers from the alert and raise a separate, non-paging finding for them — “container has no memory limit” is a real defect, but it is a change-management problem, not a 03:00 problem.

Healthcheck state transitions

A healthcheck has three states: starting, healthy, unhealthy. The transitions between them are the signal, and they are emitted as events.

$ docker events --filter event=health_status --format '{{.Time}} {{.Actor.Attributes.name}} {{.Status}}'

1786533012 api health_status: unhealthy
1786533072 api health_status: healthy
1786533132 api health_status: unhealthy

The daemon counts healthchecks for you, and the counter is one of the few Docker engine metrics whose name is stable and genuinely useful: engine_daemon_health_checks_failed_total, alongside engine_daemon_health_checks_total.

docker inspect gives per-container detail, including the field that distinguishes a flap from an outage:

$ docker inspect -f '{{.State.Health.Status}} streak={{.State.Health.FailingStreak}}' api

unhealthy streak=7

FailingStreak counts consecutive failures and resets on any success. A container flapping between healthy and unhealthy never accumulates a streak, which is exactly the distinction you want: a streak of 7 is a service that is down, and a flap is a service that is overloaded or a healthcheck with too tight a timeout.

The alert: state is unhealthy continuously for 2 minutes, or equivalently a failing streak beyond the container’s configured --health-retries. Two minutes rather than instantly, because --health-start-period does not always cover a slow start after a host reboot when every container competes for I/O at once.

The false positive: the healthcheck itself is the failure. A --health-cmd that runs curl against an endpoint which does a database round-trip will report unhealthy during any database blip, and a healthcheck with a 1-second interval on a busy host will time out under load and report a perfectly healthy service as broken. When a healthcheck alert fires, check whether the healthcheck can pass by hand before you touch the application.

CPU, and why the percentage misleads

Docker layer

Image pull failures

There is no engine metric for failed pulls. This is worth stating plainly rather than inventing one, because a pull failure is a very common deploy failure — expired registry credentials, a rate limit, a tag that was deleted, a proxy that started intercepting TLS.

What you have instead:

# Successful pulls in the last day
docker events --since 24h --until 1s --filter type=image --filter event=pull \
--format '{{.Time}} {{.Actor.ID}}'

# Failures live in the daemon log, not the event stream
sudo journalctl -u docker --since '24 hours ago' \
| grep -iE 'error pulling|failed to (pull|resolve)|toomanyrequests|unauthorized'

The asymmetry is the lesson: the event stream records what happened, and a pull that failed did not happen. Any monitoring built purely on docker events is blind to the entire class of “the thing never started”.

The practical alert is therefore indirect and better: a container that should exist does not exist, expressed as an absence of the expected container_last_seen series from cAdvisor for a named container. Absence is awkward in Prometheus and worth the trouble, because it catches pull failures, image-not-found, and a Compose file that was never applied, all with one rule.

Disk: split /var/lib/docker before you alert on it

“Disk above 90%” tells you there is a problem. It does not tell you whether to prune images, rotate logs, or find the volume a developer filled — and those have very different blast radii. Split it first.

$ docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          38        6         41.2GB    27.9GB (67%)
Containers      11        6         2.1GB     1.4GB (66%)
Local Volumes   14        7         88.4GB    31.0GB (35%)
Build Cache     221       0         19.7GB    19.7GB

Four rows, four different remedies, four different risks:

docker system df -v expands each row: per-image SHARED SIZE and UNIQUE SIZE, per-volume LINKS and SIZE. UNIQUE SIZE is the only number that answers “how much do I get back if I delete this image”, and it is usually a small fraction of SIZE because base layers are shared.

The alert set worth having, in priority order:

Inode exhaustion deserves its own row because it produces the most confusing symptom in this whole lesson: df -h shows plenty of space, and every write fails with “No space left on device”. Check df -i before you believe df -h.

Daemon health

The daemon exposes Prometheus metrics when you set metrics-addr in daemon.json; the next lesson wires that up. For this lesson the point is narrower and important:

A responsive /metrics endpoint is not proof the daemon works. The metrics handler is served by an HTTP listener that can answer perfectly while the parts of the daemon that create containers are blocked on a deadlocked storage driver or an unresponsive containerd. The daemon process is alive, the scrape succeeds, every dashboard is green, and docker run hello-world hangs forever.

The test that actually distinguishes them is an end-to-end one:

if timeout 10 docker version --format '{{.Server.Version}}' >/dev/null 2>&1; then
echo 'OK: daemon answered within 10s'
else
echo 'FAIL: daemon did not answer - check journalctl -u docker'
fi

Ten seconds is deliberate. A healthy daemon answers docker version in milliseconds; a daemon under heavy image load may take one or two seconds; a wedged daemon never answers at all. Anything above ten seconds is a fault whatever the cause.

Knowledge check