cAdvisor Overview

A checkout container has been using 92 percent of its memory limit for six hours. The on-call engineer opens the container dashboard in Grafana. The container panel reports the limit and the working set, both correctly labelled, both correctly scoped to the container ID. The host panel, if there were one, would show the host memory at 28 percent. The container is at 92 percent because its limit is small, and the application genuinely needs more than the limit. The diagnosis is one query, made possible because cAdvisor is running.

This lesson is about what cAdvisor is, what it actually measures, and how to run it standalone under Docker 28.x.

What it is

cAdvisor (Container Advisor) is Google’s open-source container metrics engine. It reads the cgroup hierarchy, the network namespace counters, and the per-container filesystem usage of every container on the host and serves the result as a Prometheus exposition endpoint on port 8080. It is the canonical answer to the question “what is this container doing right now?” and the only widely used exporter that produces a per-container working set, throttling ratio, and OOM-event counter.

cAdvisor is a deliberately small piece of software. It does not alert, it does not aggregate across hosts, and it does not store metrics. Its job ends when the metrics are served. Prometheus does the rest.

The two main ways to run it:

• Inside kubelet. Kubernetes bundles cAdvisor into kubelet on every node, so a Kubernetes host gets container metrics automatically at /metrics/cadvisor on the kubelet port. The Kubernetes path is out of scope for this course; the standalone-Docker path is the focus here.
• Standalone as a Docker container. The supported invocation on a non-Kubernetes host. This is what a Docker host running docker compose up uses, and it is what this lesson covers.

Why a sysadmin cares

The container metrics that matter for a Docker host are produced by cAdvisor. There is no honest alternative for the metric container_cpu_cfs_throttled_periods_total, no honest alternative for the metric container_memory_working_set_bytes, and no honest alternative for container_network_*_bytes_total per container.

Two production shapes put cAdvisor on the critical path:

• Throttling that nobody can explain. A container is slow and the host CPU panel says the host is at 30 percent. The answer is in container_cpu_cfs_throttled_seconds_total: the container is hitting its quota, the kernel is throttling it, and the host has plenty of headroom. cAdvisor is the only exporter that reports this number.
• Memory growth that nobody can explain. A container is using 92 percent of a memory limit that has not changed. cAdvisor’s working-set metric tells the on-call whether the growth is the application (working set rising), the page cache (only container_memory_usage_bytes rising), or the kernel accounting (only container_memory_failcnt rising).

How it works

The data flow is short and almost entirely read-only.

    /sys/fs/cgroup/*     /var/lib/docker/containers/*    /sys/class/net/*
            |                      |                            |
            +----------+-----------+------------+---------------+
                       |                        |
                       v                        v
                container handler        network / fs handler
                       |                        |
                       +-------+----------------+
                               v
                          cAdvisor
                          (Go binary, :8080)
                               |
                               v
                       Prometheus exposition
                       /metrics on :8080

cAdvisor does not speak to the Docker daemon directly. It reads the cgroup hierarchy directly (on cgroup v1 the per-cpu, per-memory, per-blkio files; on cgroup v2 the unified hierarchy under /sys/fs/cgroup), reads the network interface counters in /sys/class/net, and reads per-container filesystem usage from /var/lib/docker/containers/<id>/. The cgroup view is the authoritative source; the Docker socket is used only to translate the cgroup ID into a human-readable container name and image.

The output of cAdvisor is the container_* namespace in Prometheus. A Prometheus that scrapes cAdvisor sees per-container counters and gauges labelled by name, image, container_label_*, and id. The lesson on cAdvisor metrics covers the canonical ones in detail; the architectural lesson here is that the labels come from the cgroup itself plus the Docker container metadata, and they are stable across the lifetime of the container.

Why it is built into kubelet

In Kubernetes, kubelet is the per-node agent that owns the container runtime. cAdvisor is part of kubelet precisely because every node already has a kubelet and every node already needs container metrics for scheduling decisions (bin-packing, eviction, quality of service). The standalone Docker path is the same binary running outside kubelet, with the same metric names and the same exposure format. Nothing about the metric schema changes; only the deployment shape does.

The Kubernetes path is out of scope here. The rest of this lesson assumes you are running docker compose on a Linux host without a kubelet.

How to configure it

The canonical standalone invocation on Docker 28.x:

CADVISOR_VERSION=v0.55.1

docker run -d \
  --name cadvisor \
  --restart=unless-stopped \
  --publish 127.0.0.1:8080:8080 \
  --volume /:/rootfs:ro \
  --volume /var/run:/var/run:ro \
  --volume /sys:/sys:ro \
  --volume /var/lib/docker/:/var/lib/docker:ro \
  --volume /dev/disk/:/dev/disk:ro \
  --device /dev/kmsg \
  --privileged \
  "ghcr.io/google/cadvisor:${CADVISOR_VERSION}" \
  --housekeeping_interval=15s \
  --docker_only \
  --disable_metrics=advtcp,app,cpu_topology,cpuset,hugetlb,memory_numa,network_numa,perf_event,process,resctrl,sched,referenced_memory,sched_debug_tcp,system,cpu_load_average,cpu_frequency,cpuset_cpus,cpuset_mems,hugetlb_usage_bytes,network_interfaces_tcp,network_tcp_usage_total,network_udp_usage_total

Severity: CONFIGURATION. Restart the container to apply.

Walk through the important flags:

• --publish 127.0.0.1:8080:8080 binds only to the loopback. The exporter publishes the entire container inventory with no authentication. A listener on 0.0.0.0 hands that inventory to anyone who can reach the host. Loopback plus a reverse proxy or a firewall rule is the production posture.
• --privileged is required by cAdvisor’s documented invocation. It needs setns(2) to enter the network namespace of each container, read access to /dev/kmsg to see kernel OOM messages, and walk the cgroup hierarchy across every controller. Several of these are blocked by the default seccomp and capability sets.
• --volume /:/rootfs:ro and /sys:/sys:ro give cAdvisor read access to the host filesystem tree. The ro flag is deliberate: a writable mount would be a privilege escalation path.
• --volume /var/lib/docker/:/var/lib/docker:ro lets cAdvisor map cgroup IDs to container names and images. Without it, every container arrives with an empty name label.
• --device /dev/kmsg is the path through which the kernel exposes OOM and panic messages. cAdvisor needs it to record host-level OOM kills against the right container.
• --housekeeping_interval=15s sets the cgroup polling rate. Lower values are expensive; higher values miss short container lifetimes.
• --docker_only tells cAdvisor to publish series only for containers cAdvisor recognises as Docker containers. Without it, every systemd unit on the host arrives with an empty name label and the Prometheus TSDB grows by the number of systemd units on the machine.
• --disable_metrics turns off collectors that are empty or unreliable on a typical server. The list above is the upstream-recommended set for a production Docker host.

How to validate it

Three checks, cheapest first.

# READ-ONLY: the container is running.
docker ps --filter name=cadvisor \
  --format '{{.Names}} {{.Status}} {{.Image}}'
# cadvisor   Up 14 minutes   ghcr.io/google/cadvisor:v0.55.1

# READ-ONLY: the metrics endpoint answers.
curl -fsS http://127.0.0.1:8080/metrics | head -5
# HELP cadvisor_version_info Information about the version of cAdvisor.
# TYPE cadvisor_version_info gauge
# cadvisor_version_info{version="v0.55.1",dockerVersion="28.x",kernelVersion="..."} 1

# READ-ONLY: a per-container CPU series exists.
curl -fsS http://127.0.0.1:8080/metrics \
  | grep '^container_cpu_usage_seconds_total' \
  | head -3
# container_cpu_usage_seconds_total{name="checkout",id="..."} 184.31
# container_cpu_usage_seconds_total{name="postgres",id="..."} 91.02

# READ-ONLY: Prometheus sees the target.
curl -fsS http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=up{job="cadvisor"}'
# {"status":"success","data":{"resultType":"vector","result":[{"value":[1734259200,"1"]}]}}

A clean validation: the container is Up, /metrics returns the cadvisor_version_info series, the container_cpu_usage_* series exist with non-empty name labels, and up{job="cadvisor"} is 1. Each failure mode below maps to one of these signals failing.

How it can fail

1. No per-container series; only the host root cgroup appears. Cause: --docker_only is not set, or the bind mount to /var/lib/docker/ is missing. Every systemd unit on the host arrives with an empty name label and Prometheus stores them all. Detection: count(container_last_seen{name=""}) is in the thousands.
2. The metrics endpoint returns 200 but every metric has missing labels. Cause: --privileged was dropped, or the seccomp profile blocked setns(2). cAdvisor runs but cannot enter the network namespace of each container. Detection: container_network_* is absent.
3. Scrape timeouts. Cause: --housekeeping_interval is lower than 15 s, the host has many cgroups, and the scrape takes longer than scrape_timeout. Detection: scrape_duration_seconds{job="cadvisor"} is greater than scrape_timeout.
4. The container refuses to start. Cause: the daemon was started with a seccomp or apparmor profile that blocks setns(2) and mount. cAdvisor fails immediately on startup. Detection: docker logs cadvisor shows an operation not permitted error.
5. container_memory_failcnt is non-zero for every container. Cause: this is not a failure but a signal. The fail counter increments each time a container hits its memory limit; non-zero means the host or the cgroup has applied memory pressure. Detection: alert on rate(container_memory_failcnt[5m]) > 0.
6. The bind mount /var/lib/docker shows a different container set than docker ps. Cause: the graphdriver path differs from /var/lib/docker. On a rootless Docker install the path is typically ~/.local/share/docker/. Detection: the bind mount on the host does not show the expected overlay2/ directory.

How to troubleshoot it

1. Is the container running? docker ps --filter name=cadvisor. If it is Restarting, read docker logs cadvisor; the most common reason for a restart loop is the setns(2) denial.
2. Does the metrics endpoint answer? curl -fsS http://127.0.0.1:8080/metrics. A 200 confirms the listener is bound.
3. Are the per-container series present? curl -fsS http://127.0.0.1:8080/metrics | grep '^container_' and check that name= is non-empty on most lines. Empty name= means --docker_only is not set or the /var/lib/docker/ bind is missing.
4. Is Prometheus scraping it? up{job="cadvisor"} should be 1. If up is 0, the Prometheus configuration is wrong; cAdvisor is fine.
5. Are the metrics dropping? If the series count for container_last_seen drops to zero for a container that is clearly running, the cgroup has been removed and recreated; the container’s cgroup ID has changed. This happens on docker engine restart.

Security implications

• Privileged container. cAdvisor needs setns(2) and read access to the entire filesystem tree. The mitigation is network isolation, not a stripped-down invocation: the metrics cannot be collected without these capabilities, and a partial grant produces a silently incomplete dataset.
• Unauthenticated exposition. cAdvisor publishes the entire container inventory (names, image references, cgroup IDs) on port 8080 with no authentication. Bind to 127.0.0.1 and let Prometheus reach it over loopback or a dedicated bridge network.
• cgroup visibility. A reader of cAdvisor’s /metrics can enumerate every container on the host and infer the workload shape. Treat the listener as sensitive and firewall it.
• Image references in labels. cAdvisor exposes the image reference as a label value. If the image reference contains a registry credential (it should not), the credential is scraped and stored in Prometheus. Audit your image tags.

Performance implications

• The cgroup walk. cAdvisor polls every cgroup on the host every housekeeping interval. On a host with thousands of systemd units and --docker_only set, the walk is cheap. On a host without --docker_only the walk scales with the number of systemd units, and a 5-second housekeeping interval can push the exporter to a noticeable fraction of a CPU.
• The network namespace probe. cAdvisor enters each container’s network namespace to read its counters. The cost is one setns(2) per container per housekeeping interval; on a host with hundreds of containers this is a few hundred syscalls per scrape, which is negligible.
• Scrape duration. cAdvisor’s /metrics response time scales with the number of containers and the number of metrics kept enabled. On a host with hundreds of containers the response can take several seconds; scrape_timeout must be sized accordingly.

Production guidance

• Pin the version. :latest on a privileged container is an unreviewed image gaining host-level access on the next pull.
• Set --docker_only. Without it cAdvisor publishes every cgroup on the host and the Prometheus TSDB grows by the number of systemd units.
• Bind to 127.0.0.1 and let Prometheus scrape over loopback or a dedicated bridge network.
• Scrape cAdvisor on a 30-second interval with a 25-second timeout; the default 15-second interval is fine for a small host and tight on a busy one.
• Drop the metric_relabel_configs filter for empty name labels in the Prometheus job; this is your second line of defence if cAdvisor configuration drifts.
• Audit the disabled-metrics list. The list in this lesson is the upstream-recommended baseline; add or remove based on what your dashboards actually use.

Verification

You should now be able to answer:

• What three sources does cAdvisor read from, and what does each contribute?
• Why is cAdvisor built into kubelet, and what changes when it runs standalone under Docker?
• Why is --docker_only important on a non-Kubernetes host?
• What capability does cAdvisor need that requires --privileged?
• Which Prometheus metric confirms that cAdvisor is mapping cgroups to Docker container names?

Quiz