cAdvisor Metrics

A container is slow. The host CPU panel says 30 percent. The on-call engineer opens a Grafana panel, types rate(container_cpu_usage_seconds_total{name="checkout"}[5m]), and sees 0.7 cores. Then types rate(container_cpu_cfs_throttled_periods_total\{name="checkout"\}[5m]) / rate(container_cpu_cfs_periods_total\{name="checkout"\}[5m]) and sees 0.42. The container is using 0.7 cores and being throttled 42 percent of the time. The kernel is taking CPU away from it because the cgroup has hit its quota. The host has plenty of headroom. The fix is to raise the container’s CPU quota, not to add hosts.

This lesson is about the metrics that make that diagnosis possible: the container_* namespace produced by cAdvisor, the mental model behind each metric, and the cgroup driver details that change the underlying data without changing the metric names.

What it is

The cAdvisor metric namespace is container_*. The metrics are the canonical per-container counters and gauges for CPU, memory, network, filesystem, and cgroup health. They are derived directly from the cgroup hierarchy and the kernel, with no sampling and no application cooperation. The namespace is large but the operationally important subset is small:

• CPU: container_cpu_usage_seconds_total, container_cpu_cfs_throttled_periods_total, container_cpu_cfs_periods_total, container_spec_cpu_quota, container_spec_cpu_period.
• Memory: container_memory_working_set_bytes, container_memory_usage_bytes, container_memory_failcnt, container_memory_mapped_file, container_spec_memory_limit_bytes, container_oom_events_total.
• Network: container_network_receive_bytes_total, container_network_transmit_bytes_total, container_network_receive_packets_total, container_network_transmit_packets_total, container_network_receive_errors_total.
• Filesystem: container_fs_usage_bytes, container_fs_limit_bytes, container_fs_inodes_free.
• Lifecycle: container_last_seen, container_start_time_seconds.

The metric names are the contract. A Grafana panel written against these names will work on every cAdvisor version that exports them, and the canonical subset has been stable since the v0.20 line.

Why a sysadmin cares

These are the metrics an on-call engineer reaches for first on a container incident. They are the only metrics that answer three questions that nothing else answers:

• Is this container CPU-starved? The throttling ratio is the only signal of CPU starvation that is invisible to the host view.
• Is this container about to OOM? The working set relative to the cgroup limit is the only signal that matters; usage is contaminated by page cache.
• Is this container’s network usage normal? The per-container network counters are the only signal that lets you blame a specific container for traffic growth.

Every container dashboard that earns its place contains at least the CPU, memory, network, and lifecycle subset. A team that ships without these metrics cannot answer “is this container the one that is slow” in under ten minutes.

How it works

The mental model: every container is a cgroup, every cgroup is a directory under /sys/fs/cgroup, and every metric in the container_* namespace is a derivative of one or more files in that directory.

    /sys/fs/cgroup/system.slice/docker-<id>.scope/
        cpu.max                 -> container_spec_cpu_quota,
                                  container_spec_cpu_period
        cpu.stat                -> container_cpu_usage_seconds_total,
                                  container_cpu_cfs_throttled_periods_total,
                                  container_cpu_cfs_periods_total
        memory.current          -> container_memory_usage_bytes
        memory.events           -> container_oom_events_total
        memory.swap.current     -> (unused, swap accounting)
        memory.stat             -> container_memory_working_set_bytes
                                  (total - cache - inactive_anon)
                                  container_memory_mapped_file
        pids.current            -> (not exposed)
        io.stat                 -> container_fs_usage_bytes
                                  (via the underlying block device)

cAdvisor polls this directory every housekeeping interval (default 15 s), reads the counter files, computes the deltas, and serves the result as a Prometheus exposition. The counters in cAdvisor are therefore poll-based, not push-based; if Prometheus misses a scrape, the counter is incremented by the lost interval on the next scrape.

How to configure it

There is nothing to configure inside the metrics themselves; the metric names and labels are fixed by cAdvisor. The configuration that matters is the Prometheus scrape job, the metric relabel rules that drop uninteresting series, and the recording rules and alert rules that use the metrics.

Scrape job

scrape_configs:
  - job_name: cadvisor
    scrape_interval: 30s
    scrape_timeout: 25s
    static_configs:
      - targets: ['127.0.0.1:8080']
        labels:
          layer: container
    metric_relabel_configs:
      # Drop every cgroup that is not a Docker container.
      # cAdvisor publishes every cgroup on the host; systemd
      # slices, the root cgroup, and machine.slice all arrive
      # with an empty name label.
      - source_labels: [name]
        regex: ''
        action: drop
      # Drop the per-interface network series for the loopback.
      - source_labels: [interface]
        regex: 'lo'
        action: drop
      # Keep the canonical container_* subset; drop the rest.
      # The list below is the operationally important subset.
      - source_labels: [__name__]
        regex: 'container_(cpu_usage_seconds_total|cpu_cfs_throttled_periods_total|cpu_cfs_periods_total|spec_cpu_quota|spec_cpu_period|memory_working_set_bytes|memory_usage_bytes|memory_failcnt|memory_mapped_file|spec_memory_limit_bytes|oom_events_total|network_receive_bytes_total|network_transmit_bytes_total|network_receive_packets_total|network_transmit_packets_total|network_receive_errors_total|fs_usage_bytes|fs_limit_bytes|fs_inodes_free|last_seen|start_time_seconds)'
        action: keep

Severity: CONFIGURATION. Reload Prometheus to apply.

The metric_relabel_configs are the operational lever. Without the first two rules, cAdvisor’s metric volume is dominated by systemd units and the loopback interface. Without the third, the Prometheus TSDB grows by the full set of metrics cAdvisor publishes, which is hundreds of names and a number of series that scales with the number of cgroups on the host.

Alert rules

groups:
  - name: cadvisor-containers
    rules:
      - alert: ContainerCPUThrottled
        expr: |
          rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m])
            / rate(container_cpu_cfs_periods_total{name!=""}[5m]) > 0.25
        for: 10m
        labels: { severity: ticket }
        annotations:
          summary: 'Container throttled in over 25% of scheduling periods'

      - alert: ContainerMemoryNearLimit
        expr: |
          container_memory_working_set_bytes{name!=""}
            / container_spec_memory_limit_bytes{name!=""} > 0.9
          and container_spec_memory_limit_bytes{name!=""} > 0
        for: 10m
        labels: { severity: ticket }
        annotations:
          summary: 'Working set above 90 percent of the cgroup limit'

      - alert: ContainerOOMKilled
        expr: increase(container_oom_events_total{name!=""}[10m]) > 0
        for: 0m
        labels: { severity: page }
        annotations:
          summary: 'OOM kill inside container cgroup'

      - alert: ContainerRestartLoop
        expr: changes(container_start_time_seconds{name!=""}[15m]) >= 3
        for: 0m
        labels: { severity: ticket }
        annotations:
          summary: 'Container restarted 3+ times in 15 minutes'

Severity: CONFIGURATION. Reload Prometheus to apply.

The CPU throttling rule is the only signal that the host view cannot supply. The memory-near-limit rule uses container_memory_working_set_bytes, not container_memory_usage_bytes, because the latter includes page cache. The guard and container_spec_memory_limit_bytes > 0 is the guard against the false positive from containers with no limit; cAdvisor reports a value derived from host memory for them.

How to validate it

Five checks, cheapest first.

# READ-ONLY: the canonical subset is present.
curl -fsS http://127.0.0.1:8080/metrics \
  | grep -E '^container_(cpu_usage_seconds_total|memory_working_set_bytes|network_receive_bytes_total)' \
  | head -6
# container_cpu_usage_seconds_total{name="checkout",id="..."} 184.31
# container_memory_working_set_bytes{name="checkout",id="..."} 5.84e+08
# container_network_receive_bytes_total{name="checkout",interface="eth0",id="..."} 4.12e+08

# READ-ONLY: the working set is sane.
curl -fsS http://127.0.0.1:8080/metrics \
  | grep '^container_memory_working_set_bytes{name="checkout"' \
  | awk '{ print $2 }'
# 5.84e+08
# (working set < container_memory_usage_bytes for the same container)

# READ-ONLY: CPU usage as a rate.
curl -fsS http://prometheus.internal:9090/api/v1/query \
  --data-urlencode \
    'query=rate(container_cpu_usage_seconds_total{name="checkout"}[5m])'
# {"status":"success","data":{"resultType":"vector","result":[{"value":[1734259200,"0.7"]}]}}

# READ-ONLY: CPU throttling ratio.
curl -fsS http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=rate(container_cpu_cfs_throttled_periods_total{name="checkout"}[5m])
                            / rate(container_cpu_cfs_periods_total{name="checkout"}[5m])'
# {"status":"success","data":{"resultType":"vector","result":[{"value":[1734259200,"0.42"]}]}}

# READ-ONLY: a Prometheus rules check.
promtool check rules /etc/prometheus/rules/cadvisor.yml

A clean validation: the canonical subset is exposed with non-empty labels, the working set is smaller than the total memory usage, and rate() plus the throttling ratio compute correctly.

How it can fail

1. container_memory_working_set_bytes rises but the application reports no growth. Cause: the kernel is promoting anonymous pages to inactive; cAdvisor’s computation total - inactive_file - inactive_anon_5 is sensitive to the age of the inactive list. This is rarely noise; it usually means the application genuinely retained more pages. Cross-check with the application’s own heap metric.
2. The CPU throttling ratio is high but rate(container_cpu_usage_seconds_total) is low. Cause: the cgroup quota is set very low (--cpus=0.1 for example) and the container is hitting it without saturating it. The fix is to raise the quota; the alert is correct.
3. container_memory_failcnt rises but the alert does not fire. Cause: cAdvisor reports failcnt as a counter of times the limit was hit; on cgroup v2 the counter resets on memory.high events but not on memory.max. Use container_oom_events_total for the alert and treat failcnt as a diagnostic.
4. container_network_* reports zero for every container. Cause: cAdvisor cannot enter the network namespace of the container; --privileged was dropped or the seccomp profile blocks setns(2). Detection: container_network_* is absent for non-empty name labels.
5. The scrape takes seconds and Prometheus reports a timeout. Cause: scrape_timeout is below the time cAdvisor needs to walk every cgroup and emit metrics. Raise it; the metric volume is bounded by --disable_metrics and the cgroup count.
6. Per-container metrics exist but labels are empty for specific images. Cause: the image uses a base that does not write to /etc/os-release, or the image sets container_label_* to empty. Cross-check with docker inspect <id>. The fix is at the image level, not in cAdvisor.

How to troubleshoot it

1. Are the canonical series present? curl -fsS http://127.0.0.1:8080/metrics | grep -c '^container_cpu_usage_seconds_total'. A non-zero count confirms cAdvisor is publishing per-container CPU.
2. Are the working-set values sane? For a quiet container, working_set_bytes should be a fraction of usage_bytes. If they are equal, the metric_relabel rule has dropped the wrong series.
3. Does rate() work? The classic bug is forgetting that container_cpu_usage_seconds_total is a counter. Direct graphing shows a sawtooth. Wrap it in rate() to get a sample per second.
4. Does Prometheus see the target? up{job="cadvisor"} should be 1. A 0 here is a Prometheus problem, not a cAdvisor problem.
5. Does promtool check rules pass? Run it on every alert rule. A typo in a label name is caught at compile time; the cost of running it once is minutes.

Security implications

• Per-container labels can carry sensitive values. cAdvisor exposes container_label_* for every Docker label set on the container. A label that contains a registry credential or an internal URL is scraped and stored in Prometheus. Audit the labels on production containers.
• The exposition endpoint is unauthenticated. cAdvisor’s /metrics endpoint publishes the container inventory with no authentication. The same posture as the lesson on cAdvisor overview applies: bind to 127.0.0.1 and let Prometheus reach it over loopback.
• Filesystem metrics reveal content. container_fs_usage_bytes reports the size of the writable layer, which can leak the shape of the data the container is producing. Treat the listener as sensitive.

Performance implications

• cAdvisor’s CPU cost scales with the cgroup count. A host with thousands of systemd units and --docker_only set is cheap; without --docker_only the cost grows.
• Prometheus’s CPU cost scales with the cardinality. A container_* rule with a missing name!="" guard multiplies by the number of systemd units on the host. The guards in the alert rules above are not optional.
• The metric_relabel keep rule bounds series growth. Without it, cAdvisor’s full metric set is several thousand series per container; with it, the operationally important subset is bounded.

Production guidance

• Alert on container_memory_working_set_bytes, not container_memory_usage_bytes. The latter includes page cache and produces false pages.
• Use the throttling ratio, not the absolute throttled seconds. The ratio is comparable across containers with different quotas.
• Always guard memory and CPU rules against containers with no limit. The and container_spec_memory_limit_bytes > 0 pattern is the standard guard.
• Pin cAdvisor’s metric set with a metric_relabel_configs keep rule. Drop the rest at the source.
• Cross-reference every container_* panel with the application exporter. The cgroup tells you what the kernel sees; the application tells you what the application sees. Both are needed.

Verification

You should now be able to answer:

• What is the difference between container_memory_working_set_bytes and container_memory_usage_bytes, and which should an alert be built on?
• How do you compute the CPU throttling ratio from the canonical container CPU metrics?
• Why is the guard and container_spec_memory_limit_bytes > 0 needed on the memory-near-limit rule?
• What changes between cgroup v1 and cgroup v2 in terms of the cAdvisor metric set?
• Which PromQL expression gives a container’s CPU usage in cores?

Quiz