Blind spots — what the monitoring stack cannot see

A monitoring stack has a coverage boundary, and it is not drawn where most teams assume. The dashboards are green because the queries return values; the queries return values because the things they measure are the things that get measured.

This lesson is about the other side of that boundary.

Containers that live between two scrapes

Prometheus samples on an interval, typically 15 or 30 seconds. A container that runs for eight seconds produces zero samples. A container that runs for forty produces one or two, at arbitrary points in its lifetime.

That covers a large and important class of workload:

• Backup jobs run from docker run --rm on a timer.
• Database migrations during a deploy.
• One-shot CLI containers in CI.
• Compose depends_on init containers that fix permissions and exit.
• Anything a cron job starts.

None of it appears in a CPU graph. A backup container that has been exiting with status 1 for three weeks has left no trace in the time-series database at all — no failed scrape, no alert, no gap in a chart, because there was never a series to have a gap in.

Two things fix this, and you need both:

docker events --filter 'event=die' \
--format '{{.Time}} {{.Actor.Attributes.name}} exit={{.Actor.Attributes.exitCode}}'

For jobs whose result matters, push a metric at the end of the run rather than hoping to be scraped during it. That is what the Pushgateway is for — batch jobs whose lifetime is shorter than the scrape interval — and it is the one legitimate use of it. Push the completion timestamp and the exit status, then alert on the timestamp going stale:

time() - push_time_seconds{job="nightly-backup"} > 90000

That rule fires when the backup has not completed in 25 hours, whether it failed, hung, or was never started because the timer was disabled during maintenance. Alerting on staleness catches the absence; alerting on a failure metric only catches the failures that ran.

The monitoring stack inside the failure domain

The default Docker monitoring tutorial puts Prometheus, Grafana and Alertmanager in a Compose stack on the host being monitored. It works perfectly until the day it is needed.

flowchart TB
  subgraph host01 [host01 — the thing being monitored]
    App[application containers]
    CAdv[cAdvisor]
    Node[node_exporter]
    Prom[Prometheus]
    AM[Alertmanager]
  end
  App --> CAdv --> Prom
  Node --> Prom
  Prom --> AM
  AM -->|page| Oncall[on-call]
  Prom -.->|dies with the host| X[ ]

When host01 runs out of memory, loses its disk, or is accidentally powered off, the containers stop — and so do the Prometheus that would have recorded why and the Alertmanager that would have told you.

The failure is silent by construction. Nothing pages, because the thing that pages is part of what failed.

An empty query is not a healthy query

This is the most common way a dashboard lies.

container_memory_working_set_bytes{name="postgres"}
  / container_spec_memory_limit_bytes{name="postgres"} > 0.9

If the postgres container has been deleted, renamed, or moved to another host, this expression returns no series. No series means the alert cannot fire. The panel shows “No data”, which on a busy dashboard reads as a rendering glitch.

Every rule that matches on a specific target needs a companion that asserts the target exists:

- alert: PostgresTargetMissing
  expr: absent(container_last_seen{name="postgres"})
  for: 5m
  labels:
    severity: page

The same applies at the scrape layer. up is 1 when a scrape succeeded and 0 when it failed — but there is no up series at all for a target that was removed from the configuration:

- alert: ExporterDown
  expr: up{job=~"cadvisor|node|docker"} == 0
  for: 5m
- alert: ExporterTargetVanished
  expr: absent(up{job="cadvisor"})
  for: 10m

The exporters are not free

cAdvisor walks every cgroup on the host on each housekeeping cycle. On a host with a handful of containers this is invisible; on a host with hundreds it is a measurable and occasionally surprising CPU consumer.

The flags worth knowing:

Raising --housekeeping_interval to match your scrape interval removes work whose output is discarded anyway — there is little value in gathering stats every second when Prometheus samples every fifteen.

Cardinality deserves the same scrutiny. Every distinct label combination is a separate time series, and containers are a cardinality generator: each docker run --rm creates a new container ID, a new set of series, and permanent churn in the index. A CI host running a thousand short-lived containers a day will degrade a Prometheus that a production host of the same size does not.

Drop the labels you do not query, with metric_relabel_configs, at scrape time — before they are stored, not after.

What has no metric at all

Some things simply are not in the stack, and knowing the list stops you from trusting a green dashboard to cover them:

• Whether the running image is the image you built. That is a supply-chain question, answered by digests, not by metrics.
• Whether the backup restores. Covered by a restore test, and by nothing else.
• Whether the firewall rule survived the reboot. Covered by an external port scan.
• Whether logs are being rotated. Visible as disk growth, hours after it matters.
• Whether DNS resolves inside the container. The application’s error rate is the proxy, and it is a lagging one.

Each of these has a check elsewhere in this course. None of them is a Prometheus rule.

Sanity check