Short-Lived Container Metrics

A nightly batch container starts at 02:00 and exits at 02:04. It migrates 1.4 million records and writes a summary to its log. The on-call engineer opens the container dashboard at 09:00 the next morning and sees nothing: no CPU series, no memory series, no network series. The container is gone; cAdvisor dropped its metrics when the cgroup was removed; Prometheus never had a chance to scrape.

The job ran. The metrics are lost. The team has no record of how long the migration actually took, how much memory it consumed, or whether the migration had been slowly getting worse over the last three weeks. The lesson is the canonical failure mode of pull-based scrape and the patterns that recover from it.

What it is

A short-lived container is one whose lifetime is shorter than the scrape interval of the Prometheus that should observe it. The canonical Prometheus scrape interval is 15 s. A container that lives for 12 s is, statistically, missed by half of all scrapes and never appears in any panel.

Three workloads hit this shape:

• Batch jobs. A nightly migration, an hourly report, a five-minute cron-driven reconciliation.
• CI pipelines. A test runner that starts, runs, and exits inside a job lifetime.
• Crash loops. A container that restarts faster than the scrape interval. Every restart leaves a gap.

The patterns that recover from the gap are not interchangeable. Each has a cost the team accepts in exchange for the metric.

Why a sysadmin cares

The metrics of a finished job are the record of what the job actually did. Without them the team cannot answer “is the migration getting slower over time”, “did the test runner’s p99 latency regress”, or “did this container actually run at all”.

The default Prometheus model is pull. Pull works for long-lived services because Prometheus knows where they are and they are running on the next scrape. For short-lived containers pull fails silently: the scrape target list contains the container’s cgroup, the scrape runs, the container is gone, and the scrape returns no data. There is no error; the scrape target is just absent from the next response.

How it works

The data flow for a long-lived container is well understood: cAdvisor exposes, Prometheus scrapes, the series lives for the container’s lifetime. The data flow for a short-lived container depends on the lifetime relative to the scrape interval.

    long-lived container
    |-------|-------------------|---------|
       scrape          scrape        scrape
                  metric series created
                  and retained for the lifetime

    short-lived container, scrape interval > lifetime
    |--|
    no scrape occurred; the cgroup is removed on exit;
    cAdvisor drops the in-memory state; Prometheus never
    sees a sample.

cAdvisor itself is not the failure point. cAdvisor polls the cgroup at the housekeeping interval (default 15 s), so any container that lives for at least one polling cycle has at least one sample. The failure is between cAdvisor and Prometheus: Prometheus never scraped the sample, and cAdvisor drops the in-memory state when the cgroup is removed.

The two patterns that recover from the gap:

1. Pushgateway. The container pushes its metrics to an intermediary that holds them until Prometheus scrapes. The canonical tool. Has caveats that are not optional.
2. Application instrumentation that survives container exit. The container writes its metrics to a stable location (a file, a Loki stream, a remote OTLP endpoint) before exit. The container exits; the metric remains.

How to configure it

Three configuration shapes cover most production needs. The shape you choose depends on what the container is doing.

Pattern A: pre-stop hook that pushes final metrics

# docker-compose.yml
services:
  migrate:
    image: registry.example.com/migrations:v1.4.0
    command: ['./run-migration.sh']
    labels:
      prometheus.job: "migration"
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://pushgateway:9091"
      OTEL_RESOURCE_ATTRIBUTES: "service.name=migration,service.instance=nightly"
    stop_grace_period: 30s

# push the final metric on container stop, before the main
# process exits.
docker run --rm -d --name migrate \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://pushgateway:9091 \
  registry.example.com/migrations:v1.4.0
# wait for the container to exit
docker wait migrate
# the OTel SDK inside the container pushes a final summary
# metric on context cancellation

Severity: CONFIGURATION. The compose file is the durable configuration.

The pattern is to instrument the application with the OpenTelemetry SDK and push the final metric on context cancellation. The container’s lifecycle exit triggers a context cancel; the SDK flushes its pending metrics before the process terminates.

Pattern B: sidecar with a stable metrics endpoint

# docker-compose.yml
services:
  worker:
    image: registry.example.com/worker:v2.3.1
    command: ['./run-worker.sh']
    labels:
      prometheus.port: "9100"

  push-sidecar:
    image: ghcr.io/prometheus/pushgateway:v1.10.0
    command:
      - '--persistence.file=/data/pushgateway.data'
      - '--persistence.interval=5m'
    volumes:
      - pushgateway-data:/data
    restart: unless-stopped

Severity: CONFIGURATION. Compose up to apply.

The pattern is to run a long-lived pushgateway container and have short-lived workers push to it. The pushgateway persists across worker lifetimes; the workers push on exit. Prometheus scrapes the pushgateway on its normal schedule.

Pattern C: cAdvisor alone, with longer Prometheus retention of the lifecycle signal

# prometheus.yml
scrape_configs:
  - job_name: cadvisor
    scrape_interval: 5s      # tighter than the shortest lifetime
    scrape_timeout: 5s
    static_configs:
      - targets: ['127.0.0.1:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'container_(start_time_seconds|last_seen)'
        action: keep

Severity: CONFIGURATION. Prometheus reload to apply.

The pattern is to keep cAdvisor’s lifecycle signal even after the container exits. The container_last_seen and container_start_time_seconds metrics persist briefly after the cgroup is removed, and Prometheus retains them for its configured stale interval. This pattern does not recover the consumption metrics, only the lifecycle evidence.

Prometheus scrape of the Pushgateway

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway:9091']
    metric_relabel_configs:
      # Pushgateway pushes labels verbatim; honour them and
      # do not rewrite job to "pushgateway".
      - source_labels: [__name__]
        regex: 'go_.*|process_.*|pushgateway_.*'
        action: drop

Severity: CONFIGURATION. Prometheus reload to apply.

honor_labels: true is essential. Without it, Prometheus rewrites the pushed job and instance labels to the Pushgateway job and the pushed metrics become indistinguishable from the Pushgateway’s own self-metrics.

How to validate it

# READ-ONLY: a synthetic push to the Pushgateway.
echo "migration_duration_seconds 145.2" \
  | curl --data-binary @- http://pushgateway:9091/metrics/job/migration/instance/nightly
# (no output on success)

# READ-ONLY: Prometheus sees the pushed metric.
curl -fsS http://prometheus.internal:9090/api/v1/query \
  --data-urlencode 'query=migration_duration_seconds'
# {"status":"success","data":{"resultType":"vector","result":[{"value":[1734259200,"145.2"]}]}}

# READ-ONLY: cAdvisor's lifecycle metric for a container that has exited.
curl -fsS http://127.0.0.1:8080/metrics \
  | grep '^container_last_seen{' | head -5
# container_last_seen{name="...",id="..."} 1.734e+09

# READ-ONLY: a Prometheus rules check.
promtool check rules /etc/prometheus/rules/pushgateway.yml

A clean validation: the synthetic push lands in the Pushgateway and in Prometheus, the lifecycle metric is visible briefly after exit, and the rules file passes promtool check rules.

How it can fail

1. The Pushgateway is down when the worker pushes. Cause: the worker is a Docker service with no health check on the Pushgateway; the push fails silently because the SDK swallowed the error. Detection: the worker logs show a connection refused at exit time.
2. Two workers push with the same job and instance labels. Cause: the labels are templated from a shared file and not unique per worker. The second push replaces the first. Detection: only one worker’s metrics appear in Prometheus.
3. Pushed metrics never expire. Cause: no --persistence.file and no --push.disable-consistency job to expire the metric after the batch completes. The metric lives forever and the dashboard shows a stale value. Detection: migration_duration_seconds still reports the value from the last successful run days after the run.
4. cAdvisor polling misses the container. Cause: the container’s lifetime is shorter than the housekeeping interval. The metric is empty. Detection: the container_last_seen series never appears for the container.
5. Prometheus retention is shorter than the batch interval. Cause: the local Prometheus has a 5-day retention and the batch runs weekly; the metric expires before the trend is visible. Detection: the trend panel in Grafana has gaps.
6. The Pushgateway is the only path and it is a single instance. Cause: the team has not deployed HA. A Pushgateway restart with no persistence loses every pushed metric. Detection: any restart drops metrics.

How to troubleshoot it

1. Is the Pushgateway up? curl -fsS http://pushgateway:9091/-/ready. A 200 confirms it is ready.
2. Was the push accepted? curl -fsS http://pushgateway:9091/metrics | grep <metric_name>. The pushed metric appears immediately.
3. Is Prometheus scraping the Pushgateway? up{job="pushgateway"} should be 1. A 0 is a Prometheus job problem.
4. Are the labels correct? curl -fsS http://pushgateway:9091/metrics | grep '^migration'. The metric should carry the pushed job and instance labels, not the Pushgateway’s own.
5. Did the worker exit before pushing? docker logs <worker> should show a “metrics flushed” log line on exit. If it does not, the SDK never had a chance to flush.

Security implications

• The Pushgateway is unauthenticated by default. Anyone who can reach port 9091 can push any metric under any labels. Treat the listener as sensitive and firewall it to the workers that need to push and the Prometheus that needs to scrape.
• The Pushgateway is a denial-of-service target. A worker that pushes a million labels into a single metric exhausts the Pushgateway’s memory. Validate the cardinality of every metric before pushing.
• Persisted metrics persist across worker credentials. A metric pushed by an ex-employee’s batch job is still on disk six months later. Rotate the Pushgateway’s storage alongside the worker credentials.

Performance implications

• Pushgateway memory scales with the number of distinct label sets. A Pushgateway holding 10,000 distinct label sets per metric consumes tens of megabytes. A Pushgateway holding 10 million is in trouble.
• Prometheus’s scrape of the Pushgateway scales with the number of metrics held. A Pushgateway that has been running for a year with no expiry holds a year’s worth of pushed series. Set --push.disable-consistency and a job-cleanup cron.
• cAdvisor’s polling cost is the same regardless of container lifetime. The cost is per housekeeping interval, not per scrape.

Production guidance

• Treat the Pushgateway as a tool for batch jobs whose metrics are not retrievable by scraping. Do not push service metrics through it.
• Run the Pushgateway with persistence enabled and a single replica behind a stateful-set. Multi-replica Pushgateway is not supported.
• Set explicit label cardinality on every pushed metric. Validate that no label is unbounded.
• For Docker-native batch jobs, prefer the pre-stop hook pattern with the OpenTelemetry SDK; the container’s lifecycle exit triggers the flush.
• Always honor_labels: true on the Pushgateway scrape job. Without it, the pushed labels are silently rewritten.
• Cross-reference Pushgateway metrics with Loki streams for the same job. The Pushgateway holds the metric; Loki holds the structured log. Together they reconstruct the run.

Verification

You should now be able to answer:

• Why does the default pull model miss short-lived containers?
• When is the Pushgateway the right pattern, and when is it the wrong pattern?
• What does honor_labels: true do on the Pushgateway scrape job, and what happens without it?
• Which metric confirms that cAdvisor saw a container that has since exited?
• What is the operational difference between the pre-stop hook pattern and the sidecar Pushgateway pattern?

Quiz