Operating the monitoring stack - retention, cardinality and who watches the watcher

The previous lessons in this part built a monitoring stack. This one treats it as what it now is: a production service with a capacity limit, a set of failure modes, and - uniquely - no monitoring of its own unless you build it.

The distinguishing property of this service is that its failures are silent. A Prometheus that has stopped ingesting looks like a quiet night, and an Alertmanager that cannot deliver looks like an estate with no problems.

Size the TSDB before it sizes itself

Prometheus storage is driven by one number: samples ingested per second.

samples/sec = active series / scrape interval

A fleet of 500 hosts, each exporting about 900 series, scraped every 15 seconds, is 500 * 900 / 15 = 30,000 samples per second. Compressed samples land at roughly one to two bytes each in practice, so a rough figure for 30 days is:

30,000 samples/s * 86,400 s/day * 30 days * 1.5 bytes ~= 116 GB

Treat that as an order of magnitude, not a promise: the compression ratio depends heavily on how much your values actually change. Measure it on your own data once you have a week of it, then set both retention flags.

# flags in the unit file or /etc/default/prometheus
--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=200GB

Set both. Time-based retention alone gives you no protection when series count grows, and the failure mode of a full Prometheus disk is not a truncated history - it is a stopped database.

$ curl -sS http://localhost:9090/api/v1/status/runtimeinfo | head -c 400; echo; du -sh /var/lib/prometheus; df -h /var/lib/prometheus

{"status":"success","data":{"startTime":"2026-07-14T02:11:09Z","CWD":"/","reloadConfigSuccess":true,"lastConfigTime":"2026-08-11T07:02:00Z","timeSeriesCount":451208,"storageRetention":"30d"}}
71G	/var/lib/prometheus
Filesystem      Size  Used Avail Use% Mounted on
/dev/mapper/vg0-prom  250G   71G  167G  30% /var/lib/prometheus

Find the metric that exploded

Series growth is almost never gradual. It arrives as a step, because somebody added a label, deployed an exporter, or started a job that puts an id in a metric name.

$ curl -sS http://localhost:9090/api/v1/status/tsdb | python3 -m json.tool | head -30

{
  "status": "success",
  "data": {
      "seriesCountByMetricName": [
          { "name": "http_request_duration_seconds_bucket", "value": 184320 },
          { "name": "node_cpu_seconds_total", "value": 32000 }
      ],
      "labelValueCountByLabelName": [
          { "name": "request_id", "value": 91204 },
          { "name": "instance", "value": 500 }
      ]
  }
}

The offline equivalent, useful on a Prometheus that is too loaded to answer queries, reads the blocks on disk:

promtool tsdb analyze /var/lib/prometheus

Fixing it is a two-part job, and doing only the first part means it comes back next week:

1. Stop the bleeding at the scrape: drop the offending label with a metric_relabel_configs rule so the series stop being created
2. Fix the source, because relabelling is a workaround that costs CPU on every scrape forever
3. Old series expire with retention; they do not disappear when you stop creating them, so plan for the disk to stay large until the retention window passes

scrape_configs:
  - job_name: myapp
    static_configs:
      - targets: ['192.0.2.21:9102']
    metric_relabel_configs:
      # Drop one unbounded label from every series in this job.
      - regex: 'request_id'
        action: labeldrop

Guard rails at the scrape

A target that starts exporting a million series should not be able to take the server down. Three limits stop that, set per job or globally.

scrape_configs:
  - job_name: myapp
    sample_limit: 5000
    label_limit: 30
    label_value_length_limit: 256
    static_configs:
      - targets: ['192.0.2.21:9102']

Recording rules for what is queried repeatedly

A dashboard panel that runs an aggregation over 500 instances runs it on every refresh, for every viewer. A recording rule computes it once per evaluation interval and stores the result.

groups:
  - name: host_aggregates
    interval: 30s
    rules:
      - record: instance:node_cpu_utilisation:rate5m
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

The naming convention is level:metric:operations - the aggregation level first, then the underlying metric, then what was done to it. It is worth following: on a stack with a hundred recording rules, the name is the only documentation anybody reads.

Move an expression into a recording rule when it is used by more than one dashboard or alert, or when its evaluation appears in the slow-query list. Not before - every rule is a new series stored forever.

Reload safely

$ promtool check config /etc/prometheus/prometheus.yml && promtool check rules /etc/prometheus/rules/*.yml && curl -sS -X POST http://localhost:9090/-/reload && curl -sS http://localhost:9090/api/v1/status/runtimeinfo | grep -o 'reloadConfigSuccess":[a-z]*'

Checking /etc/prometheus/prometheus.yml
SUCCESS: 1 rule files found
Checking /etc/prometheus/rules/host.yml
SUCCESS: 12 rules found
"reloadConfigSuccess":true

A rejected reload is not a failure of the reload mechanism, it is the mechanism working. Prometheus keeps serving the last good configuration. The danger is assuming the new rules are live: check reloadConfigSuccess and confirm the new rule appears under /api/v1/rules before you rely on it.

Silences, and the discipline of expiring them

Planned maintenance needs silences, and silences are the most common way an estate ends up unmonitored.

amtool silence add alertname=NodeDown instance=web02:9100 \
    --duration=2h --author="ada" \
    --comment="kernel upgrade CHG-4471" \
    --alertmanager.url=http://localhost:9093

amtool silence query --alertmanager.url=http://localhost:9093

Three rules make silences safe:

• Always set a duration, and make it shorter than you think you need. An expired silence that has to be re-created costs thirty seconds; a silence that outlives the maintenance hides a real outage.
• Always match narrowly. A silence on severity=critical with no instance matcher silences the fleet.
• Review open silences on a cadence. amtool silence query in a weekly checklist is enough, and it is where you discover the one somebody created in March.

Who watches the watcher

Three mechanisms, and you want all three because they fail independently.

A dead man switch. An alert that always fires, routed to a receiver that expects it. If the heartbeat stops arriving, the alerting path is broken somewhere between the rule evaluator and the receiver.

groups:
  - name: watchdog
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: watchdog
        annotations:
          summary: 'Alerting pipeline is alive. Absence of this alert is the alert.'

Route severity: watchdog to an external heartbeat service that pages when it stops hearing from you. A dead man switch delivered to the same on-call rota by the same email path it is meant to be testing proves nothing.

Prometheus watching itself, and a second Prometheus watching it. Self-monitoring catches rule and delivery problems; a second instance catches the case where the first one is down.

- alert: PrometheusRuleFailures
  expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0
  for: 10m

- alert: PrometheusNotificationsDropped
  expr: increase(prometheus_notifications_dropped_total[10m]) > 0
  for: 10m

- alert: PrometheusTargetsMissing
  expr: absent(up{job="node"} == 1)
  for: 10m

absent() is the one people forget. A comparison against a series that no longer exists is not false, it is empty, so a rule written as up == 0 cannot fire for a target that vanished from service discovery entirely.

Knowledge check