Alerting rules that page — and the ones that should not

A metric you cannot alert on is not monitoring. An alert nobody acts on is worse than no alert, because it trains the on-call engineer to dismiss the channel it arrives in.

This lesson is the bridge between the two: the specific rules worth having on a Docker host, written so they fire when something is wrong and stay quiet the rest of the time.

The test every rule has to pass

Before a rule goes in, answer three questions in writing:

1. What symptom does this describe? If the answer is a cause (“CPU is high”) rather than a symptom (“requests are slow”), it probably should not page.
2. Who acts, and what do they do first? The rule needs a runbook link in its annotations. “Investigate” is not an action.
3. What happens if it fires at 03:00 on a Sunday? If the honest answer is “nothing until Monday”, it is a ticket, not a page.

Rules that fail question 3 still belong in the system. They just route somewhere that does not wake anyone.

Rules for the Docker failure modes

A container is restarting in a loop

- alert: ContainerRestartLoop
  expr: changes(container_start_time_seconds{name!=""}[15m]) > 3
  for: 5m
  labels:
    severity: page
  annotations:
    summary: '{{ $labels.name }} restarted more than 3 times in 15 minutes'
    runbook: 'https://runbooks.example.com/docker/restart-loop'

container_start_time_seconds changes value every time the container starts, so changes() over a window counts restarts. Three in fifteen minutes is a crashloop; one is a deploy.

A container was OOM-killed

- alert: ContainerOOMKilled
  expr: increase(container_oom_events_total[10m]) > 0
  labels:
    severity: page
  annotations:
    summary: '{{ $labels.name }} was OOM-killed'
    runbook: 'https://runbooks.example.com/docker/oom'

No for: clause. The condition is an event that already happened, not a state that needs to persist — waiting five minutes to confirm it just delays the page. This is the rule that catches the 02:10 kill that a memory-usage threshold misses, because usage is low again by the time anyone looks.

CPU is being throttled

- alert: ContainerCPUThrottled
  expr: |
    rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m])
      / rate(container_cpu_cfs_periods_total{name!=""}[5m]) > 0.25
  for: 15m
  labels:
    severity: ticket
  annotations:
    summary: '{{ $labels.name }} throttled in over 25% of CFS periods'

A ticket, not a page. Throttling degrades latency; it does not take the service down, and the fix is a limit change that belongs in a change window.

Memory is approaching the limit

- alert: ContainerMemoryNearLimit
  expr: |
    container_memory_working_set_bytes{name!=""}
      / container_spec_memory_limit_bytes{name!=""} > 0.9
  for: 10m
  labels:
    severity: ticket

Working set, not container_memory_usage_bytes. Usage includes reclaimable page cache and will sit near the limit on any healthy container doing file I/O, which makes this rule fire constantly if you pick the wrong series.

The Docker filesystem is filling

- alert: DockerDiskFillingUp
  expr: |
    predict_linear(
      node_filesystem_avail_bytes{mountpoint="/var/lib/docker"}[6h],
      4 * 3600
    ) < 0
  for: 30m
  labels:
    severity: page
  annotations:
    summary: '/var/lib/docker predicted full within 4 hours'

predict_linear pages on the trajectory rather than on a fixed percentage, which buys you the four hours needed to act. If /var/lib/docker is not a separate mount, change the matcher to the filesystem that actually holds it — check with df /var/lib/docker before you trust the label.

A container that should exist does not

- alert: ContainerMissing
  expr: absent(container_last_seen{name="postgres"})
  for: 5m
  labels:
    severity: page

The alert nobody writes, because it requires naming what should be running. Every other rule in this list needs the container to exist in order to fire. A container that was removed produces no metrics, no restarts and no OOM kills — it produces silence, and silence is the default state of a healthy system.

for: is what separates an alert from a flap

for: requires the condition to hold continuously before the alert fires. Without it, one bad scrape pages.

Pick the duration from how long the condition can legitimately hold during normal operation, then add margin. Guessing produces either a noisy channel or a slow one.

Routing: one failure, one page

When a host dies, every container on it goes missing, every scrape fails, and a naive configuration sends twenty pages for one event.

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'tickets'
  routes:
    - matchers: [ 'severity = page' ]
      receiver: 'oncall'

inhibit_rules:
  - source_matchers: [ 'alertname = HostDown' ]
    target_matchers: [ 'severity =~ "page|ticket"' ]
    equal: ['instance']

group_by on instance collapses the twenty into one notification. The inhibit rule suppresses the container-level alerts entirely when the host itself is down, because the host is the actionable finding and the containers are consequences of it.

Test the rules, not just the syntax

promtool check rules /etc/prometheus/rules/docker.yml
promtool test rules /etc/prometheus/rules/docker_test.yml
promtool check config /etc/prometheus/prometheus.yml
amtool check-config /etc/alertmanager/alertmanager.yml

A unit test states a series, a time, and the alerts you expect:

# docker_test.yml
rule_files:
  - docker.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'container_oom_events_total{name="db"}'
        values: '0 0 0 1 1 1 1 1 1 1 1 1'
    alert_rule_test:
      - eval_time: 5m
        alertname: ContainerOOMKilled
        exp_alerts:
          - exp_labels:
              name: db
              severity: page

1. Name the symptom the rule describes, and reject the rule if the answer is a cause.
2. Attach a runbook URL in the annotations. A page without a first action is a page that gets escalated.
3. **Set for: from observed normal behaviour**, not from intuition.
4. Split severities: page for user impact and data risk, ticket for everything else.
5. Inhibit consequences. One host failure should produce one notification.
6. Add the Watchdog rule and route it to a heartbeat monitor.
7. Unit-test the rules with promtool, and test the notification path on a schedule.

Sanity check