Common Rule Patterns

Five rules cover the bulk of paging volume in every mature Prometheus estate: error rate, availability, latency, saturation, and capacity. Each is a single PromQL expression with disciplined labels. None of them are exotic. The cost of not having them is that a real incident is detected by a user complaint instead of by an alert, and time-to-resolution is measured in tens of minutes instead of single digits.

What it is

A rule pattern is a reusable shape: a metric family, a window, a threshold, and a label set. The five that appear in nearly every estate:

• Error rate — the ratio of 5xx responses to total responses over a window. The canonical user-impact alert.
• Availability — up == 0 on a scrape target, or a blackbox-exporter probe failure. The canonical “the service is gone” alert.
• Latency — p99 (or p95) of a histogram metric above a threshold derived from the SLO. The canonical user-experience alert.
• Saturation — a host or service resource approaching its limit (CPU, memory, disk I/O, connection pool). The canonical capacity-burn alert.
• Capacity headroom — a rate-of-fill metric (disk fills at X% per day, time-to-exhaustion below a threshold). The canonical “plan a change” alert.

Two further patterns appear in mature shops:

• Exemplar-linked alerts — alerts that carry an exemplar pointer to a representative trace of the failure.
• Symptom vs cause — the discipline of choosing one or the other based on the action the operator takes.

Why a sysadmin cares

A team that adopts these five patterns as a starting template gets a service-level alert estate in a day. A team that does not adopt them ends up with bespoke rules that fire inconsistently, routing labels that do not match, and an on-call rota that learns to ignore pages. The cost of the first path is a half-day of writing; the cost of the second is a quarter of burnout.

The symptom-vs-cause split matters because the two call for different actions. A symptom alert fires because users are affected; the action is to stop the bleeding. A cause alert fires because an upstream condition known to produce user impact is present; the action is to prevent the bleed.

How it works

The patterns share a common shape:

   metric family
        |
        v
   aggregation (sum by, histogram_quantile)
        |
        v
   comparison (>, <, ==, !=)
        |
        v
   for: dwell
        |
        v
   labels: severity, team, service
        |
        v
   annotations: summary, description, runbook_url, dashboard_url

Five worked patterns, in production shape:

groups:
  - name: orders-api.slo
    interval: 30s
    rules:
      # 1. Error rate: 5xx ratio above 5% over 5 minutes.
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
          slo: availability
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

      # 2. Availability: blackbox probe of the public health endpoint.
      - alert: OrdersApiProbeFailed
        expr: probe_success{instance=~"orders-api.*"} == 0
        for: 2m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api blackbox probe failed for {{ $labels.instance }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-probe'

      # 3. Latency: p99 above the SLO budget of 300ms over 5 minutes.
      - alert: OrdersApiHighP99Latency
        expr: |
          histogram_quantile(0.99,
            sum by (le, service, region) (
              rate(http_request_duration_seconds_bucket{service="orders-api"}[5m])
            )
          ) > 0.3
        for: 10m
        labels:
          severity: warning
          team: checkout
          service: orders-api
          slo: latency
        annotations:
          summary: 'orders-api p99 above 300ms in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-latency'

      # 4. Saturation: host CPU above 90% for 15 minutes.
      - alert: HostCpuSaturated
        expr: |
          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
          > 90
        for: 15m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: 'CPU above 90% on {{ $labels.instance }} for 15 minutes'
          runbook_url: 'https://runbooks.example.com/platform/host-cpu'

      # 5. Capacity headroom: disk fills within 24 hours.
      - alert: DiskFillsWithin24h
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24 * 3600) < 0
        for: 1h
        labels:
          severity: warning
          team: platform
        annotations:
          summary: 'disk on {{ $labels.instance }} projected to fill within 24h'
          runbook_url: 'https://runbooks.example.com/platform/disk-capacity'

The five expressions have a lot in common. Each is a comparison against a threshold, expressed as a number rather than a boolean, aggregated by the label set that the Alertmanager route needs.

How to configure it

The five rules above are the minimal set. A few extras that appear in mature shops:

      # Scrape failure: the Prometheus job cannot reach the target.
      - alert: PrometheusTargetDown
        expr: up{job=~"orders-api.*"} == 0
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: 'prometheus cannot scrape {{ $labels.instance }}'
          runbook_url: 'https://runbooks.example.com/platform/scrape-failure'

      # Missing metric: a high-priority metric is absent for too long.
      - alert: OrdersApiRequestRateMissing
        expr: absent(http_requests_total{service="orders-api"}) == 1
        for: 10m
        labels:
          severity: warning
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api request-rate metric absent for 10 minutes'
          runbook_url: 'https://runbooks.example.com/checkout/missing-metric'

      # Exemplar-linked alert: surface a trace of the failure.
      - alert: OrdersApiSlowExemplar
        expr: |
          histogram_quantile(0.99,
            sum by (le, service) (
              rate(http_request_duration_seconds_bucket{service="orders-api"}[5m])
            )
          ) > 0.3
        for: 10m
        labels:
          severity: warning
          team: checkout
          service: orders-api
          exemplar_attached: 'true'
        annotations:
          summary: 'orders-api p99 above 300ms; exemplar trace linked'
          dashboard_url: 'https://grafana.example.com/d/orders-api/exemplars?var-service=orders-api'

The third rule relies on exemplars: Prometheus attaches a representative trace ID to the histogram bucket when the bucket is updated. The Grafana dashboard reads the exemplar from the metric and links the operator to a Tempo trace of one of the slow requests.

How to validate it

Three checks against a running Prometheus. The first confirms the rule is loaded; the second confirms the expr returns a series; the third confirms the rule fires under the right conditions.

# 1. Confirm the rules are loaded.
curl -s http://prometheus:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name == "OrdersApiHighErrorRate")
        | {state, query, lastEvaluation}'

Expected output:

{
  "state": "inactive",
  "query": "sum by (service, region) (...)  > 0.05",
  "lastEvaluation": "2026-08-13T03:00:00.000Z"
}

state: inactive is fine; the rule is loaded and the expr returned no result series at the last evaluation. A rule with state: pending or state: firing is one that is actively tracking an alert.

# 2. Confirm the expr returns a series in normal conditions.
curl -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=
    sum by (service, region) (
      rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
    )
    /
    sum by (service, region) (
      rate(http_requests_total{service="orders-api"}[5m])
    )' \
  | jq '.data.result[] | {labels: .metric, ratio: .value[1]}'

Expected output:

{
  "labels": { "service": "orders-api", "region": "eu-west-1" },
  "ratio": "0.0023"
}

If data.result is empty, the metric is missing or the label selector is wrong.

# 3. Replay a synthetic series that breaches the threshold.
# (covered in detail in lesson 06; the test rule file under
# test/orders-api_test.yml contains the input series and the
# expected alert.)
promtool test rules /etc/prometheus/rules/test/orders-api_test.yml

Expected output:

SUCCESS

How it can fail

Six failure modes:

1. Division by zero on a low-traffic service. Symptom: the rule fires spuriously when the total request rate is zero. Cause: rate(http_requests_total[5m]) returns no series, so the divisor is empty. Mitigate with clamp_min or a small additive on the denominator; better, alert on the absolute error count when traffic is too low for a ratio to be meaningful.

2. up == 0 fires on a healthy service that was just re-scraped on a different label. Symptom: the alert fires because the new scrape job uses a different job label. Cause: the rule’s selector does not include the new job. Update the selector and audit the entire rule set for the same gap.

3. p99 alert masks a p50 regression. Symptom: the p50 latency doubled but p99 stayed within budget. Cause: only p99 is alerted. Add a parallel p50 rule at the same severity, or alert on the histogram mean (rate * sum / count).

4. Saturation alert fires on a CPU steal from a noisy neighbour. Symptom: CPU above 90% on a host that is healthy. Cause: the saturation metric is host-wide, not per-process. Add a label selector (process_cpu_seconds_total) or scope the rule to the cgroup.

5. Capacity alert mispredicts. Symptom: the rule fires when the disk has 50% free, because the regression line over the past 6h has a sharp slope from a one-time log write. Cause: predict_linear() over a short window is noisy. Lengthen the window to 24h or use deriv() instead.

6. Exemplar alert has no exemplar. Symptom: the alert fires but the dashboard link points to no trace. Cause: the exemplar store is misconfigured, or the histogram bucket is not exposed by the application. Confirm by inspecting the metric with exemplar_show.

How to troubleshoot it

In order:

1. Did the rule load? /api/v1/rules. Missing rule: fix the glob and reload.
2. Does the expr return data? Compute it in Grafana Explore over the past hour. Empty result: the metric is missing or the selector is wrong.
3. What does the threshold plot look like? Plot the expr value, not just the threshold. The expr value should be jumping around the threshold, not flatlining at zero.
4. Are the labels right? Inspect the result series metric map. The label set should match what the route tree expects.
5. What did the unit test say? Run promtool test rules with a synthetic series that breaches the threshold. If the test passes, the rule is correct against the test; if the test fails, the rule is wrong.

Security implications

Common patterns are not security-sensitive on their own. The expressions read metrics; they do not exfiltrate data. The risks are operational:

• A rule whose expr aggregates over a high-cardinality label (for example user_id) creates a denial-of-service in the rule evaluator. Bound the expr with a label selector.
• A runbook_url that points at an external system can leak the existence of an incident to a third party. Confirm the hostname.

Performance implications

The five patterns above are all bounded aggregations; they should evaluate in milliseconds against a healthy Prometheus. The patterns that cost CPU are:

• Wide histogram_quantile() without sum by aggregation. Always aggregate the bucket vector first.
• predict_linear() over a long range. Use a 6h window, not a 24h window.
• rate() over a counter with very high cardinality. Bound the selector.

For high-cardinality estates, convert the heavy patterns into recording rules and alert on the recording rule. The recording rule runs once per evaluation and the alert rule reads from a pre-aggregated series.

Production guidance

• Adopt the five patterns as the starting template for every new service. Add bespoke rules only after the five are firing correctly.
• Pair every symptom alert with at least one cause alert and every cause alert with at least one symptom alert. The two together are what makes an incident actionable.
• Use predict_linear() for capacity alerts, but cap the window at 6h. Longer windows produce noisy predictions.
• For exemplar-linked alerts, confirm the dashboard is set up to read the exemplar from the metric. An exemplar alert that points at a dashboard without exemplar support is no better than a non-exemplar alert.

Verification

• What is the difference between a symptom alert and a cause alert, and why should both exist?
• Why does a rate-based error-rate alert fail on a low-traffic service?
• What is the difference between up == 0 and a blackbox probe_success == 0?
• What does predict_linear() predict, and what window should it use?

Quiz