Rule Too Broad

A checkout team writes a rule against the error-rate metric. The intent is “page if the checkout production service sustains a 5xx rate above 5%”. The rule is:

- alert: CheckoutHighErrorRate
  expr: |
    sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (service) (rate(http_requests_total[5m]))
    > 0.05

It pages on a Sunday morning. The engineer opens Grafana. Production is fine. The firing alert carries the labels service=checkout. So far so good. The engineer drills deeper:

curl -s 'http://prometheus:9090/api/v1/query?query=sum+by+(service,environment)(rate(http_requests_total[5m]))' \
  | jq '.data.result[]'

The series includes service=checkout environment=staging and service=checkout environment=canary. Both fire on a 5xx-staging-suite that was deliberately run over the weekend. The rule has no environment matcher, no synthetic matcher, no canary matcher. The team page.

That is a rule too broad. The expression selected more series than the team intended.

What “rule too broad” means

A rule is too broad when its expression — the expr: block — selects a wider label set than the team intends to monitor. Two shapes:

1. Wrong positive matchers. The expression includes a series set that the team did not intend to alert on (staging, canary, synthetic probes, shadow services).
2. Wrong negative matchers. The expression lacks the exclusions that would have narrowed the selection.

The shape is almost always one of these. The pattern in expr: is “match what you intend, exclude what you don't”. A rule that omits the second half is too broad.

The phrase is not the same as “the rule has too many matchers” or “the rule selects too many series”. A rule can be too broad without its selection being numerically large. A rule with a single matcher that excludes an important series is also wrong, in the opposite direction (too narrow).

Why a sysadmin cares

A rule that is too broad produces the same operational pains as any false-positive generator:

• Pages on staging events. The on-call engineer pages, opens the alert, walks through the dashboard, finds the label is environment=staging. The page was wrong.
• Pages on synthetic probes. Synthetic monitoring runs a 5xx-generating test against the production host on a timer; the rule fires because the synthetic probe is in the same target as the production metric.
• Pages on canary deployments. The canary had a 50% error rate for five minutes during a load test; the rule fires because the canary shares job= with production.
• Runbook noise. The runbook assumes the production environment; the alert fires on staging, the engineer follows the runbook, no production service matches, the engineer wastes time.
• Slow tuning. A team that cannot trust the rule to select the right series will not trust the threshold; the next tune is a “just in case” lift, and the rule’s sensitivity drifts.

How it works

The aggregation in a Prometheus alerting rule is the part of the expression that defines which series enter the calculation. It is the sum by, the avg by, the count without grouping, the matcher filter applied before the function. Anything that increases the cardinality of the input to the rate or histogram pushes the rule towards “too broad”.

+----------------------+      +-----------------------------+
| Underlying metric    |      | Rule expression             |
| with all labels      | --> |   matchers (filter)         |
| attached             |      |   by-clause (group)         |
| instance, region,    |      |   threshold (gate)         |
| job, environment,    |      +-------------+---------------+
| code, synthetic, ... |                    |
+----------------------+                    v
                                  +-----------------------------+
                                  |  Series set that will be   |
                                  |  evaluated                 |
                                  +-----------------------------+

The matches filter is the easy place to be too broad. Common oversights:

• job=~".+" matches every job. So does the absence of any job= matcher (Prometheus would not require it).
• code=~"5.." is fine. code=~"5" matches every label string starting with 5 (which is what you want) AND every label string 5 (also what you want). But code!~"200" matches 200, 201, 2001. Intent is harder to read.
• environment!~"staging" excludes only staging. A label value STAGING or Staging is not excluded unless the pipeline normalises case. Look for environments labelled dev, Dev, DEV, development, develop.

The right shape is positive matchers that specify the intended population and negative matchers that exclude the known unintended ones.

The most common shape

In roughly two-thirds of rule-too-broad findings, the cause is a missing exclusion for traffic the team already knows is not production. The shape is one of:

1. Synthetic probes. The synthetic monitoring system uses the same target as production and stamps a label such as synthetic="true". The rule does not exclude the label.
2. Canary or shadow deployments. The canary shares job= with production. The canary generates errors during a load test. The rule fires.
3. Multi-environment services. The same exporter serves staging, dev, and canary; the metric has no environment label; the alerting rule assumes production.

A second common shape is missing environment labels on the underlying metric: the metric does not surface enough labels for the rule to filter precisely. The fix there is to extend the exporter, not the rule.

A third is rule aggregation by the wrong key: a rule that uses sum by (job) when the same job value covers production and staging in different timezones, leaving the team unable to tell which side fired.

Under the hood

Aggregation in PromQL happens in two stages:

1. Vector matching. The matchers in the expression select series from the underlying metric. job="checkout" selects every series with that label value.
2. Aggregation operators. sum by (label) collapses the selected series into one series per group.

A rule is too broad when either stage admits series the team did not intend to alert on. The fix is at the matcher stage or by adding without to the by-clause to omit labels that the team does not care about.

The downside of an over-specific matcher is silent rules: a new team renames a label, the rule does not fire, and the team finds out from a customer. This is why the right rule keeps a label-distribution inventory: the team owns the labels, the labels are reviewed against the rules.

How to configure it

For a rule whose matcher is too broad, the configuration fix is to add the missing exclusions. Real annotated rule:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum by (service, environment, region) (
            rate(
              http_requests_total{
                job=~"checkout(-svc)?",
                code=~"5..",
                environment="production",
                canary!="true",
                synthetic!="true",
                shadow_traffic!="true",
              }[5m]
            )
          )
          /
          sum by (service, environment, region) (
            rate(
              http_requests_total{
                job=~"checkout(-svc)?",
                code=~"2..|3..|4..|5..",
                environment="production",
                canary!="true",
                synthetic!="true",
                shadow_traffic!="true",
              }[5m]
            )
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout 5xx rate above 5% in {{ $labels.region }} ({{ $labels.service }})'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'
          exclusions: 'canary, synthetic, shadow_traffic'

Six things to read into that:

• environment="production" is a positive matcher. It narrows the rule to the production environment by label value.
• canary!="true", synthetic!="true", shadow_traffic!="true" are negative matchers. They keep the rule quiet on traffic the team already knows is non-production.
• The by-clause groups by service, environment, region so the firing alert identifies the deployment shape the team intends.
• The exclusions annotation documents the exclusions. The next engineer to read the rule sees what is excluded and why.
• job=~"checkout(-svc)?" allows either checkout or checkout-svc (the team is migrating the job name).
• The 5.. and 2..|3..|4..|5.. patterns match all 5xx and all 2xx/3xx/4xx/5xx respectively. This is the canonical shape for 5xx-rate alerting.

How to validate it

Validate by enumerating the series the rule actually selects:

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'sum by (service, environment, region, canary, synthetic) (
     rate(http_requests_total{job=~"checkout(-svc)?",code=~"5.."}[5m])
   )'

Sample output:

service=checkout environment=production region=eu-west-1 canary=false synthetic=false  0.0031
service=checkout environment=production region=us-east-1 canary=false synthetic=false  0.0048
service=checkout environment=production region=ap-south-1 canary=true  synthetic=false  0.12
service=checkout environment=staging    region=eu-west-1 canary=false synthetic=true   0.45

The fourth line is the one that produced the false positive. The third line is the canary deployment. The new expression filters both with canary!="true" and synthetic!="true".

Check the rule against the new expression:

promtool check rules /etc/prometheus/rules/checkout.yml

SUCCESS: rule files validated; 12 rules found, 0 errors

Check the alert state under load:

curl -s 'http://prometheus:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="CheckoutHighErrorRate")'

The expected output is no current alerts and at most the production series labels under load.

How it can fail

Six specific shapes, each with the symptom that distinguishes it from other broadness failures:

1. No environment= matcher and the metric has the label. The expression was written before the team standardised the label. Symptom: the alert fires in every environment the service runs in.

2. synthetic label added later, rule not updated. A synthetic monitoring system was added six months ago and stamps synthetic="true". The rule predates it. Symptom: the alert correlates with synthetic monitoring cron windows.

3. Canary and prod share job=. A canary deployment under load produces a 5xx burst. The rule fires. Symptom: the alert correlates with canary releases.

4. sum by (job) covers a renamed job. Production and staging share an old job value because the rename was incomplete. The rule fires across both. Symptom: the firing alert’s labels show a single job value spanning two environments.

5. code=~"5.." matches 5 too. A misdesigned regex matches code="5" (a stale or test label) as well as code="500", code="503". Symptom: the alert fires on any series with code="5", regardless of the rest of the label.

6. No != clause for deprecated labels. A label like legacy_canary=true was retired six months ago but remains on some scrapes. The rule fires on the legacy label while ignoring the new canary label. Symptom: the alert correlates with hosts still running the legacy scrape configuration.

How to troubleshoot it

1. Open the rule. Read the matcher list. For each missing positive or negative matcher, ask: is there a label set on the underlying metric that should not be here?
2. Look at the labels attached to the firing alert. For each label, identify whether it represents the intended population. If environment=staging is on the alert and the rule should be production-only, the matcher is missing.
3. Run the rule expression with an extra by clause that surfaces all the labels the team cares about. Confirm that no excluded label values are present.
4. Add the missing exclusions. Document them in the rule’s annotations. Run promtool check rules. Reload Prometheus.
5. Re-run the test that originally produced the false positive. The alert should stay in inactive. If it still fires, return to step 1.

Security implications

A rule that fires on production and non-production traffic can disclose operational information to operators who should not see staging metrics (a smaller blast radius than exposing to the public, but still a violation of least-privilege).

A rule that fires on shadow traffic (real production requests, mirrored to a canary service) can also disclose production traffic patterns to anyone with read access to the alerting system. The fix is to exclude shadow traffic explicitly, not to assume it is filtered at the metric level.

Performance implications

A rule that selects more series than it needs is more expensive to evaluate. The cost is bounded by the selected series, not by the team’s intent. A rule tightened by adding an exclusion typically drops its evaluation cost proportional to the cardinality it excluded.

For Alertmanager 0.28.x, broader rules also inflate the notification grouping log. Each firing alert produces one log entry; broader rules produce more entries. The cost is small for tens of alerts and meaningful for thousands.

Production guidance

• Document the exclusion policy in the rule. The next engineer needs to know which label sets are out.
• Run a quarterly label-distribution review. New label values appear faster than the team reviews rules.
• Keep a label inventory for every metric used in alerting. A label that disappears or is renamed should be visible to the team that owns the rules.
• Pair exclusions with positive matchers. Negations of defaults age badly.

Verification

You should now be able to answer:

• What is the difference between a rule that is too broad and a rule that fires on normal traffic?
• Which label, in the typical false-positive story, is most often responsible for the broadness?
• How do you confirm a rule is no longer too broad using promtool query instant?
• Why is “narrows the rule, narrows the alert” a poor long-term defence against broad rules?

Quiz