Rule Wrong

The rule loads, promtool check rules returns SUCCESS, the team believes the alert is live. A real outage occurs two months later; the alert does not fire. The team’s first guess is that the scrape is broken. They check up{job="X"}; it is 1. They check the long-term store; the metric is there. They run the expression from the rule body in the Grafana PromQL editor; the result is an empty vector. The expression the team believes is checking the condition is not the expression the rule is actually evaluating.

This lesson is that gap. The expression is wrong. It was wrong when the team merged it. The defect was silent. The fix is not at the receiver or the route; the fix is in the expr: field of the rule.

What “rule wrong” is

A rule wrong condition in production terms is an alert rule whose expression does not evaluate to the result set the operator believes it does. Three conditions must hold:

1. The rule file has loaded. promtool check rules returned SUCCESS and the rule appears in /api/v1/rules.
2. The expression evaluates against the live metric without error.
3. The vector the expression returns is not the vector the operator intended.

Condition one excludes parse-time errors. Condition two excludes metric rename or label drift (covered by lesson 02). Condition three isolates the cause to the rule’s own expression shape.

A rule wrong condition is distinct from a threshold wrong (lesson 04) condition. The two overlap: a threshold that is two orders of magnitude wrong is also an expression that returns a vector the operator did not intend to alert on. The convention used in this module is to call the defect a rule wrong if the expression itself is malformed (a function name that does not exist, a label matcher that references the wrong label, an aggregation that drops everything) and a threshold wrong if the expression is sound but the comparison constant is mis-set against the metric’s natural range.

Why a sysadmin cares

A rule wrong is the most expensive defect class because it is invisible until the condition that should page actually appears. Three failure shapes repeat:

• The rule never fires. A typo in a label matcher, a function call that always returns no vector, an aggregation whose by clause omits a label. The rule sits in inactive forever. The team discovers the bug when the outage happens.
• The rule fires on noise. A rate window that is too short, a comparison whose > is >=, an aggregation that includes traffic the rule was meant to exclude. The alert pages for non-incidents. The team silences or disables it.
• The rule has too many series. A count by (...) that returns every permutation produces a flood of alerts, one per series. Each is correct individually; the route is overwhelmed.

The discipline that prevents these is the same: validate at write time, validate at load time, validate at query time. Three validation steps that catch different shapes of the same defect class.

How it works

A Prometheus alerting rule is a YAML record that pairs an expression with metadata. The expression has its own grammar, its own semantics, and its own failure modes. The grammar is PromQL; the failure modes fall into four shapes.

+-----------------------+    +-----------------------+
| Shape 1: parse-time   |    | Shape 2: type-time     |
| YAML indentation      |    | Wrong function name   |
| missing colon         |    | wrong argument type    |
| unbalanced quote       |    | metric not found       |
+-----------+-----------+    +-----------+-----------+
            |                            |
            v                            v
+-----------------------+    +-----------------------+
| Shape 3: empty-result |    | Shape 4: cardinality  |
| expr returns []       |    | expr returns thousands |
| at evaluation time    |    | of series at one time |
+-----------------------+    +-----------------------+
            |                            |
            +-------------+--------------+
                          |
                          v
+--------------------------------------------------+
| Symptom: rule in /api/v1/rules but not in ALERTS  |
| Or: rule in ALERTS but never crosses firing       |
+--------------------------------------------------+

Shape 1 is the parse-time defect. The YAML is malformed or the PromQL expression contains a syntactically invalid construct. promtool check rules catches this with a line number. The rule fails to load; every rule in the same file is dropped.

Shape 2 is a type-time defect. The expression parses but evaluates with an error: a metric that does not exist in any of the loaded targets, a function called with the wrong arithmetic type, a regex that is invalid Go. The expression returns no vector; the rule sits inactive. promtool check rules does not catch shape 2; promtool query instant catches it.

Shape 3 is the empty-result defect. The expression parses and evaluates without error, but returns [] because the label matchers do not match the live series, the recording rule does not exist, or the rate window is empty. The rule sits inactive. The most expensive defect shape because the absence of action looks like the absence of condition.

Shape 4 is a cardinality defect. The expression returns thousands of series. The rule fires correctly on each one but AM’s group machinery is overwhelmed. Most often this is caused by an aggregation that does not have a by clause, so the expression summarises the entire population at once.

The most common cause

In roughly half of rule-wrong investigations the team reviews, the cause is at Shape 3 (empty result) with a specific pattern: a label matcher refers to a label whose value drifted. The metric exists, the rule file is correct, but the combination the expression expects does not match the live series. This is the same cause as the Stage-2 cause in lesson 01; lesson 01 looks at it from the alert’s perspective, this lesson looks at it from the rule file’s perspective.

The second most common cause is at Shape 1 (parse time): a YAML indentation error or a missing quote drops every rule in the file. This shape is loud and fast to find; teams catch it on the first promtool check rules run after merge.

The third most common cause is at Shape 4 (cardinality): a rule added without a by clause or with a by clause that includes labels with high cardinality (per-request labels, per-user labels). This is the slowest to find because the initial symptom is a noisy alert, not a non-firing one.

Under the hood

The lifecycle of a rule expression:

1. YAML parse. The whole file is parsed into Go structs. Indentation, quoting, and groups/rules/expr chain validity are checked. Errors here fail the whole file, not just one rule.
2. PromQL parse. Each rule’s expression is turned into an AST. Function names, argument types, and aggregation grammar are validated. Errors here fail the offending rule only.
3. Evaluation. On every evaluation_interval, each expression is bound to the live metric set. The result is a vector. Series with positive state advance through the state machine.
4. State machine advance. Per-series state updates according to the for: clause.

promtool check rules runs steps 1 and 2 against the rule file. promtool test rules runs steps 1, 2, and 3 against unit-test fixtures. promtool query instant runs step 3 against the live Prometheus. None of the three catch all shapes; you need at least two to validate a rule.

How to configure it

The configuration of a rule wrong is mostly about how the rule is written. The annotation pattern matters less than the expression itself. Real annotated rule with the diagnostic hooks needed to catch the four shapes:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"5..",
                environment="production",
              }[5m]
            )
          )
          /
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"2..|3..|4..|5..",
                environment="production",
              }[5m]
            )
          )
          > 0.05
        for: 5m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout error rate above 5% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The diagnostic hooks:

• The expression has a by (service, region) clause that names the labels the rule summarises by. Without it, sum() drops all labels and returns a single series; with a missing by clause in the wrong place, the rule summarises everything instead of the intended set.
• Both numerator and denominator carry the same set of matchers (job="checkout-svc", environment="production"). A drift between the two produces a rate calculation against mismatched denominators; the team may end up with rates above 100% or undefined behaviour.
• The comparison is > 0.05 (literal > HTML-escaped for MDX, which renders as > in the YAML). A flipped comparator (< 0.05) silently inverts the alert.

The unit-test pattern that catches the four shapes:

rule_files:
  - /etc/prometheus/rules/checkout.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout-svc",code="500",environment="production",region="us-east-2",service="checkout"}'
        values: '0+0x100 100+0x100 100+0x100'
      - series: 'http_requests_total{job="checkout-svc",code="200",environment="production",region="us-east-2",service="checkout"}'
        values: '1000+0x100 1000+0x100 1000+0x100'
    alert_rule_test:
      - eval_time: 5m
        alertname: CheckoutHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              team: checkout
              service: checkout
              region: us-east-2
            exp_annotations:
              summary: 'Checkout error rate above 5% in us-east-2'
              runbook: 'https://runbooks.example.com/checkout/high-error-rate'

A unit test that covers three intervals plus the expected alert catches shape 2 (the metric exists), shape 3 (the empty result returns no alert), and shape 4 (the cardinality of the return is bounded). Shape 1 fails at the YAML parse step and does not reach the test body.

How to validate it

Three validation steps. Run all three.

Step 1: parse-time check.

promtool check rules /etc/prometheus/rules/checkout.yml

Output:

SUCCESS: rule files validated; 14 rules found, 0 errors

A parse error includes the offending line:

err: yaml: line 42: did not find expected key
SUCCESS: rule files validated; 13 rules found, 1 errors

Step 2: unit tests.

promtool test rules /etc/prometheus/rules/checkout_test.yml

Output:

SUCCESS: 1 tests, 14 rules, 12 unbound, 0 errors

A unit-test failure includes the expected vs received alert and the eval_time at which it diverges. Run these in CI for every rule file.

Step 3: instant query against the live expression.

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'sum by (service, region) (
     rate(
       http_requests_total{
         job="checkout-svc",
         code=~"5..",
         environment="production",
       }[5m]
     )
   )'

The result is the vector the rule will evaluate against. An empty result with the metric confirmed present in storage is a shape-3 defect.

Then the rule’s actual state:

curl -s http://prometheus:9090/api/v1/query?query=ALERTS \
  | jq '.data.result[] | select(.metric.alertname=="CheckoutHighErrorRate")'

A rule that loads and returns no vector has no entry in ALERTS. A rule that returns a vector with the right labels but never enters firing has a for: mismatch (covered in lesson 04).

How it can fail

Six failure shapes, each tied to a defect shape:

1. Parse-time YAML indentation. A two-space indent breaks to four spaces on line 27 of a rule file. Every rule in the file drops. Symptom: promtool check rules reports a YAML error and 0 rules loaded.
2. Function name typo. histogram_quantile(0.95, ...) becomes histogram_quantilie(0.95, ...). The PromQL parser does not recognise the function name. Symptom: promtool check rules reports unknown function and the rule fails to load.
3. Label matcher typo. job="checkout-svc" becomes job="checkouts-svc" (extra s). The expression parses, evaluates, and returns []. Symptom: promtool check rules reports SUCCESS, but promtool query instant against the same expression returns [].
4. Aggregator without by. sum(rate(...)) without a by clause summarises everything in the metric to a single series. The alert fires once for the entire population. Symptom: the alert fires on a label set with one series and the labels include job="..." only.
5. Rate window too short. [1m] on a metric that has natural smoothing over a longer window. The rate calculation underestimates by a factor of the smoothing window. Symptom: the rule fires intermittently on short-burst noise and does not fire on the sustained condition.
6. Comparison constant off. > 0.5 for a metric whose natural range is [0, 1] and 0.05 is the intended threshold. The rule fires only on extreme incidents. Symptom: the team sees a fire during a partial outage and misses the early signal.

How to troubleshoot it

Follow the six-step diagnostic. The order matters because each step catches a different defect shape.

1. Step 1, check rule file. promtool check rules. Catches shape 1 (parse) and shape 2 (function name). If this step fails, fix the rule file. If it passes, move on.
2. Step 2, run the expression. promtool query instant. If the result is empty, the rule sits inactive. Look at the live label set with /api/v1/series?match[]=metric_name and confirm the label the rule expects.
3. Step 3, inspect state. ALERTS{alertname=...}. If the rule has no entries and the metric is in storage, the rule is shape 3 (empty result).
4. Step 4, compare the metric’s labels to the rule’s matchers. If the rule expects job="checkout-svc" and the live labels show job="checkout-app", the rule’s matcher is wrong. Edit the rule.
5. Step 5, count the returned series. If the result has more series than expected, the aggregation has the wrong by clause. Confirm the cardinality the rule was supposed to produce.
6. Step 6, run unit tests. promtool test rules. Confirm the test fixtures still match the rule. A test that passes for 0+0x100 but the rule now uses [10m] is a stale test, not a correct rule.

Security implications

A rule wrong is not a security exposure in itself. The security implication is that the alerts that should fire on security-relevant conditions (anomalous authentication, permission grants, payload inspection failures) are not firing. Treat security-critical rules with the same validation discipline as availability rules.

The interaction between rule wrong and Alertmanager silencing is also security-relevant. A rule whose expression is empty produces no firing alerts and therefore no entries in AM’s nflog. A team that audits AM for “alerts that should have fired but did not” sees nothing to investigate. The defect is invisible to the audit.

Performance implications

A rule wrong is typically more expensive than the correct version, not less. The rule is evaluated every evaluation_interval regardless of whether it fires; a malformed expression whose aggregation has no by clause evaluates against every series in the TSDB every minute. The cost is bounded by the rule’s evaluation interval and the underlying metric’s cardinality, but it is paid even when the rule does nothing useful.

A unit test in CI is cheap (milliseconds per fixture). A full end-to-end test against the live Prometheus is more expensive (one HTTP round-trip per fixture). The right mix is unit tests in CI plus a quarterly instant-query audit against every production rule.

Production guidance

• Run promtool check rules in CI for every rule file change. Reject merges that fail.
• Run promtool test rules in CI for every rule with at least three test intervals. A test that covers the breach boundary (just below threshold, just above) catches the most expensive defect shapes.
• Run promtool query instant against every production rule on a quarterly schedule. Store the result; a rule whose instant query returns [] for two quarters is a retirement candidate.
• Pin rule expression with at least one labelled example per new contributor, so the by clause and the comparison constant are explicit.

Verification

You should now be able to answer:

• What are the four shapes of a rule-wrong defect?
• Which two validation steps together catch all four shapes?
• How do you distinguish an empty-result defect (shape 3) from a missing-telemetry defect (lesson 02)?
• Why is the by clause of an aggregator a security-relevant configuration knob?

Quiz