Threshold Wrong

The rule loads. The expression evaluates. The series comes back with the labels the team expects. The alert does not fire during the incident the team bought it to catch. The postmortem names the threshold; the threshold was set six months ago from a sample and the system has moved on. The team has a rule that is correct in shape but wrong in magnitude, and the mis-set value is the only thing between the alert firing and the page landing.

This lesson is that magnitude. The expression is sound. The labels match. The rule file is fine. The comparison constant in expr: is wrong against the metric’s actual range. The fix is to derive the threshold from the observed distribution, not from a guess.

What a threshold wrong is

A threshold wrong condition in production terms is an alert rule whose comparison constant (the value after >, <, ==, !=, >=, <=) does not correspond to the level of the metric the team intends to alert on. Three conditions hold:

1. The rule file has loaded.
2. The expression evaluates without error and returns a vector with the expected labels.
3. The vector crosses the threshold inconsistently with the team’s intended firing condition.

The third condition is operational, not technical. A threshold is wrong if it does not match what the team believes is happening. A threshold is correct if it does.

Two adjacent concepts to be specific about:

• Mis-set against the natural range. The threshold is far above or below the metric’s typical range. A http_request_duration_seconds whose natural range is [0.01, 0.5] is alerted on at > 5; the rule fires only on the catastrophic end and is silent on the slow drift.
• Tied to a unit mismatch. The threshold’s unit does not match the metric’s unit. A 1 GiB threshold compared against a value reported in 1 MiB; the rule fires only on a 1024x overshoot. The opposite direction silently keeps the alert dead.

The first is the most common shape; the second accounts for roughly one in twenty of the threshold-wrong cases the team reviews.

Why a sysadmin cares

A threshold wrong is the silent sibling of the rule wrong. The expression is correct; the team trusts the rule. The metric’s natural range shifts under the rule (because traffic grows, the service’s expected latency changes, the deploy alters the baseline) and the threshold does not move with it. The alert fires on less and less of the population until it does not fire at all.

The cost is a slow devaluation. The team believes the alert is live. The alert is live technically. The alert is informative only on the catastrophic end. The first time the team notices, the threshold has been wrong for a quarter.

The second failure shape is the noisy rule. The team raises a threshold to > 0 (the default literal in many rule templates) against a metric whose range includes 0 for healthy periods. The alert fires on every minor dip; Alertmanager’s group logic catches most of it, but the underlying metric is wrong about what is healthy. This is the failure shape of the boilerplate template most often.

Both costs are avoided by deriving the threshold from the metric’s distribution over a representative window.

How it works

A threshold is a constant in the rule’s comparison. It is interpreted in the metric’s unit. It advances the state machine when the metric’s instantaneous value crosses it, subject to for:. The cost of getting it wrong scales linearly with how far from operational truth it is.

+-------------------------+    +-------------------------+
| Shape A: too tight      |    | Shape B: too loose      |
| threshold << natural    |    | threshold >> natural    |
| range midpoint          |    | range midpoint          |
+-----------+-------------+    +-----------+-------------+
            |                              |
            v                              v
+-------------------------+    +-------------------------+
| Page-ware: fires on     |    | Silent: fires only on   |
| noise; team silences or |    | catastrophic end; team  |
| disables the rule       |    | trusts a rule that is   |
|                         |    | not informative at the  |
|                         |    | service's range         |
+-------------------------+    +-------------------------+
            |                              |
            +-------------+----------------+
                          |
                          v
+-------------------------------------------------+
| Symptom: rule in ALERTS, but only at the wrong  |
| end of the metric's range                      |
+-------------------------------------------------+

The four production shapes, each with the diagnostic observable:

• Shape A (too tight, default 0). > 0 against a metric whose natural minimum is below zero (latency is always positive but a counter rate can be zero), or a gauge whose minimum is also zero (queue depth). The alert fires on every non-zero sample.
• Shape B (too loose, environment-specific default). > 80 against a metric whose typical range is [0, 60]. The alert fires at 80 only on extreme conditions.
• Shape C (unit mismatch). > 1 against a metric whose natural range is [0, 1] but values are reported in percent. The alert fires at the catastrophic end of the wrong scale.
• Shape D (drifted against the natural range). A threshold set six months ago against a 30 p99 latency; the service has been redeployed, the natural range is now [0.05, 0.4]. The threshold sits above any realistic value.

The most common cause

In roughly half of the threshold-wrong investigations the team reviews, the cause is Shape A (default left in place). A rule template was copied, the placeholder threshold (0, 80%, 1) was not replaced, and the rule shipped with a no-op or a page-ware threshold. The shape is distinctive: the threshold is a round number typical of a template default, and the rule has never fired or has fired every minute.

The second most common cause is Shape D (drift). A threshold was set against an early baseline; the system grew; the threshold was not revisited. The shape is distinctive: the threshold is precise (not a round number), the rule fires rarely, and the postmortem finds a metric whose range has clearly moved past the threshold.

Under the hood

The threshold in a Prometheus rule is a literal constant in the right-hand side of a comparison. It is interpreted in the metric’s unit and applied per series. Each series has its own state machine; the threshold is the gate that initiates pending. There is no smoothing, no hysteresis, and no automatic adjustment built into the expression.

The discipline that makes a threshold accurate is distribution awareness: a representative sample of the metric over a quiet period plus a representative sample during a known incident. The threshold sits between the two. This is the same discipline the SRE workbook calls “alerting on SLOs”; the rule’s purpose is to fire when the SLO is at risk of being missed, not when the metric is at a number the team picked from a template.

For latency metrics, the relevant distribution function is histogram_quantile(0.99, sum by (le) (rate(...[5m]))). For success ratios, the relevant computation is sum(rate(success[5m])) / sum(rate(total[5m])). The comparison constant is then derived from the SLO and the alert’s purpose (warning vs page).

How to configure it

A rule with a defensible threshold is one whose comparison constant matches the metric’s range at the boundary between “informative” and “catastrophic”. Real annotated rule with the threshold-derivation pattern:

groups:
  - name: checkout.rules
    interval: 30s
    rules:
      - alert: CheckoutHighErrorRate
        # Threshold derived from SLO: 99% of checkouts
        # must succeed. The rate of 5xx is alertable at
        # 1% (giving 30 minutes of error budget at the
        # page tier) and at 5% (giving 6 minutes).
        expr: |
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"5..",
                environment="production",
              }[5m]
            )
          )
          /
          sum by (service, region) (
            rate(
              http_requests_total{
                job="checkout-svc",
                code=~"2..|3..|4..|5..",
                environment="production",
              }[5m]
            )
          )
          > 0.01
        for: 5m
        labels:
          severity: page
          team: checkout
        annotations:
          summary: 'Checkout error rate above 1% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/checkout/high-error-rate'

The > 0.01 is the SLO-derived threshold; the rule’s purpose is to page at the point where the team has roughly thirty minutes of error budget remaining.

A rule with a distribution-derived threshold uses histogram_quantile against a chosen quantile:

- alert: CheckoutP99LatencyHigh
  expr: |
    histogram_quantile(
      0.99,
      sum by (le) (
        rate(
          http_request_duration_seconds_bucket{
            job="checkout-svc",
            environment="production",
          }[5m]
        )
      )
    ) > 0.5
  for: 10m
  labels:
    severity: page
    team: checkout
  annotations:
    summary: 'Checkout p99 latency above 500ms'
    runbook: 'https://runbooks.example.com/checkout/latency'

The > 0.5 is the threshold; histogram_quantile(0.99, ...) is the distribution function. The threshold sits where the SLO and the latency budget intersect. A drift in the natural latency range requires revisiting the constant; the histogram structure continues to work.

How to validate it

Three steps. Run all three before the rule ships.

Step 1: distribution over a representative window.

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'histogram_quantile(
     0.99,
     sum by (le) (
       rate(
         http_request_duration_seconds_bucket{
           job="checkout-svc",
           environment="production",
         }[5m]
       )
     )
   )'

Run against a quiet week, a typical week, and a known incident. The values form the range the threshold should sit within.

Step 2: range-query over the last 24h.

curl -s 'http://prometheus:9090/api/v1/query_range?query=histogram_quantile(0.99,sum%20by(le)(rate(http_request_duration_seconds_bucket{job=%22checkout-svc%22,environment=%22production%22}[5m])))&start=2026-08-13T00:00:00Z&end=2026-08-14T00:00:00Z&step=60' \
  | jq '.data.result[] | {series: .metric, samples: [.values[] | {t: .[0], v: .[1]}]}'

Export the result to CSV; draw the distribution. The threshold line should sit just above the natural peak of healthy traffic and below the inflection the team believes is the incident-start signal.

Step 3: alert history check.

curl -s http://prometheus:9090/api/v1/query?query=ALERTS \
  | jq '.data.result[] | select(.metric.alertname=="CheckoutP99LatencyHigh") | .metric.alertstate'

A rule that fires only on rare catastrophic events is too loose; a rule that fires every minute is too tight; the truth is in the firing pattern of the last quarter.

For a more rigorous check, run the rule against promtool test rules with three fixtures: one below the threshold (rule should not fire), one just above the threshold (rule should advance through for:), and one far above (rule should fire immediately).

How it can fail

Six failure shapes, each tied to a defect shape:

1. Default 0 left in place. A template-provided threshold that no one replaced. Symptom: the rule fires on every non-zero sample and pages the on-call constantly.
2. Threshold tied to a unit mismatch. A > 1 against a value in percent produces a 100x looser threshold than intended; the same constant in the opposite direction produces a 100x tighter one. Symptom: the alert either never fires or fires every evaluation.
3. Threshold from another team’s rule. A threshold copied from a similar-looking service whose metric range differs. Symptom: the rule fires rarely on the host it is meant to monitor and never fires on the host whose range is wider.
4. Threshold at the natural midpoint. A median guess: > 0.05 against a typical range [0.001, 0.06]. The rule is informative on the upper bound of normal traffic. Symptom: the rule fires twice a day on traffic the team calls healthy.
5. Threshold never revisited after a deployment. The threshold was set against a baseline that has since halved. Symptom: the rule fires half as often as the team expects; the postmortem observes the threshold sits above the new natural peak.
6. Threshold at the maximum (1 for a ratio, 100 for a percent). A ceiling rule that only fires when the metric is pinned at the ceiling. Symptom: the alert is dead for the service’s actual range and only fires on the catastrophic end.

How to troubleshoot it

Follow the order. Six steps.

1. Step 1, confirm the expression is sound. If the expression itself is shape-3 (empty result) or shape-2 (function error), that is lesson 03, not this lesson. Validate that first.
2. Step 2, plot the metric over 24h. A range query against the underlying metric, exported to CSV, plotted in Grafana. The threshold is drawn as a horizontal line. The line should sit between the natural peak and the incident inflection.
3. Step 3, derive a candidate from the SLO. For an availability SLO of 99.9%, the alertable error rate is roughly 1 / 9 of the budget per 30 minutes plus the burn rate the team has decided is actionable. For a latency SLO of p99 < 200 ms, the threshold is 0.2.
4. Step 4, test against fixtures. promtool test rules with three fixtures (below, just above, far above). The rule should advance, not skip states.
5. Step 5, observe a quarter of production behaviour. The ratio of “rule fired” to “incident confirmed” should be roughly 1:1. Anything else is a mis-set threshold.
6. Step 6, re-derive on a quarterly cadence. Thresholds drift. Re-plot the metric, re-derive the candidate, and commit the change as a routine maintenance edit.

Security implications

A threshold tied to a security-relevant metric (anomalous authentication attempts, rate of failed permission grants, volume of payload inspection failures) must be tighter than the equivalent availability threshold. The risk of firing is low compared with the risk of missing the signal. The rule must also be reviewed against the SLO budget to ensure it does not consume the budget for incidents that are not security-relevant.

For metrics whose values are sensitive (an authentication attempt rate that maps onto the user base) the threshold itself is not security-relevant, but the labelled alert that fires when the threshold is crossed is. Treat alert labels as a leakage channel and strip customer identifiers from the annotations.

Performance implications

A threshold that is too tight produces more firing alerts, more notification traffic, and more load on AM’s grouping machinery. The cost is small per alert but scales linearly with the firing rate. A threshold that is too loose produces fewer alerts but fails to deliver value; the team has paid the engineering cost of the rule without the operational benefit.

The right discipline balances alerting cost against incident detection latency. A reasonable target is roughly one real incident per five alert fires; above that, the threshold is too loose, and below, the threshold is too tight.

Production guidance

• Derive each threshold from an SLO or an observed distribution. Do not copy a number from another team’s rule without verifying against the local metric.
• Re-derive on a quarterly cadence. Thresholds drift; the team that does not revisit them accumulates silent misconfiguration.
• Track the ratio of “rule fired” to “incident confirmed”. A ratio above 1:5 means the threshold is too loose or the condition is too rare to warrant an alert at all.
• Pair every threshold with a fixture test in CI. The test covers the boundary (just above threshold) and the calm-period (well below).

Verification

You should now be able to answer:

• What are the four shapes of a threshold-wrong defect?
• Which shape accounts for the largest share of threshold-wrong cases?
• How do you derive a defensible threshold from an observed distribution?
• Why is raising a threshold the wrong first response to a recurring false positive?

Quiz