Threshold Too Low

A webhook intake service runs at 50% CPU most of the day. A new alert rule appears: WebhookIngestHighCPU > 60%. The alert fires on day one and stays firing for the rest of the week. The on-call engineer pages, opens Grafana, finds nothing wrong. The team reduces the threshold to > 30%. The alert now fires constantly. They raise it to > 80% and forget about it.

Six months later, a runaway consumer thread pegs the CPU at 95%. The alert was at 80%; it should have fired, but the expressions grouped by container and the runaway thread was inside a sidecar the rule ignored. The alert was right in principle, but it had been lowered into noise, raised into silence, and finally broken by aggregation drift. The team paged for an hour on a manual symptom.

That is a threshold tuned by opinion, and the consequences follow.

What “threshold too low” means

A threshold is the numeric boundary embedded in the rule expression. It is the line > 0.05 in ... > 0.05. A threshold is “too low” when normal traffic crosses it often enough that the alert becomes noise, while the same threshold is too low in the opposite sense when it is set so close to the normal band that legitimate transient spikes out of the band look identical to a real incident.

The shape is wrong in two directions. The typical failure is “too low” (constant firing), but a threshold can also be “too high” (never fires when it should). The lesson’s vocabulary covers both: too low means the alert fires when the service is fine; too high means the alert does not fire when the service is broken. The diagnostic shape for “too low” is what the series looks like around the threshold.

Why a sysadmin cares

A threshold that is too low produces three operational pains:

• The page is uninteresting. The on-call engineer opens the alert, looks at the dashboard, sees a number, sees nothing, and learns to dismiss this rule.
• The signal pool is diluted. When every page looks the same, the rare real incident is harder to find in the PagerDuty feed.
• The threshold becomes harder to change. Once a rule fires often, raising the threshold feels like a fix; the underlying cause (the rule fires against normal traffic, not against failures) is forgotten. A year later the threshold is > 95% and a real saturation event at 88% goes unnoticed.

Setting the threshold from observed data prevents all three.

How the threshold is supposed to be chosen

The right way to pick a threshold is to look at what the metric actually does for a representative window and place the line at a percentile that catches real failures without catching normal variation.

observed distribution
       ^
       |
count  |  *        *
       | *  *      * *
       |*    *  *  *  *  *
       |             [P95]
       +-------------------------------------> metric value
       0           threshold         ^
                                   real
                                  incidents
                                  live above
                                  this line

The threshold sits above the 95th percentile of the normal band, below the lower bound of past incidents. The shape of the distribution comes from observing the metric across a representative window — typically the last 7 days, with 30 days used for services with weekly or seasonal patterns.

The math is not the whole story. Two services may share a threshold and mean different things by it. A webhook ingest service that processes 50 events per second and a transaction processor that handles 5,000 per second are not “high” at the same CPU percentage. The threshold is per-service, sometimes per-population (region, environment, tier).

Where thresholds come from that are too low

The most common shape is a copied number. A blog post or community rule sample sets > 0.5% error rate, > 80% CPU, > 200ms latency. The team copies it into a rule for a service whose normal band is 4% error rate, 35% CPU, 80ms latency. The threshold sits below normal traffic and the rule fires every day.

The second most common shape is a threshold carried over from an older version of the same service. A service that was single-region ten months ago had a tight threshold. When the service moved to multi-region with shuffle sharding, the traffic profile changed. The threshold did not follow.

The third shape is misreading the metric’s units. A counter-derived rate that returns a value 0.005 (a value not in percentage) is treated as if it were 0.5%. The threshold is then set to > 0.01. The rule fires on any non-trivial traffic.

Under the hood

A Prometheus alerting rule evaluates every interval, against the same expression body, against the same set of series the expression selects. The threshold is part of the expression body; it is a numeric literal in the PromQL. The rule has no memory of past values, no rolling window, no quantile; the for: clause controls timing, not magnitude.

Two consequences follow:

1. The threshold is only as good as the series set the expression selects. A threshold that is calibrated against service=checkout is meaningless if the rule inadvertently also selects series with service=checkout-canary.
2. The threshold is only as good as the PromQL function used. > 0.05 against rate(...[5m]) is a smoothed fraction, not a single-sample value. > 0.05 against an instantaneous metric is a single-sample decision. The two thresholds mean different things.

The right threshold for rate(...[5m]) is well below the failure mode’s lower bound, because the rate window already smooths. The right threshold for count(...)>0 for “any error at all” is the value at which a real error is unlikely to be present in any single sample.

How to configure it

For a service whose current threshold misfires, the change is small and explicit. The shape of a healthy threshold block is:

groups:
  - name: webhook.rules
    interval: 30s
    rules:
      - alert: WebhookIngestHighCPUSustained
        expr: |
          avg by (service, region) (
            rate(node_cpu_seconds_total{
              mode!="idle",
              job="webhook-ingest",
              environment="production",
            }[2m])
          )
          > 0.75
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: 'Webhook ingest CPU above 75% in {{ $labels.region }}'
          runbook: 'https://runbooks.example.com/webhook/cpu'
          threshold_basis: 'p99 over 30d window + 5pt buffer'

Three things to read into that:

• rate(...[2m]) smooths two minutes; the threshold is against that smoothed rate, not against a 1-second sample.
• > 0.75 is in the 0–1 range that node_cpu_* produces, not in per-cent. The threshold is 0.75, not 75%. The common mistake is to write > 75.
• threshold_basis is an annotation. It records the evidence used to set the line. Six months from now, when the threshold is questioned, the answer is in the rule file.

How to validate it

Validate by reading the metric, not by reading the rule.

promtool query instant \
  http://prometheus:9090/api/v1/query \
  'avg by (service, region) (
     rate(node_cpu_seconds_total{
       mode!="idle",
       job="webhook-ingest",
       environment="production",
     }[2m])
   )'

That returns the same vector the rule will evaluate. The question: how often does this vector cross 0.75?

promtool query range \
  http://prometheus:9090/api/v1/query_range \
  --query='avg by (service, region) (
     rate(node_cpu_seconds_total{
       mode!="idle",
       job="webhook-ingest",
       environment="production",
     }[2m])
   )' \
  --start=2026-08-01T00:00:00Z \
  --end=2026-08-08T00:00:00Z \
  --step=60s

Read the series in Grafana. Compute the P95, the P99, the max. Decide whether 0.75 sits above the normal band for the right amount. If the P95 is 0.6, 0.75 is reasonable. If the P95 is 0.78, the threshold needs to lift to 0.85 or above. If the P95 is 0.45, the threshold is fine.

Run promtool check rules on the rule file before shipping:

promtool check rules /etc/prometheus/rules/webhook.yml

SUCCESS: rule files validated; 8 rules found, 0 errors

How it can fail

Six distinct shapes, each with the symptom that distinguishes it from other threshold failures:

1. Threshold below the P50. A service whose CPU median is 0.55 cannot host a > 0.50 alert without firing for half the day. Symptom: the alert has been firing, by inspection, for most of the past 7 days.

2. Threshold calibrated for rate but used on a derivative. A rule that uses delta(...[5m]) (which subtracts) gets treated as if it were rate(...), and the threshold is set to a sensible-looking value that has the wrong units. Symptom: the alert fires at boundaries (restarts, scrape gaps) but stays silent during normal traffic.

3. Threshold copied from another service without re-reading. The > 0.05 error rate rule was originally written for a transaction processor; it now guards a search service whose normal 5xx rate is 0.4%. Symptom: the alert has fired every day for the past month with “no incident found” in the close-out notes.

4. Threshold inside the warning level. A team’s runbook sets warning at 60% and page at 60% because the alert threshold was forgotten. Symptom: the team treats pages as warnings and ignores them.

5. Threshold raised into noise. The team raised the threshold from > 80% to > 95% because the rule fired every Thursday during batch processing. They missed a real 93% saturation event a month later. Symptom: the rule fires less often than weekly.

6. Threshold ignores time-of-day. A service has nightly-batch CPU peaks at 0.78. The threshold is > 0.70, which fires every night at 02:00. Symptom: the alert fires only during scheduled job windows.

How to troubleshoot it

1. Open the rule. Read the threshold value and the PromQL function it is compared against. Confirm the units match (rate() returns seconds-per-second for CPU, not per-cent).
2. Read the metric across the last 7 days. Compute the P50/P95/P99/max. Read the P95 and P99 in the same units as the threshold.
3. Read the metric across the last 30 days if the service has weekly or seasonal patterns (batch jobs, end-of-month rolls). Threshold-calibrated against 7 days will misfire on 30-day events.
4. Identify incident-correlated spikes. The boundary between “the service is fine” and “the service is broken” sits at the lower envelope of incident spikes, not the upper envelope of normal variation.
5. Set the threshold at that line. Add a threshold_basis annotation. Run promtool check rules. Ship as a config change with the same canary pattern as a deploy.

Security implications

A threshold on a security-relevant metric (failed logins, denied ingress, certificate age) is more important than a threshold on a resource metric. A threshold set too high on authentication failures is itself a security defect; a false positive on failed-logins costs a page, a false negative on failed-logins lets brute-force attempts pass.

Do not borrow thresholds from external sources for authentication, authorisation, or audit metrics. Compute them from the platform’s own observation history.

Performance implications

A threshold on a counter-derived rate carries no extra cost beyond the expression evaluation. A threshold on a histogram quantile (histogram_quantile) is more expensive, because the quantile must be recomputed across the bucket set every interval. A threshold on a high-cardinality expression is even more expensive: every selected series pays the evaluation cost. Validate that the threshold is set against an expression whose selected series set is bounded.

Production guidance

• Compute thresholds from 7-day and 30-day distributions, not from blog posts.
• Annotate the threshold basis in the rule file. The future team will need to know why 0.75 was the line.
• Recompute quarterly. A service’s traffic shifts; the threshold should follow.
• Treat a threshold as a hypothesis, not a setting.

Verification

You should now be able to answer:

• Where do thresholds that are too low actually come from in production teams?
• How do you distinguish a threshold-too-low failure from an aggregation failure using observable symptoms?
• Which percentile of the metric’s distribution should sit below a sane page-level threshold?
• Why is a threshold copied from a blog post or another service a guess rather than a calibration?

Quiz