Units and Thresholds

Two engineers stare at the same Grafana panel. One says “latency is up”. The other says “no it isn’t”. They have the same data. The disagreement is units — one panel renders nanoseconds, the other renders seconds. Neither is wrong; both are reading the wrong thing.

A single source of unit and threshold conventions stops that argument. Conventions are short — one page — and the absence of them costs hours per incident.

What it is

Two related decisions on every panel:

• Unit — the formatter that Grafana applies to the numeric value before it reaches the tick labels. Examples: s (seconds), bytes, reqps (requests per second), percent (0–1 → 0–100), ms (milliseconds), binode (binary bytes).
• Threshold — the colour band that the panel uses to flag out-of-bound values. A step has a colour and a value. The panel paints the cell green, yellow, or red as the value crosses each step.

The threshold semantics come in two flavours:

• Absolute — the threshold value is a constant. “Anything above 1 second is red”.
• Percentile — the threshold value is computed from the current series distribution. “Anything above the 95th percentile for this series over the last hour is red”.

The unit and threshold choices are not cosmetic. They decide whether the panel is read as a number or read as a status.

Why a sysadmin cares

Three operational costs come from inconsistent units or missing thresholds:

• Misread incident — a 5xx rate panel that omits percent renders 0.003 instead of 0.3%. The on-call engineer reads it as “tiny”, and never pages.
• False-page incident — a CPU saturation panel with a fixed threshold of 80% will fire on small hosts and stay silent on large hosts. The alert and the human disagree about what “bad” means.
• Eye-scan tax — a 30-panel dashboard without threshold colours is a wall of numbers. The operator must read every axis. A 30-panel dashboard with thresholds turns into 30 status lights, readable in one second.

The thresholds link to alerts in a way many teams miss: the alert rule and the panel should use the same number. If the panel says “red above 1%” and the alert says “fire above 5%”, the operator sees red on the panel without a paging event, and assumes something is wrong with the alert.

How it works

Two layers sit behind the rendering:

1. Unit — Grafana has a built-in catalog of unit formatters. Each formatter knows how to convert the raw value (which is always in the base unit — seconds, bytes, requests, percents-as-0–1) into a human label. bytes renders 1048576 as 1.00 MiB. reqps renders 0.5 as 0.5 req/s. percent renders 0.4 as 40%. s renders 12.345 as 12.345s.
2. Threshold steps — a sorted list of {color, value} pairs in fieldConfig.defaults.thresholds.steps. The first step whose value is less than or equal to the data point determines the colour.

  panel value (latency, seconds)
              |
   ------+---+-----+---------+---------
          |   |     |         |
          |   |  green <=0.5 | red > 1.0
          |  yellow (>0.5 <=1.0)
          |
        default (null)
        no threshold

Absolute thresholds use the panel value directly. Percentile thresholds first request the series from the data source, then Grafana computes the percentile from the returned samples and applies the step values against the percentile boundary rather than against the raw value.

How to configure it

A fieldConfig.defaults block on a service-tier panel, configured for latency with a single threshold step and a percentile mode for a per-route baseline:

{
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "decimals": 3,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.5 },
          { "color": "red", "value": 1.0 }
        ]
      }
    },
    "overrides": [
      {
        "matcher": { "id": "byName", "options": "p99 baseline" },
        "properties": [
          { "id": "thresholds", "value": {
            "mode": "percentile",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 0.9 },
              { "color": "red", "value": 0.99 }
            ]
          }}
        ]
      }
    ]
  }
}

Matching Prometheus alert rule (so the panel colour and the alert fire on the same boundary):

groups:
  - name: checkout-latency
    interval: 30s
    rules:
      - alert: CheckoutLatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum by (le) (
              rate(http_server_requests_seconds_bucket{
                service="checkout"
              }[5m])
            )
          ) > 1.0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "checkout p99 above 1s for 5m"
          runbook_url: "https://runbooks.example/checkout-latency"

The alert expression uses > HTML entity in the YAML — the literal > sign breaks YAML block-scalar interpretation. The Grafana side renders the panel red on the same boundary so the operator’s eye and the page match.

A panel that should not alert — the deployment-cadence chart — just omits thresholds. The chart renders blue lines and the operator reads trends by eye.

How to validate it

# READ-ONLY: confirm a panel reports data and is over a threshold
curl -s -u admin:admin \
  http://grafana:3000/api/ds/query \
  -H 'content-type: application/json' \
  -d '{
    "queries": [{
      "refId": "A",
      "datasource": { "type": "prometheus", "uid": "prom-prod" },
      "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket{service=\"checkout\"}[5m])))"
    }],
    "from": 1700000000,
    "to":   1700000600
  }' | jq '.results.A.frames[0].data.values'

Expected output:

[[1700000030,1700000090,1700000150], [0.42,0.78,1.21]]

A value of 1.21 is above the red step. Open the dashboard and confirm the panel cell renders red. If it renders green, the threshold is set to absolute but the panel received the value in milliseconds (0.00121) — the unit mismatch shifts the visual colour.

# READ-ONLY: confirm alertmanager picked up the same boundary
amtool alert query --alertmanager.url=http://alertmanager:9093 \
  --filter=alertname=CheckoutLatencyHigh

The expected output lists CheckoutLatencyHigh with state firing if the panel is red.

How it can fail

The high-frequency failure modes:

1. Unit mismatch between data source and panel — Prometheus returns seconds, the panel unit is ms, so 0.5s renders as 500ms. The threshold steps were written assuming s, so the red boundary at 1.0 never fires. Symptom: panel visual is “all green” while the alert fires; or “all red” while the alert is silent.
2. Threshold value lacks null sentinel — the first step has no null value, so any point below the lowest threshold has no colour and renders as grey. Symptom: half the panel is grey, the other half is the lowest colour. Fix: the first step must have value: null.
3. Percentile threshold with sparse data — fewer than ~30 samples in the lookback window make the percentile unstable. Symptom: panel flips between green and red every 30 seconds. Switch to absolute or extend the window.
4. Drift between alert rule and panel threshold — alert fires at >0.5, panel turns red at >1.0. Symptom: panel stays green until well after the page fires, so the operator learns to ignore the visual. Audit promtool alert lint and the panel JSON against the same file.
5. Wrong unit name — seconds instead of s, bytes with a rate series (rate is unitless until you pick a unit). Grafana silently accepts unknown units and renders raw numbers. Symptom: panel renders like 0.000001 with no formatter. Audit fieldConfig.defaults.unit.
6. Percentile mode toggled in a query that returns one point — the percentile calculation divides by zero or returns NaN. Symptom: panel renders empty cells. Use absolute mode for single-point series.

How to troubleshoot it

The diagnostic order:

1. Click the panel and inspect the data point. Hover at the same instant the alert fired. If the panel value does not match the alert value, the unit or the threshold is wrong.
2. Cross-check the unit. The panel JSON has fieldConfig.defaults.unit. The Prometheus metric has a suffix (_seconds, _bytes, _total). Confirm they agree.
3. Cross-check the threshold against the alert rule. Find the alert YAML in prometheus_alert_rules.yaml. The number in &gt; <value> should equal the threshold step value.
4. Toggle the panel to absolute mode briefly. If the colour becomes stable, the percentile mode was the problem (not enough data).
5. Reload the dashboard. Re-running the query clears interpolated state from a transient condition.

Security implications

• Unit names are advisory only. A user can rewrite the unit field in any editable panel. Provisioned dashboards that need consistent rendering should set editable: false.
• Thresholds can leak business thresholds. A published dashboard that publishes “checkout fails when error rate is above 0.5%” leaks the SLO. Public dashboards should use generic panels; the SLO thresholds stay on the internal tier.
• Alert rule fingerprints in commit history are public if the runbook repo is public. Review the alert YAML carefully before publishing.

Performance implications

• Threshold calculation runs per data point, on the browser. The cost is linear in the series count. A panel with 10,000 series and 4 steps is O(40,000) per refresh — fine on modern hardware, slow on a phone.
• Percentile mode requires the panel to ask the data source for raw samples, not aggregated values. With Prometheus this means wider le buckets; with Loki it means more log lines. Confirm the storage cost before turning on percentile mode on a panel that runs every 30 seconds for 30 days.
• Decimal place rendering has no measurable cost. Set decimals: 3 and accept the formatting.

Production guidance

• One page in the runbook that lists every dashboard unit and every SLO threshold. Pin a link in the team channel.
• PromQL alert rules and Grafana panels in the same repo, with a CI step that checks the threshold numbers match.
• Use promtool alert lint and a panel lint (the dashboard-linter Grafana plugin) in the same pipeline.
• For service-tier percentile panels, default to a 30-day window. For burst-tier panels, default to a 5-minute window.
• Every panel ships with value: null as the first step — encode it in the dashboard template.

Verification

You should now be able to answer:

• What does the unit field on a Grafana panel actually change?
• When do absolute thresholds fail and percentile thresholds work?
• Why must the first threshold step include value: null?
• How do you reconcile a panel threshold with the matching alert rule so the colours and the pages agree?

Quiz