Meaningless Gauges

A panel at the top of the “Checkout Service” dashboard reads 1048576. Below it is a gauge from green to red, but the green is in the centre and the red is at the bottom. Another panel reads 82.3% and the threshold below it reads 0%. A third panel reads 42 with no unit, no threshold, and a tooltip that says “Tracked value”.

The operator glances at the dashboard during an incident. None of those three numbers tells them what to do. They have been trained to ignore the panels, and the panels are now wallpaper in stat-panel form. They are also worse than wallpaper: they consume attention to confirm “nothing to see”, and they train trust in numbers that have no defined meaning.

A meaningless gauge is a panel that displays a numeric value without one of the three properties that make a number operational: a unit, a baseline, or an action.

What it is

A gauge becomes meaningless in production when any of the following is true:

1. The number has no unit. A panel reading 42 could be requests per second, requests per minute, total requests, or the length of a queue. Without a unit, the value is not comparable to memory, to other panels, or to thresholds.
2. The number has no baseline or threshold. A percentage without a target is decoration. 82.3% with a threshold of 0% means the gauge is always green. A percentage without a denominator is even worse: 0.5% of one million is not the same as 0.5% of one hundred.
3. The number is a counter shown as a gauge. Prometheus counters only ever increase. A stat panel that displays http_requests_total directly shows the cumulative count since process start. That number tells the operator nothing about the rate. The rate is rate(http_requests_total[5m]) or irate over a short window, and the rate has units of 1/s.

The third shape is the most common. Teams copy a query from a blog post, paste it into Grafana, and ship. The panel works; the panel is wrong; the panel persists.

Why a sysadmin cares

A meaningless gauge consumes operator attention for no return. During an incident, attention is the bottleneck. The on-call engineer is reading the dashboard under time pressure; each panel they must mentally translate (“is 1048576 bytes or requests?”) is a context switch away from the actual investigation.

The secondary cost is the alert it generates. A panel that shows a counter is also, often, the source of an alert. A threshold of > 1000 on a cumulative counter will fire once at process start and never again. The alert is configured, the alert does nothing useful, and the on-call engineer has muted the alert in their head.

How it works

The three shapes come from three different authoring mistakes:

   Metric exists
        |
   +----+----+----+
   |         |    |
 Counter  Gauge Histogram
   |         |    |
   v         v    v
 Rendered  Rendered Rendered
 as gauge  as %     as gauge
   |         |    |
   v         v    v
 Cumulative  % of  Last-sample
 count since nothing  only
 start      (denominator)
              missing

1. Counter rendered as a stat panel. The panel shows the current value of http_requests_total. The number grows forever. The threshold never trips meaningfully.
2. Percentage rendered without a denominator. The query is sum(rate(errors[5m])) / sum(rate(total[5m])) * 100, which is fine, but the panel is configured with a threshold of 0%, which makes it always green. Or the denominator is up, which makes the percentage “of pods that are up”, which is not what the dashboard claims.
3. Histogram summary value rendered as a gauge. The panel shows histogram_quantile(0.99, sum(rate(http_duration_seconds_bucket[5m])) by (le)). That is a quantile, not a gauge. It does not have a stable value to set a threshold against without knowing the traffic.

The discipline is the same in all three cases: define the unit, define the baseline, define the threshold, and document the action.

Under the hood

Grafana 11.x has two places where the unit of a panel is set:

1. The data source. Prometheus exposition formats send numeric values; the data source does not annotate them with units. The unit is a property of how the team labelled the metric (http_requests_total, http_request_duration_seconds).
2. The panel field options. Each panel has a “Standard options” block with a unit dropdown. Picking reqps (requests per second) tells Grafana to render 42 as 42 req/s. Picking percent (0-100) tells it to render 82.3%. Picking bytes renders 1048576 as 1.0 MiB.

Without a unit, the panel renders the raw number and the operator is left guessing. With the wrong unit, the panel renders a confidently wrong number. The third option, picking short (1k, 1M, 1G suffix without context), is a particular trap: the number reads “1.0M” and the operator does not know whether the M means million, mega, megabytes, or minutes.

Thresholds follow the unit. A threshold of 80 on a panel unitless means 80. A threshold of 80 on a panel in req/s means 80 req/s. The same number on a percentage panel means 80%. Threshold values do not migrate with the unit; the author must redo them when the unit changes.

How to configure it

Counter rendered as a rate

# PromQL. The panel query, with rate and unit comment.
sum by (job) (
  rate(http_requests_total[5m])
)
# Unit hint for the dashboard author: req/s.

In Grafana 11.x, the panel field options:

{
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "unit": "reqps",
      "decimals": 2,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "red",   "value": 1000 }
        ]
      },
      "custom": {
        "drawStyle": "line",
        "lineWidth": 2,
        "fillOpacity": 10
      }
    }
  },
  "options": {
    "legend": { "displayMode": "table", "placement": "bottom" }
  }
}

The threshold (> 1000 req/s) is meaningful. The unit (req/s) is meaningful. The action (“if red, scale out the worker pool”) is in the runbook linked from the alert rule.

Percentage with denominator

# PromQL. The query shows the percentage over a defined
# denominator (total requests), not over an undefined one.
100 *
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
# Threshold: > 1% for 5m.

{
  "type": "stat",
  "fieldConfig": {
    "defaults": {
      "unit": "percent",
      "decimals": 2,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.5 },
          { "color": "red",    "value": 1 }
        ]
      }
    }
  },
  "options": {
    "colorMode": "background",
    "graphMode": "area",
    "textMode": "auto"
  }
}

The threshold of 1% matches the team’s SLO budget (Part III covers SLOs in detail). The colour changes mean something. The action (“if yellow, watch; if red, page”) is in the alert rule.

Tracked value

A panel that reads 42 with no unit and no threshold is not a panel. It is a sensor output displayed in the wrong place. The right discipline is to delete it and let the dashboard say one of two things:

• “This metric is below its SLO” — a stat panel with a threshold that maps to the SLO.
• “This metric is not below its SLO” — the absence of the panel.

A “tracked value” panel that exists only so someone can “see the number” is a billboard for the dashboard author, not an operational tool.

How to validate it

Walk every dashboard in the team’s folder. For each stat or gauge panel:

# READ-ONLY. List all dashboards in the team's folder.
curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  "${GRAFANA_URL}/api/search?folderIds=${FOLDER_ID}&type=dash-db" \
  | jq -r '.[] | "\(.uid)\t\(.title)"'

For each dashboard, dump the panels and check for unitless absolute thresholds:

# READ-ONLY. Pull panel field options for one dashboard.
curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  "${GRAFANA_URL}/api/dashboards/uid/checkout-overview" \
  | jq '
      .dashboard.panels[]
      | {
          title: .title,
          type: .type,
          unit: .fieldConfig.defaults.unit,
          thresholds: .fieldConfig.defaults.thresholds,
          decimals: .fieldConfig.defaults.decimals
        }
    '

Expected output (illustrative):

{
  "title": "Request rate",
  "type": "timeseries",
  "unit": "reqps",
  "thresholds": { "mode": "absolute", "steps": [...] },
  "decimals": 2
}
{
  "title": "Tracked value",
  "type": "stat",
  "unit": null,
  "thresholds": null,
  "decimals": 0
}

The second entry is the meaningless gauge. Either set a unit and threshold, or delete the panel.

For PromQL itself, validate that the query returns a rate, not a counter:

# READ-ONLY. Inspect the raw counter and its rate.
promtool query instant \
  http://prometheus:9090 \
  'sum(rate(http_requests_total[5m]))'

# Output (illustrative):
# http_requests_total{job="checkout"}  42.13

If the panel query is the same value without rate(...), the panel is showing a counter.

How it can fail

Six specific failure shapes:

1. The cumulative counter. A stat panel reading http_requests_total shows 5,283,991. The operator cannot tell if traffic is up or down without a second panel.
2. The unitless percentage. A stat panel reading 42% for “cache hit rate”. The threshold is hard-coded at 80% but nobody can say whether 42% is good, bad, or recent.
3. The wrong unit. A latency panel set to unit: "ms" while the underlying metric is in seconds. The panel renders 0.250 and looks healthy; the real value is 250 ms, which is over the SLO.
4. The short-suffix trap. A panel set to unit: "short" renders 1.0M for one million requests and 1.0M for one megabyte of memory. Two panels with the same rendering, two entirely different meanings.
5. The threshold-by-mistake. A panel whose threshold was set in a unit and never migrated when the unit changed. The threshold reads 1.0M but the panel value is 1.0M of seconds. Threshold trips at the wrong value.
6. The “tracked value” billboard. A stat panel showing vm_uptime_seconds because “someone wanted to see how long the box has been up”. The panel is not actionable. It is not part of an SLO. It is decoration.

How to troubleshoot it

When a panel is wrong, the fix is in this order:

1. Identify the metric. Run promtool query instant http://prometheus:9090 '<metric>'. Confirm whether the metric is a counter, gauge, histogram, or summary.
2. Identify the right rate. For counters, the panel query must include rate() or irate() over a defined window. For gauges, the panel query must be the gauge directly. For histograms, the panel query must be histogram_quantile over a bucket, never the bucket raw.
3. Set the unit. Match the unit to the metric. Seconds for latency in seconds, bytes for memory in bytes, percent for percentages, reqps for request rates. Do not rely on short; pick the explicit unit.
4. Set the threshold from the SLO. The threshold is the value at which the SLO has been or will be breached. Part III of the course covers SLO construction in detail.
5. Document the action. In the panel description or the dashboard description, write what to do when the panel turns yellow or red.
6. Delete the dead panel. If the panel does not survive steps 3 through 5, delete it. The dashboard is better for the absence.

Security implications

A meaningless gauge can be a soft leak. A panel showing auth_tokens_issued_total without thresholds, displayed on a screen visible to anyone with a browser session, reveals the traffic shape of an authentication endpoint. Even if the panel itself is harmless, the absence of the unit context is informative: an attacker who knows the panel renders 42 knows there is a counter and not a rate, and can plan around that.

The discipline is the same as for wallpaper: folder-level ACLs control who sees the dashboard at all, and meaningful panels keep the dashboard set small enough that ACLs are reviewed quarterly.

Performance implications

Stat and gauge panels in Grafana 11.x evaluate one query per panel. They do not retry or stream updates. They are cheap per panel, but they encourage proliferation: an operator adds a “small stat panel” for every metric they want to watch, and the dashboard grows to 80 stat panels. The query load against Prometheus grows linearly with the panel count. The cost is real and rarely measured until an incident doubles the panel count temporarily.

The discipline: stat panels on overview dashboards only. Detail dashboards get timeseries. The number of stat panels per dashboard should be countable on one hand.

Production guidance

• Never render a counter directly. Always render rate() or irate().
• Pick the unit that matches the metric. Use bytes for bytes, s for seconds, reqps for requests per second, percent for percentages.
• Thresholds must come from the SLO. If there is no SLO, there is no threshold. A panel without a threshold is the candidate for deletion.
• Document the action in the panel description. “If red, page on-call-checkout. Runbook URL in the panel description link”.

Verification

You should now be able to answer:

• What are the three properties that make a number operational?
• Why is a counter rendered directly in a stat panel wrong?
• Which Grafana 11.x panel field option determines whether the panel reads 42, 42 req/s, or 42%?
• What is the right rate function to wrap a counter in a panel?
• Why is the threshold-by-mistake failure shape so easy to miss in code review?

Quiz