Inconsistent Units

Two panels sit side by side on the “Checkout Service” dashboard. The first shows latency in milliseconds, with the value 350 and a threshold of 1000. The second shows latency in seconds, with the value 0.35 and a threshold of 1. Both panels describe the same metric; both panels are green; both panels claim the service is healthy.

A third panel, from a different team’s dashboard for the same service, shows latency in milliseconds with the value 0.000350 and a threshold of 1. The team that built it divided by 1000 twice. The dashboard has been showing 0 for six months.

A fourth panel, on the platform team’s “Latency Comparison” dashboard, mixes all three. The legend says ms, the value reads 0.350, and the on-call engineer who opened it during the last incident spent forty-five seconds figuring out which panel was which.

Inconsistent units are any combination of mixed bases (seconds and milliseconds), mixed prefixes (bytes and megabytes), or mixed rates (per second and per minute) on panels that are compared by the operator during the same incident.

What it is

A unit mismatch in production is one of four shapes:

1. Mixed bases. One panel reports latency in seconds, another in milliseconds. The threshold of 1000 on the first is 1 on the second. The values look identical visually; the implications differ by 1000x.
2. Mixed prefixes. One panel reports memory in bytes, a second in megabytes, a third in gibibytes. The thresholds are not comparable across panels.
3. Mixed rate bases. One panel reports requests per second, another reports requests per minute. The values differ by 60x. A panel showing 60 req/s next to a panel showing 60 req/min looks identical; the operator cannot compare them without a calculator.
4. Mixed SI/IEC prefixes. One panel uses KB (1000 bytes), another uses KiB (1024 bytes). The values are within 2.4% of each other but not equal. Over 1 TB the discrepancy is 24 GB.

The discipline is the same in all four: pick one base, one prefix convention, one rate window, and apply it to every panel in the dashboard set. Document the convention in the team’s instrumentation guide and enforce it in CI.

Why a sysadmin cares

Three production costs:

• Wrong threshold calibration. A threshold of 1000 on a panel set to ms is 1 on a panel set to s. If the threshold was calibrated against one unit and the panel is rendered in another, the panel is either too sensitive or not sensitive enough. Either way, the alert fires at the wrong moment.
• False comparison. Two panels showing the same metric with different units side by side look comparable. The operator assumes they are. The comparison is wrong by 60x, 1000x or 1024x.
• Migration cost. When the team decides to standardise on one unit, every panel whose unit does not match must be updated. The cost is proportional to the number of panels with mixed units. The discipline that prevents the mix is cheaper than the migration.

How it works

The Grafana 11.x unit system is layered:

   Metric raw value
        |
   +----+----+----+
   |         |    |
 Base    Prefix   Rate
 unit    (SI/IEC) window
   |         |    |
   v         v    v
 Grafana "unit"
 string
 (e.g. "reqps",
  "bytes",
  "ms")

The unit string in the panel field options selects how Grafana renders the value. It does not change the underlying metric. The metric is http_request_duration_seconds, and the panel author must pick the right unit to render it.

Common Grafana 11.x unit strings:

The bytes unit is IEC binary (1 KiB = 1024 bytes); the decbytes unit is SI decimal (1 KB = 1000 bytes). The choice between the two depends on the metric source: Prometheus’s node_memory_Active_bytes is IEC binary at the kernel level (although Prometheus exposes it as bytes); aws_ec2_cpu_total is decimal. The dashboard’s unit must match what the operator expects to read.

The most expensive mistake is to pick bytes for a disk panel when the on-call engineer is used to GB from the cloud console. GiB is the rendered suffix in Grafana; GB is the rendered suffix in decbytes. They differ by 7% at the gigabyte scale and by 24 GB at the terabyte scale.

Under the hood

The Prometheus convention is base SI units in the metric name. A metric called http_request_duration_seconds is in seconds; a metric called process_resident_memory_bytes is in bytes; a metric called node_network_transmit_bytes_total is in bytes. The base unit is the contract.

OpenTelemetry follows the same convention with a more explicit syntax: the metric name is http.server.request.duration, the unit is declared as part of the instrument, and the exported value is in seconds. The Grafana data source for OpenTelemetry respects the declared unit and suggests the matching unit string in the panel field options.

The mismatch occurs at the panel author step. The author picks unit: "ms" for a metric in seconds. The threshold is 100. The panel renders 0.350 (correct), but the operator reads 350 in their head because the unit string says ms. The threshold of 100 was calibrated against ms, so the panel turns red at 100 (which is 100 seconds in reality). The service has been breached for hours.

The fix is to enforce the unit at provisioning time: every panel for a metric whose name ends in _seconds must set unit: "s"; every panel for _milliseconds must set unit: "ms"; every panel for _bytes must set unit: "bytes" or unit: "decbytes"; every panel for _total rate must set unit: "reqps".

How to configure it

Latency panel

# PromQL. Metric in seconds.
histogram_quantile(
  0.99,
  sum by (le) (
    rate(checkout_request_duration_seconds_bucket[5m])
  )
)
# Threshold at the SLO: 1 second. Unit: "s".

{
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "decimals": 3,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green",  "value": null },
          { "color": "yellow", "value": 0.5 },
          { "color": "red",    "value": 1 }
        ]
      }
    }
  }
}

Memory panel

# PromQL. Metric in bytes.
process_resident_memory_bytes{service="checkout"}
# Unit: "bytes" for IEC binary (KiB/MiB/GiB suffix).

{
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "unit": "bytes",
      "decimals": 2,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green",  "value": null },
          { "color": "yellow", "value": 1073741824 },
          { "color": "red",    "value": 2147483648 }
        ]
      }
    }
  }
}

The thresholds use raw bytes (1073741824 = 1 GiB, 2147483648 = 2 GiB). The panel renders the value with the GiB suffix automatically.

Rate panel

# PromQL. Metric is a counter; rate is per second.
sum by (endpoint) (
  rate(http_requests_total[5m])
)
# Unit: "reqps" (requests per second).

{
  "type": "timeseries",
  "fieldConfig": {
    "defaults": {
      "unit": "reqps",
      "decimals": 2,
      "custom": {
        "drawStyle": "line",
        "lineWidth": 2
      }
    }
  }
}

How to validate it

The CI lint enforces the unit convention. Every panel whose query references a metric ending in _seconds must have unit: "s". Every panel whose query references a metric ending in _bytes must have unit: "bytes" or unit: "decbytes". Every panel whose query uses rate() or irate() must have a rate-compatible unit (reqps, Bps, or a metric-specific unit).

# READ-ONLY. Walk every dashboard and report panels whose
# unit string does not match the metric suffix.
for uid in $(curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
              "${GRAFANA_URL}/api/search?type=dash-db" \
              | jq -r '.[].uid'); do
  curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
    "${GRAFANA_URL}/api/dashboards/uid/${uid}" \
    | jq -r --arg uid "$uid" '
        .dashboard.panels[]? as $p
        | ($p.targets[]?.expr // "") as $expr
        | ($p.fieldConfig.defaults.unit // null) as $unit
        | select($expr != "" and $unit != null)
        | (
            (($expr | test("_seconds\\b")) and ($unit != "s")) as $bad_s
          | (($expr | test("_milliseconds\\b")) and ($unit != "ms")) as $bad_ms
          | (($expr | test("_bytes\\b")) and ($unit != "bytes" and $unit != "decbytes")) as $bad_b
          | (($expr | test("rate\\(")) and ($unit != "reqps" and $unit != "Bps" and $unit != "s")) as $bad_r
          | select($bad_s or $bad_ms or $bad_b or $bad_r)
          | "\($uid)\t\($p.title)\tunit=\($unit)\texpr=\($expr | gsub("\n"; " "))"
        )
      '
done

Expected output (illustrative):

checkout-overview  Request rate        unit=short expr=sum(rate(http_requests_total[5m]))
payments-detail    Latency             unit=ms   expr=histogram_quantile(0.99, ...)

For each match, set the unit to the metric’s base unit and re-calibrate the threshold.

How it can fail

Six specific failure shapes:

1. Mixed seconds and milliseconds. Two dashboards for the same service, one in s and one in ms. The on-call engineer compares them and concludes the service is faster on one dashboard. The conclusion is wrong by 1000x.
2. Mixed bytes and decbytes. A memory panel in bytes next to a disk panel in decbytes. The suffixes are different (MiB vs MB). The operator reads them as the same and the threshold calibration is off by 7%.
3. Mixed percent and percentunit. A panel in percent (0-100) next to a panel in percentunit (0-1). The second panel renders 0.95 and the operator thinks it is 0.95% rather than 95%. The threshold was calibrated for percent.
4. Mixed rate windows. A panel showing 60 req/s next to a panel showing 60 req/min. The operator compares them as if they are the same rate; the second is 100x slower.
5. The wrong-decimal-marker panel. A metric exported as a string-suffixed value ("42.3s") is rendered by a panel that assumes the value is in seconds, but the suffix says milliseconds. The panel renders 42.3 with s after it, misleading the operator by 1000x.
6. The dimensionless counter. A panel for node_network_transmit_packets_total with unit: "short". The value renders as 1234 packets but the operator reads it as 1234 bytes. The unit was forgotten.

How to troubleshoot it

When an incident has been caused or worsened by unit mismatches:

1. Identify the metric. Run promtool query instant http://prometheus:9090 '<metric>'. Note the metric’s documented base unit (from the instrumentation guide or the metric’s HELP string).
2. Identify the panel’s unit. Open the dashboard. For each panel involved in the confusion, inspect the field options. Note the unit value.
3. Match the unit to the metric. If the metric is in seconds, the panel unit must be s. If the metric is in bytes, the panel unit must be bytes or decbytes. If the panel uses rate(), the panel unit must be a rate (reqps, Bps, or the metric’s rate-compatible unit).
4. Recalibrate the threshold. The threshold must be in the panel’s unit, not the metric’s. For a panel in ms with a metric in s, the threshold is 1000 (not 1).
5. Re-validate the panel rendering. Open the dashboard, check the rendered value, and confirm the suffix is the one the operator expects.
6. Document the convention. Add the unit to the team’s instrumentation guide. Add a CI lint rule that catches the next mismatch.

Security implications

Unit mismatches are mostly a usability issue, with one security edge case: a panel that reports authentication failure rates per minute on one dashboard and per second on another. The two dashboards disagree on the rate by 60x. An attacker who probes the authentication endpoint sees one dashboard in their browser and a different number in the SOC’s dashboard. The mismatch trains the SOC to mistrust the data, which is the attacker’s goal.

The discipline is the same: standardise on one base, one prefix, one rate window. The security benefit is consistency across the operator’s view of the system.

Performance implications

The unit choice has no server-side performance impact; it is a rendering decision in the browser. The cost is human, as above: the minutes between alert and correct interpretation, and the alerts that fire too late or too early because the threshold was calibrated against the wrong unit.

Production guidance

• Adopt the Prometheus / OpenTelemetry base-units convention: seconds for time, bytes for storage, ratios as 0-1.
• Pick one prefix convention per metric family. bytes (IEC) for memory, decbytes (SI) for disk and network. Document the choice.
• Render rates per second (reqps, Bps). If the team prefers per minute, render reqpm and Bpm consistently; do not mix.
• Lint at provisioning time. Every panel whose metric ends in _seconds must set unit: "s". Every panel whose metric ends in _bytes must set unit: "bytes" or unit: "decbytes".

Verification

You should now be able to answer:

• What are the four shapes of unit mismatch on a Grafana 11.x dashboard?
• Why is the IEC bytes unit different from the SI decbytes unit, and which should memory panels use?
• What is the Prometheus convention for the base unit of a metric whose name ends in _seconds?
• Why is unit: "short" a particular trap?
• What is the right unit for a panel whose query uses rate() on a counter of HTTP requests?

Quiz