Error Pivot Workflow

The error rate spikes. The counter 5xx_count rose from 0.3% to 18% in the last minute. The graph is alarming. The counter alone says many requests are failing. It does not say why they are failing. The histogram of latency labelled status="500" does, because its bucket observations are representative failed requests.

The error pivot is the workflow that turns a counter spike into a histogram lookup that points at the trace of a failed request. The pivot is the same as the latency pivot except the entry-point metric is a counter and the pivot-bearing metric is the histogram.

What it is

The error pivot has two distinct metrics playing different roles:

• Counter — the alert metric. Sum of 5xx_count divided by request_count produces the error rate the alerting system watches. The counter tells the operator that errors are rising. The counter does not point at individual requests; counters are aggregate by construction.
• Histogram — the pivot metric. request_latency labelled with status. The bucket for status="500" increments every time a request returns a failure. Each bucket has a representative observation, and the bucket with the exemplar is the one the diamond lands on. The histogram tells the operator which failed request.

  Error rate counter                      Latency histogram
  (alert source)                          (pivot source)
        |                                       |
        v                                       v
  Alert: 5xx_ratio > 5%                Panel: bars grouped by status
  Operator notices spike                 Diamond appears on status="500" bar
        |                                       |
        +--------------+------------------------+
                       |
                       v
                Click diamond -> Tempo trace
                       |
                       v
                Read error span

The two metrics are tied together by the status label and the request’s trace context. The Counter is the alarm; the Histogram is the click.

Why a sysadmin cares

An error rate without a pivot turns every incident into a log-grep war. Three operational situations collapse with the error pivot:

1. The 5xx attributed to a single dependency. The counter says 18% of requests return 500. The pivot shows every failed request’s trace ending with a 503 from a downstream service. The fix is in the downstream’s configuration.
2. The 5xx attributed to a single client. The counter says 18% but the same histogram with customer as a label shows the 5xx is concentrated in a single tenant. The fix is to investigate that tenant’s request shape, not the service.
3. The 5xx attributed to a code path. The counter says 18% and the pivot shows all failed traces converging on the same code path. The fix is a code review of that path; the histogram label is route="/checkout" and the pivot is the regression the team pushed at 14:20.

In each case, the counter says “more errors than usual” and the pivot shows the dependency, the client, or the code path. Both are required for an investigation that ends in a fix.

How it works

The pivot hinges on the histogram’s bucket labelled with the status code. The producer SDK must record the same observation in both metrics at the same point of the request lifecycle:

  Handler receives request
        |
        | Producer records:
        |   request_count{status=..., route=...}      += 1
        |   request_latency_seconds_bucket{status=..., route=..., le=...}    ++
        |   request_latency_seconds_sum                  += duration
        v
  Handler returns response with status
        |
        | Producer attaches active trace context to the bucket
        v
  Prometheus stores both counter and histogram with exemplars
        |
        v
  Grafana renders two panels:
        |  error_rate_panel — drawn from the counter (no exemplar, just red)
        |  error_latency_panel — drawn from the histogram (diamonds on 5xx bars)
        v
  Operator clicks the diamond on a 5xx bar
        |
        v
  Tempo serves the failing trace

The crucial detail is the histogram observation point is at the return path, where the status code is known. Observing at the entry path means the status label is unknown and the histogram cannot be sliced into 200 vs 500 buckets.

How to configure it

Two configuration steps. The histogram definition is the producer’s responsibility; the panel configuration is the operator’s.

1. Producer-side: instrument both metrics

The lesson snippet above is the canonical pattern. The key discipline is:

• Both metrics are labelled identically on status, route, and method.
• The bucket list includes a bucket small enough to bracket the latency of a typical 5xx. A 5xx that returns in 50 ms lands in the le="0.1" bucket; a 5xx that hangs the timeout lands in le="10.0" or +Inf.

2. Panel-side: render the error histogram

# Panel options (CONFIGURATION)
panel:
  type: timeseries
  title: 'latency by status (with exemplars)'
  datasource: prom
  options:
    drawStyle: bars
    stacking: normal
    mode: histogram
  fieldConfig:
    defaults:
      unit: s
      custom:
        exemplarShow: on
        # Limit the visible buckets to ones bracketing the alert threshold
        scaleDistribution:
          type: linear
          log: 10
          lowerBound: 0.05
          upperBound: 10.0
  targets:
    - refId: A
      # The histogram grouped by status — this is where diamonds appear
      expr: 'sum by (le, status) (rate(http_request_duration_seconds_bucket[5m]))'
    - refId: B
      # The error rate counter — does not carry exemplars; drives the alert
      expr: 'sum(rate(http_requests_total{status=~"5..|5.."})[5m]) / sum(rate(http_requests_total[5m]))'

The expr lines illustrate the split: A is the histogram and carries the diamonds; B is the counter and drives the alert. The two queries share the status label and the route label, so the operator can correlate the rate spike with the latency distribution.

3. Verify the pivot end-to-end

# READ-ONLY — confirm a 5xx bucket has an exemplar
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{status="500",le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.data | length'

Expected: a positive integer. Zero is the canonical symptom of “no exemplar is reaching the 5xx bucket” — the next section covers the four causes.

How to validate it

Three validation layers.

1. Both metrics exist with matching labels

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep -E '^(http_requests_total|http_request_duration_seconds_bucket)' \
  | head -20

Expected: lines for both metrics, both labelled with the same status, route, and method values. Mismatched label sets are a common silent failure.

2. The 5xx bucket has an exemplar

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{status="500"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.data | length'

Expected: a positive integer when there have been recent 5xx responses. Empty result on a service that is currently firing 5xx alerts is the most common failure mode.

3. The trace of a 5xx has an error status

# READ-ONLY — end-to-end
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{status="500",le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '[.traces[0].spans[] | select(.status.code == "ERROR")] | length'

Expected: at least one span with status.code == "ERROR". The spans with error status are the place to start reading the trace.

How it can fail

Four failure modes, ordered by frequency.

1. The histogram is recorded at the entry path, before the status is known. The bucket increments with a placeholder status. The 5xx never gets a representative observation. Symptom: the counter spikes; the histogram’s le distribution shows activity only on status="200"; the status="500" bucket is empty.
2. The histogram is recorded only on the success path. The 5xx path is unmonitored by the histogram. Symptom: the counter spikes; the histogram is flat. No diamond on the 5xx bars because there are no 5xx bars.
3. The labels on the counter and the histogram drift. The counter has code="500", the histogram has status="500". The histogram panel query selects nothing. Symptom: the panel is empty; the diamond does not appear; the counter panel is fine.
4. The 5xx latency is below the smallest bucket boundary. The histogram is le="0.05, 0.1, ... but the 5xx response returns in 8 ms. The exemplar lands at le="0.05" and the diamond is hidden by the panel’s lower bound. Symptom: the diamond is in the data but not visible on the panel.

How to troubleshoot it

Steps in order. Each step rules out one of the four failure modes.

1. Check that the histogram has any status="500" buckets. Query the histogram directly: sum by (status) (rate(http_request_duration_seconds_count[5m])). If status="500" is absent, the histogram is not recording on the failure path. This is failure mode 2.
2. Check that the bucket count matches the counter. The rate(http_requests_total{status=~"5.."}[5m]) and the rate(http_request_duration_seconds_count{status=~"5.."}[5m]) should match. If the counter is high and the histogram count is zero, the histogram recording point is wrong (failure mode 1 or 2).
3. Inspect the panel query. Open the panel in Grafana; run the histogram query in Explore; confirm the buckets are not filtered out by an unintended status=~"2..|3.." selector (failure mode 3).
4. Inspect the panel’s scale and bucket distribution. A diamond below the panel’s lower bound is invisible. Adjust the linear scale or move the panel to use log scale starting from a finer lower bound (failure mode 4).

Security implications

The error pivot reuses every component from the latency pivot. The status code itself is part of the trace’s metadata and follows the same access controls.

One new consideration: an error trace often contains the exception’s stack trace and arguments. These are the source of multiple production-data leaks. The pivot is not the leak itself, but the pivot is what accelerates an operator from “5xx spike” to “read the stack trace” in a single click. The producer should sanitise stack frames and argument values at record time; once recorded, the trace carries them.

Performance implications

The error pivot has the same performance profile as the latency pivot at the producer, Prometheus, and Grafana layers. The new consideration is the cardinality of status. Most services have five to ten distinct status codes; the label adds nine series per (method, route) combination. For a service with 20 routes and 6 methods, that is 1,080 status- bucketed histogram series. Cardinality budgeting (Part LI) applies.

Production guidance

• Use a single status label taxonomy. “OK”, “CLIENT_ERROR”, “SERVER_ERROR” is enough; per-error-code labels (code=500, code=502, code=503) inflate cardinality without operational benefit. The 5xx rate panel groups by class, not by code.
• Record the histogram in the same finally block as the counter. The two metrics must update at the same point; any code path that updates one and not the other is a source of drift.
• Add a separate histogram for the error path even if the combined status histogram works in steady state. A 5xx that resolves in 8 ms is below the typical latency floor of the combined histogram and is invisible by construction.

Verification

You should now be able to answer:

• Why is a counter-based error rate the alert source rather than the pivot source?
• What is the role of the status label on the histogram bucket, and what happens when it is missing?
• How do you distinguish a histogram-not-recording-on- failure-path failure from a panel-query-mismatch failure?
• Why does the producer-side recording site matter more for the error pivot than for the latency pivot?

Quiz