Latency Pivot Workflow

The alert fires: p99 latency above 1 second for the last 10 minutes. The histogram in Grafana shows the curve climbing. Somewhere under the curve, one or two bars in the p99 tail hold a representative observation — a request that took 1.2 seconds, 1.4 seconds, 4.8 seconds. The histogram alone tells the operator that the service is slow; the exemplar on the bar tells the operator which request was slow. The workflow is the same for every latency investigation: confirm the metric is real, draw the pivot, click into the trace, read the span.

This lesson covers the histogram side of that workflow. The trace side (reading the span) is lesson 06.

What it is

The latency pivot is the workflow that turns a histogram spike into the trace of the request that produced the spike. The workflow has four steps, each with a specific configuration that must be in place:

  1. Producer emits a histogram with a trace context attached
                |
                v
  2. Prometheus stores the bucket counter and the exemplar suffix
                |
                v
  3. Grafana renders the histogram panel and overlays diamonds
                |
                v
  4. Operator clicks a diamond; Tempo serves the trace

Steps 1 and 2 are lesson 01’s territory: SDK instrumentation and the --enable-feature=exemplar-storage flag. Steps 3 and 4 are this lesson’s territory: the panel options that turn a histogram into a clickable one, and the read patterns that turn a clicked trace into a span.

The pivot is a latency pivot because the metric type is a histogram and the unit is the duration of a request. There is no size pivot, no queue-depth pivot; the histogram is the only metric type that holds a duration as a representative value.

Why a sysadmin cares

Three operational situations collapse with the pivot that were otherwise expensive to triage:

1. The p99 spike that isn’t a regression. A single noisy client opens 3,000 connections, retries each request, and drowns the histogram. The pivot shows a 14-second span that is a client-side timeout, not a server-side regression. The alert is silenced in minutes rather than triggering an hour-long code review.
2. The regression that is a single dependency. A p99 histogram over the entire request is flat most of the time and tall when the payment service is slow. The pivot shows the dependency span holding 1.3 of the 1.4 seconds. The fix is upstream of the service and would be unidentifiable without the trace.
3. The cache stampede. The histogram climbs every hour at minute :50 when a TTL expires. The pivot shows a single cache miss resolving against a 1.2-second DB query. The fix is a cache stampede protection pattern, not a code change.

In all three cases the metric alone says “service slow at top of hour”. The pivot says “this specific request, this specific dependency, this specific code path”. The difference in time-to-resolution is the difference between an investigation and a denial.

How it works

The latency pivot depends on three things being true at the moment of the click:

  Producer histogram observation point
        |
        | Active W3C trace context is attached to the bucket
        v
  Histogram bucket holds the counter and exemplar suffix
        |
        v
  Prometheus stores both via the exemplar-storage appender
        |
        v
  Grafana panel renders counter as bar, exemplar as diamond
        |
        v
  Click diamond -> Internal link -> /explore -> Tempo trace

The pivot is read-only after the producer emits. No code changes the trace; the histogram and the trace live parallel lives. The only synchronisation is the timestamp on the exemplar and the timestamp on the spans, which are the wall-clock when the request was made.

The pivot reads as a downward drill: from the population metric to a representative trace to a specific span to a specific dependency. Each step removes a layer of averaging.

How to configure it

Two configuration steps at the panel level. The bucket layout at the producer is also a configuration step; that is covered in lesson 01.

1. Configure the histogram panel options

In Grafana 11.x, the histogram panel for a Prometheus data source has three options that must be set:

# Panel options (CONFIGURATION — UI or provisioning)
# Verified on Grafana 11.x
panel:
  type: timeseries
  title: 'checkout latency (with exemplars)'
  datasource: prom
  options:
    # Render as histogram, not timeseries
    drawStyle: bars
    stacking: normal
    mode: histogram
  fieldConfig:
    defaults:
      unit: s
      # The exemplar option is the one that turns on the diamond
      custom:
        exemplarShow: on
        # Optional: a regex to pick which observations to show
        # when multiple exemplars land on the same bucket
        exemplarMaxWidth: 80
  targets:
    - refId: A
      expr: 'histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))'

The key line is exemplarShow: on. When set, Grafana reads the exemplar suffix from Prometheus for each bar and overlays the diamond. When absent, the histogram is rendered as a plain bar chart and the click target does not exist.

For the panel via the UI, open the panel editor and enable the Exemplars option under Field settings.

2. Verify the bucket layout brackets the alert threshold

The bucket layout is the producer’s choice; the lesson documents the test for it rather than a write step.

# READ-ONLY — list the buckets the producer has chosen
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | awk -F'le="' '{print $2}' \
  | awk -F'"' '{print $1}'

Expected: a list starting 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +Inf. Confirm that the alert threshold sits inside the list. If the alert fires at 1.4 s and the largest finite bucket is 1.0, the exemplar lands in +Inf and pivots are coarse.

3. Pivot to the trace

The click target is built when exemplarShow: on is set. Verify the click target with a development panel or by inspecting the rendered URL after a click:

# READ-ONLY — exercise the click through the Grafana API
# (this is the URL Grafana builds when the click fires)
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

# Confirm Tempo serves the trace
curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '.traces[0].spans | length'

Expected: at least one span. A zero-span response means the producer emitted an exemplar but the spans did not arrive at Tempo.

How to validate it

Four validation layers, each catching a distinct failure mode.

1. The producer’s bucket layout includes buckets bracketing the alert

# READ-ONLY
curl -sf http://checkout.svc:8080/metrics \
  | grep '^http_request_duration_seconds_bucket' \
  | awk -F'le="' '{print $2}' \
  | awk -F'"' '{print $1}'

The output must include at least one bucket smaller than the alert threshold and at least one bucket larger.

2. The Prometheus exemplar storage returns data for the alert bucket

# READ-ONLY
curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq '.data | length'

Expected: a positive integer. The le="2.5" bucket is the target when the alert threshold is 1 second.

3. The Grafana panel has exemplars enabled

# READ-ONLY
# Pull the panel definition from the Grafana API
curl -sf -u admin:admin 'http://grafana:3000/api/dashboards/uid/checkout-latency' \
  | jq '.dashboard.panels[] | select(.title=="checkout latency (with exemplars)") | .fieldConfig.defaults.custom.exemplarShow'

Expected: "on". If empty, the panel was configured via the UI to disable exemplars or the field was never set.

4. The trace has the span that produced the exemplar

# READ-ONLY — end-to-end
TRACE_ID=$(curl -sfG http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket{le="2.5"}' \
  --data-urlencode 'start=2026-08-13T10:00:00Z' \
  --data-urlencode 'end=2026-08-13T10:10:00Z' \
  | jq -r '.data[0].exemplarLabels.trace_id')

curl -sf "http://tempo:3200/api/traces/$TRACE_ID" \
  | jq '[.traces[0].spans[] | select(.durationNanos > 1000000000)] | length'

Expected: at least one span with duration over 1 second (1,000,000,000 nanoseconds). The slow span is the place to start reading.

How it can fail

Four failure modes, ordered by frequency in real environments.

1. Bucket layout does not bracket the alert threshold. The histogram is rendered correctly; the slowest observation lands in +Inf. Symptom: the diamond appears only on the +Inf bar; pivots are coarse; the operator sees “everything above 10 seconds” with no finer resolution.
2. exemplarShow is off on the panel. The histogram renders bars but the diamond is absent. Symptom: the bar is on the chart; clicking the bar does nothing; the API response shows the exemplar is on the server but not on the panel.
3. Producer’s histogram observation point is not on the hot path. The histogram is updated at a middleware boundary after the slow dependency has already returned. The bucket holds the slow observation but the trace’s slow span is inside the producer, not where the metric says it is. Symptom: the trace shows a fast root span and a “no slow spans” appearance when the histogram is tall.
4. Tracer sampler drops the exemplar trace but not the histogram observation. The observation is recorded before sampling, the head sampler rejects the trace. The histogram is present, the trace is absent. Symptom: query_exemplars returns data; /api/traces/<id> returns zero spans.

How to troubleshoot it

Steps in order. Each step rules out one of the four failure modes.

1. Check the bucket layout. Run the bucket list command from validation step 1. If the alert threshold is not bracketed, fix the producer’s histogram definition; this is a code-side change, not a configuration change.
2. Check the panel options. Run the panel options query from validation step 3. If exemplarShow is not on, enable it in the panel UI or provisioning; re-test by opening the panel in Grafana.
3. Check the trace has a slow span. Run the end-to-end command from validation step 4. If there is no slow span, the trace’s data is incomplete; check the OTLP pipeline and the producer’s sampling.
4. Check the histogram observation point. Compare the span times in the trace against the metric value. If the trace root span is 50 ms but the exemplar value is 1.2 seconds, the producer is recording the metric at the wrong point — typically a middleware order bug.

Security implications

The latency pivot reuses every component of the metrics-to-traces pivot. The security implications are the same:

• The trace ID is in the URL; treat it as low-grade data.
• The user clicking the diamond can read whatever Tempo serves for that trace; the data source permissions gate this.
• The exemplar storage on Prometheus 2.55.x is not replicated via remote write; verify what the receiver does with the exemplar suffix.

The latency pivot does not introduce new surfaces. It is a consumer of the surfaces from lesson 01.

Performance implications

The latency pivot is bounded by the slowest pane in the operator’s workflow:

• Producer side. Observation overhead is two pointer reads per call. Negligible on modern x86_64; measurable on embedded ARM profiles.
• Prometheus side. Exemplar appender is bounded; the rate limit is per-scrape, not per-day, and is set high enough not to be the bottleneck under normal traffic.
• Grafana side. Exemplar overlay rendering is O(N diamonds per panel) per refresh. Panels with hundreds of label combinations and a tight refresh interval can become the bottleneck; the standard fix is to raise the refresh or reduce the label cardinality.

Production guidance

• Calibrate the bucket layout against the alert thresholds you expect, not the defaults. A histogram with (0.005, 0.01, 0.025, ...) is fine for a 50 ms SLO and useless for a 5 second SLO. Default layouts are oriented toward default SLOs.
• Add the exemplarShow: on field to every histogram panel in the provisioning file. A panel created without it will render fine in the preview and fail the pivot silently at runtime.
• Treat the producer’s choice of observation point as part of the histogram definition, not part of the trace. The two are written separately and read together at pivot time; their drift is the third failure mode.

Verification

You should now be able to answer:

• Why must the bucket layout bracket the alert threshold, and what is the visible symptom when it does not?
• Which Grafana panel option turns on the exemplar diamond?
• How do you distinguish a panel-without-pivot failure from a trace-not-in-Tempo failure when both look identical from the operator’s seat?

Quiz