Traces vs Metrics Correlation

The alert fires: http_server_requests_seconds:rate5m{status_code="500"} is 0.018, the SLO budget for the hour is 0.005. The on-call engineer opens the dashboard and confirms the rate. They now need a trace of one of the 500s. The metric said the rate is elevated; the trace will say which call failed. The bridge between the two is the exemplar — a trace_id stamped onto a representative sample of the metric stream.

This lesson is the discipline of distinguishing the two signals and the join that lets each one answer its own question without losing the other.

What it is

A metric is a numeric value aggregated over a population and labelled by a small set of dimensions. A counter named http_server_requests_seconds_count has labels method, uri, status_code. The value is the running count of every request that matched the labels. The data model is (name, labels, timestamp, value, optionally exemplar).

A trace is a tree of spans that describes one specific request. Each span carries a name, attributes, and a duration. The data model is (trace_id, span_id, parent_span_id, name, service.name, start_time, duration, attributes, status).

The data-model difference is the cardinality difference. A metric series is (name, label-set); the cardinality is the product of distinct values across every label. A trace is one record per span; the cardinality is the number of distinct spans.

Metric:   http_server_requests_seconds_count
          { method="POST", uri="/checkout", status_code="500" }
          value=42, timestamp=2026-08-13T03:14:22Z
          exemplar: trace_id="8f1d...a3c"

Trace:    trace_id="8f1d...a3c", 5 spans, 1.42 s
          root: api-gateway.checkout
          child: checkout.cart.read   12 ms
          child: checkout.pricing.lookup  83 ms
          child: checkout.payment.charge 1.31 s status=error
          child: checkout.receipt.write   14 ms

The metric names the population and reports one number for the whole population in one second. The trace names one member of the population and reports five numbers for that member’s decomposition. The exemplar is the trace_id of a recent member of the population that contributed to the metric value.

Why a sysadmin cares

Metrics and traces answer different questions on purpose. A team that tries to answer the metric’s questions with traces runs out of storage. A team that tries to answer the trace’s questions with metrics re-implements the trace as a histogram that loses the per-request decomposition.

Three operational pains are specific to running one signal without the other.

1. The metric that says the rate is high. The histogram is alerting. The dashboard shows the rate is climbing. Without exemplars, the engineer has to guess which request to reproduce. With exemplars, the dashboard panel carries a link directly to a sample trace; the engineer clicks and lands in the trace view.
2. The trace that says the request was slow. A trace shows 1.4 s of latency on one request. Without the metric, the engineer has to decide whether the slowness is rare or routine. With the metric, the engineer knows the histogram p99 is 1.4 s and the trace is representative.
3. The investigation that has to switch tools. The alert fires on Prometheus. The trace lives in Tempo. The engineer has to copy the timestamp, switch tools, search by timestamp, filter to the same labels, and find the matching trace. The exemplar eliminates the switch.

How it works

The two signals are produced by the same OTel SDK in the same application. The SDK has a meter provider that produces metrics and a tracer provider that produces spans. Both providers can share the same resource (the same service.name); the only bridge between them is the exemplar.

The exemplar mechanism

An exemplar is a sample of a metric’s value paired with a reference to the trace that produced it. The mechanism:

1. The application code records a metric value (e.g. http_server_requests_seconds_count += 1).
2. The application code, in the same scope, holds an active span context.
3. The SDK records the metric increment and attaches the current trace_id and span_id as an exemplar.
4. The exporter forwards the metric with the exemplar attached to the back-end (Prometheus or Mimir).

The Prometheus exposition format embeds exemplars inline with the metric:

# TYPE http_server_requests_seconds_count counter
# HELP http_server_requests_seconds_count Total count of HTTP requests
http_server_requests_seconds_count{method="POST",uri="/checkout",status_code="500"} 42
  # {trace_id="8f1d...a3c",span_id="c4e2...9b1"} 1.31 2026-08-13T03:14:22Z

The # {trace_id=...,span_id=...} line after the metric is the exemplar. The value 1.31 is the observed request duration; the timestamp is when the request happened. Grafana reads the exemplar from Prometheus and renders a small icon in the panel; clicking the icon opens the trace in Tempo.

The cardinality cost

A metric series costs roughly 16 bytes of label-set plus a few bytes per timestamp in Prometheus; the dominant cost is the index entry per (name, label-set) pair. The cardinality is the product of distinct values across labels.

A trace costs roughly 200-500 bytes per span in Tempo; the dominant cost is the block storage and the span index. The cardinality is the number of distinct traces per retention period.

The trade-off is sharp. A metric is cheap at low cardinality and expensive at high cardinality; a trace is cheap at low volume and expensive at high volume. The decision about which signal to use for a question is the decision about which cost to pay.

The exemplar as the join key

The exemplar is the only automatic bridge. The application writes both signals; the SDK attaches the trace_id to a sample of the metric stream; the dashboard renders the link. The discipline is to ensure every metric you want to pivot from has an exemplar attached.

Under the hood

How to configure it

Three pieces: the SDK records exemplars; the collector forwards them; Prometheus stores them; Grafana renders the link.

The OTel SDK configuration for an HTTP server:

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider

# Tracer provider is the source of spans and the source of
# the trace_id that the meter attaches to exemplars.
tracer_provider = TracerProvider()
trace.set_tracer_provider(tracer_provider)

# The meter provider reads the current span context for
# every recorded measurement and attaches the trace_id as
# an exemplar.
meter_provider = MeterProvider()
metrics.set_meter_provider(meter_provider)

The Prometheus configuration that enables exemplar storage:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'checkout'
    scrape_interval: 5s
    static_configs:
      - targets: ['checkout.internal:8889']

The Prometheus flag that turns on exemplar storage:

--enable-feature=exemplar-storage

The collector that forwards both signals is the same Alloy configuration as lesson 01; no additional blocks are required for exemplars. Exemplars are part of the Prometheus exposition format and are forwarded with the metric samples.

The Grafana data source that joins them:

# Grafana data source: Prometheus
type: prometheus
url: http://prometheus.internal:9090
jsonData:
  exemplarTraceIdDestinations:
    - name: 'Tempo'
      datasourceUid: 'tempo-uid'

The exemplarTraceIdDestinations block tells Grafana to render exemplars as a link to Tempo. Clicking the icon in a panel opens the matching trace.

How to validate it

Three checks confirm the bridge is live.

# READ-ONLY: confirm Prometheus is storing exemplars.
curl -s 'http://prometheus.internal:9090/api/v1/query_exemplars?query=http_server_requests_seconds_count%7Bmethod%3D%22POST%22%2Curi%3D%22%2Fcheckout%22%2Cstatus_code%3D%22500%22%7D&start=2026-08-13T03:00:00Z&end=2026-08-13T04:00:00Z' \
  | jq '.data.exemplars | length'

12

A non-zero count means exemplars exist for the time window. The trace_ids in the response are the join keys:

# READ-ONLY: confirm the trace exists in Tempo.
curl -s -u "${TEMPO_USER}:${TEMPO_PASS}" \
  'https://tempo.internal.example.com/api/traces/8f1d...a3c' \
  | jq '.resourceSpans | length'

1

A non-zero resourceSpans means Tempo has the trace. The Grafana panel test is the operational verification: open a panel that shows http_server_requests_seconds_count with status_code 500; a small diamond icon should appear next to recent data points; clicking the icon should open the trace.

How it can fail

Six failure modes specific to the metric-trace bridge.

1. The metric without an exemplar. Grafana renders the panel; no exemplar icon appears. Cause: the OTel meter provider was not configured to attach exemplars, or the Prometheus build is missing --enable-feature=exemplar-storage. Confirm with the query_exemplars API call.
2. The exemplar with a trace_id that does not resolve. The diamond icon appears; clicking it opens a 404 in Tempo. Cause: Tempo has dropped the trace due to retention, or the trace_id in the exemplar is from a different Tempo tenant than the dashboard expects.
3. The metric labels that do not match the trace attributes. The metric has uri="/checkout"; the trace has http.target="/checkout". Cause: the metric uses the OTel semantic convention http.target while the trace uses a legacy convention; the join by uri does not find the trace. The discipline is to keep metric labels and trace attributes aligned through one naming convention.
4. The exemplar that points at an uninteresting trace. The exemplar links to a successful request because the metric is count_total and the rate is dominated by successes. Cause: the SDK attaches the exemplar to the last request in the bucket, not a failing one. Filter the panel by status_code to see exemplars for the failing bucket.
5. The exemplar that points at a sampled-out trace. The trace_id resolves in Tempo but the trace has fewer spans than expected. Cause: tail-based sampling at the collector sampled the trace at 1% and the saved trace is a partial one. The exemplar should point at a representative trace; the sampling decision affects which trace is representative.
6. The cardinality that explodes the exemplar store. Exemplars are stored per (metric, label-set) per scrape window. A metric with thousands of label-sets costs thousands of exemplars. Cause: a label-set includes a high-cardinality attribute like user_id. The discipline is the same as for the metric: cap the cardinality at the source.

How to troubleshoot it

When the bridge does not work, the order matters.

1. Confirm Prometheus has the feature enabled. curl -s 'http://prometheus.internal:9090/api/v1/status/runtimeinfo' | jq '.data.featureFlags' should include exemplar-storage. If absent, the Prometheus process was started without the flag; the metric stream is correct, exemplars are simply not stored.
2. Confirm the SDK is attaching exemplars. The OTel meter provider must be created after the tracer provider; the meter provider reads the current span context for every measurement. An SDK that initialises the meter first has no span context to read; exemplars are absent.
3. Confirm Tempo has the trace. The trace_id in the exemplar must resolve. Use Tempo’s /api/traces/{id} endpoint to confirm. A 404 means retention has dropped the trace; the exemplar is correct but the link is broken.
4. Confirm the data source link in Grafana. The exemplarTraceIdDestinations block in the Prometheus data source must point at the Tempo data source by datasourceUid. A typo in the UID leaves the icon visible but the link unconfigured.
5. Filter the panel to the failing bucket. A panel showing the total rate has exemplars from the dominant bucket (usually 200s). Filter by status_code=500 to see exemplars for the failures.

Security implications

Exemplars carry a trace_id. The trace_id is a pointer to a trace in Tempo. The trace may contain PII in span attributes. The discipline is the same as for the trace itself:

• Treat the exemplar link as access to the trace. A user who can click an exemplar in Grafana can read the trace in Tempo. The Grafana ACL and the Tempo ACL must agree on access.
• Do not embed secrets in span attributes. An exemplar that points at a trace whose attributes contain an API key leaks the key to anyone who clicks the link. The discipline is the same as for the trace storage itself: redact before the span is exported.

Performance implications

The exemplar is cheap to produce. The SDK attaches one trace_id to one metric increment per scrape window per series; the cost is a few bytes per metric increment and a hash lookup to find the current span context. The cost on the Prometheus side is also bounded: Prometheus retains one exemplar per series per scrape window and drops older exemplars when the bucket fills.

The dominant cost is the storage of the trace the exemplar points at. That cost is the trace’s cost, not the exemplar’s.

Production guidance

• Enable exemplar storage on every Prometheus deployment. The flag is opt-in; the cost is bounded; the operational value is high.
• Configure the Grafana data source to link exemplars to Tempo. The data source link is the last step; without it, exemplars are stored but invisible.
• Validate the bridge after every Prometheus or SDK upgrade. A query_exemplars call that returns zero exemplars is the early-warning indicator that the bridge has been broken by a default change.
• Cap metric cardinality at the source. The exemplar cost is bounded by Prometheus; the metric’s own cost is not. The discipline is the same as without exemplars: cap the label-set at the application before the metric is emitted.

Verification

You should now be able to answer:

• What is the cardinality cost difference between a metric series and a trace span?
• What three pieces of information does an exemplar carry?
• Which Prometheus flag enables exemplar storage?
• Why does an exemplar that points at an uninteresting trace happen, and what is the panel-level fix?
• What is the first API call to confirm exemplars are being stored after a Prometheus upgrade?

Quiz