Correlation Anatomy

A user reports that checkout fails for one in three attempts at 03:00. CPU, memory and disk dashboards are green. The on-call engineer opens Grafana. There is a metric panel for the error rate, a log panel for the application, and a trace panel for the checkout journey. The three panels sit side by side. None of them joins to the others. The engineer has to copy labels by hand, paste them into a Loki query, then find the trace ID in the log line, then paste it into a Tempo search. The investigation takes forty-five minutes. The root cause is a 200 millisecond database timeout that the trace would have shown in two clicks.

The waste is the join. The metric, the log, and the trace all describe the same request. None of them carries the identifier needed to reach the others without manual copy. Correlation anatomy is the discipline that gives every signal the identifier needed to find the other two.

What it is

Correlation anatomy is the set of contracts and identifiers that allow one telemetry signal to address the others. Three signals exist in a production observability stack:

• Metrics — numeric values sampled over time at Prometheus, stored as a metric name plus a label set. Cheap, alertable, aggregatable. The signal of how many and how long.
• Logs — discrete events at Loki, indexed by a small set of stream labels and looked up by full text. The signal of what happened.
• Traces — the journey of a single request through the distributed system, stored at Tempo. The signal of where the time went.

The three signals are not interchangeable. The discipline is that each one carries the same shared identifier — a trace_id — and that identifier is the join key. With it, an operator moves from one signal to the others without losing the request context.

Why a sysadmin cares

Three operational payoffs.

1. Investigation time. A correlated stack turns a forty-five minute investigation into a five-minute one. The on-call engineer clicks the metric panel, sees the logs for that service, sees the trace for that request, and reaches the root cause before the coffee is cold.
2. Onboarding. A new engineer can follow a real incident end to end through the three signals. The signal flow is the documentation the team does not have to write.
3. Tooling leverage. Grafana 11.x exposes the correlation as native UI features: trace to logs, trace to metrics, metrics to traces, logs to traces. The features only work when the join key is present on every signal.

The cost is the discipline of stamping the identifier at every service boundary. The investment is small. The return is paid on every incident.

How it works — the shared identifier

The trace_id is the canonical join key. It is a 128-bit hexadecimal value generated by the OpenTelemetry SDK at the system edge, propagated through every service in the request path, and stamped on every log line and every metric exemplar that the request produces.

                Edge (load balancer)
                      |
                      +-- trace_id = 0af7651916cd43dd8448eb211c80319c
                      |
                      v
              +---------------+
              |  API gateway  |  span A
              +---------------+
                      |
                      v
              +---------------+        +---------------+
              |  order svc    | -----> |  payment svc   |
              |  span B       |        |  span C        |
              +---------------+        +---------------+
                      |
        +-------------+--------------+
        |             |              |
        v             v              v
    metric         log line        trace
    exemplar     (trace_id=fld)   (Tempo)
        |             |              |
        +-------------+--------------+
                      |
                   Grafana UI
                   (pivot)

Four contracts make the join work.

1. Stamp at the edge. The first service to handle a request that does not see an incoming traceparent header generates the trace_id.
2. Propagate on every hop. Every HTTP client, every gRPC client, every Kafka producer injects the header on the outbound call. The OpenTelemetry SDK does this automatically.
3. Stamp on every telemetry event. The log handler reads the active span context and adds the trace_id to the structured log record. The histogram exemplars API attaches the trace_id to a representative observation.
4. Pivot in Grafana. The data source configuration adds derived fields (Loki) and exemplar links (Prometheus) that turn a click in one signal into an open query in the next.

Correlation vs pivoting

The two words are used interchangeably in casual conversation. They are not the same thing.

Correlation is the data-side discipline. It is the property that the three signals share the same identifier for the same request. Correlation is set up by the OpenTelemetry SDK, the log handler, and the histogram exemplars API. It is invisible to the user.

Pivoting is the UI-side navigation. It is the click in Grafana that opens a query in a different data source, using a value from the row or panel as the filter. Pivoting is configured in the panel data links or the data source derived fields. It is visible to the user.

   correlation  (data-layer, set up by SDK and handlers)
        |
        |   produces
        v
   join key on every signal
        |
        |   consumed by
        v
   pivoting     (UI-layer, set up in Grafana)

A correlation without a pivot exists but is invisible. The on-call engineer can write a LogQL query by hand if they know the trace ID, but the Grafana button is missing. A pivot without a correlation is a broken link. The button is there but the query returns no rows.

How to configure it

The Grafana side — the data sources declared in the provisioning YAML. The Loki data source gets the derivedFields that turn a trace_id field into a Tempo link. The Tempo data source gets the traceToLogs block that turns a trace ID into a Loki query. The Prometheus data source gets the exemplar toggle that surfaces trace IDs alongside the buckets.

# grafana/provisioning/datasources/observability.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      # Surface the trace_id captured by the histogram exemplar
      # next to the bucket the operator clicked.
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
          urlDisplayLabel: 'Open trace'

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    jsonData:
      # Regex expects the compact JSON form emitted by the
      # structured log handler. The anchor on the closing quote
      # prevents the matcher from drifting into the next field.
      derivedFields:
        - name: traceID
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace in Tempo'

  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    jsonData:
      httpMethod: GET
      # The trace-to-logs pivot opens Loki with the same tags
      # carried by the trace and a 500ms lookback so the log
      # line at the start of the trace is included.
      traceToLogs:
        datasourceUid: loki
        tags: ['job', 'instance', 'status']
        query: '{${__tags}} | json | trace_id="$${__trace_id}"'
        spanStartTimeShift: -500ms
        spanEndTimeShift: 0

The application side — the OpenTelemetry SDK bootstraps the propagator:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# W3C TraceContext is the SDK default; set it explicitly so an
# upgrade cannot silently swap the propagator.
set_global_textmap(TraceContextTextMapPropagator())

The log handler — Python logging with the OpenTelemetry log SDK, stamping the current trace_id on every record:

import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        span = trace.get_current_span()
        ctx = span.get_span_context()
        if ctx.is_valid:
            record.trace_id = format(ctx.trace_id, '032x')
        return True

How to validate it

# 1. The edge stamps a traceparent.
curl -sv \
  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-aaaaaaaaaaaaaaaaaa-01' \
  http://gateway/api/v1/checkout 2>&1 | grep -i traceparent
# > traceparent: 00-0af7651916cd43dd8448eb211c80319c-aaaaaaaaaaaaaaaaaa-01
# < traceparent: 00-0af7651916cd43dd8448eb211c80319c-1f2e3d4c5b6a0987-01

# 2. The same trace_id appears in Loki.
logcli query --since=10m \
  '{service="checkout"} | json | trace_id="0af7651916cd43dd8448eb211c80319c"'
# 2026-08-13T03:00:12Z {service="checkout"} msg="payment failed" trace_id=0af7651916cd43dd8448eb211c80319c

# 3. The same trace_id appears in Tempo.
tempo-cli query '{ trace &#61; "0af7651916cd43dd8448eb211c80319c" }'
# Span: 0af7651916cd43dd8448eb211c80319c  service=checkout  duration=4.2s

# 4. The exemplar is present on the histogram bucket.
curl -s 'http://prometheus:9090/api/v1/query?query=http_server_request_duration_seconds_bucket' \
  | jq '.data.result[0].exemplar'
# {"labels":{"trace_id":"0af7651916cd43dd8448eb211c80319c"},"value":"4.2"}

How it can fail

Six recurring failure shapes.

1. The trace_id is not stamped on logs. The log handler does not read the active span context. Symptom: Tempo has the trace, Loki has no trace_id field. The Grafana trace to logs button is greyed out.
2. The Loki derived field regex does not match. The JSON payload is compact, the regex expects spaces. The matcher never fires. Symptom: the log lines display without a clickable trace ID. The drift is invisible until an operator clicks a line and nothing happens.
3. The exemplar toggle is off on the Prometheus data source. Grafana knows about exemplars but the UI hides them. Symptom: the metric panel shows the histogram with no exemplar dots. The metric to trace pivot is invisible.
4. The trace_id is truncated. A misconfigured log pipeline truncates strings over 32 characters. The first 32 characters of the trace_id are logged. The Tempo query for the full 32 returns no spans. Symptom: the log to trace pivot opens Tempo, Tempo returns no spans, the operator concludes the trace is missing.
5. The exemplars feature flag is off on Prometheus. Old Prometheus builds do not store exemplars. Symptom: the # HELP and # TYPE lines declare the histogram; the /api/v1/query endpoint returns no exemplar field.
6. The Tempo data source URL points to the wrong query frontend. The Grafana data source uses the wrong Tempo endpoint. Symptom: the trace to logs button on a Tempo trace opens a blank page; the URL has the trace ID but the query returns 404.

How to troubleshoot it

The diagnostic order is “is the identifier on the signal?”, “is the derived field wired?”, “is the data source configured?”, “is the pivot visible?”.

1. Is the trace_id on the log line? logcli query '{service="checkout"} | json | trace_id!=""'. The first line of the output should include a 32-hex value. If it is empty, the log handler is the suspect.
2. Is the derived field matched? Inspect the Loki data source configuration in Grafana (Administration → Data sources → Loki → Derived fields). Run a curl -s http://loki:3100/loki/api/v1/query on a known log line and inspect the derivedFields section in the JSON response.
3. Is the exemplar toggle on? curl -s http://prometheus:9090/api/v1/status/config | grep -A2 exemplars. The flag must be enabled at startup.
4. Is the data source URL right? curl -s http://tempo:3200/api/status. The Grafana data source URL must reach the same backend.
5. Is the UI seeing the pivot? Open a log line in Grafana, hover over the trace_id field. The “Open trace” link must appear. If it does not, the matcher is misconfigured.

Security implications

The trace_id is a 128-bit random value. It is opaque and carries no semantic content. It is safe to log, safe to forward, and safe to store at any retention.

The second-order risk is around values that look like trace IDs but are not. A session cookie, an OAuth token, or a PII field can be misclassified as a trace_id by an analyst chasing a correlation. The mitigation is to keep the trace_id as a separate field with a separate name, and to log only the ID itself, never the surrounding request context.

The third risk is around the derived field regex. A regex that is too loose (for example, matcherRegex: '[a-f0-9]+') will match the wrong substring and produce a broken link. The convention is to anchor the regex with a separator (comma, quote, brace) and to require the exact 32-hex length.

Performance implications

The cost of the correlation is small.

• One header per outbound call. The traceparent header is 70 to 110 bytes on the wire. At 10 000 requests per second, the header overhead is under 1 MB/s of egress.
• One attribute per log line. The trace_id field is 32 bytes. Loki’s structured metadata indexing adds roughly 100 ns per line. At 50 000 lines per second, that is 5 ms of CPU per second.
• One exemplar per histogram bucket. The exemplar is attached to a single bucket per scrape. Prometheus stores one exemplar per bucket per series. The cost is fixed, not per-observation.

The benefits — sub-minute investigations and Grafana pivots that work — outweigh the cost.

Production guidance

• Standardise on W3C TraceContext. The OTel default is the right production default. B3 is legacy and should be migrated away from.
• Audit every signal producer. The log handler, the histogram exemplars API, and the data source provisioning YAML each need a review. The audit is a single-sentence spec: every log line carries a trace_id; every histogram bucket carries an exemplar; every derived field has a matching regex.
• Run a synthetic test. A scheduled curl with a known trace_id, asserted at every signal, is the only way to catch the correlation breaking after a refactor.
• Treat the correlation as a service. The discipline has an owner. The owner writes the spec, the audit, and the test. The rest of the team uses the pivot.

Verification

You should now be able to answer:

• What is the shared identifier that joins metrics, logs, and traces?
• What is the difference between correlation and pivoting?
• Where is the trace_id stamped, and where is it read?
• Which four contracts make the join work end to end?
• Which Grafana feature on each signal exposes the pivot?

Quiz