Context, Correlation, and Causality

Three signals in three silos is three dashboards you flip between. Three signals joined by a correlation ID is one investigation. The protocol that carries the correlation — trace context — is the difference between a platform that looks rich and a platform that is rich.

This lesson explains the difference between correlation and causation, the role of correlation IDs, and the engineering discipline of propagating context across service boundaries.

Correlation vs causation

Two pieces of evidence correlate when they happen together. They have a causal relationship when one of them causes the other. The distinction is the heart of investigation:

• A spike in 5xx error rate correlates with a spike in CPU. The correlation is real. The causation may be either direction: a CPU spike may cause error rate to climb (resource exhaustion), OR the error rate spike may cause extra work that consumes CPU (retry loops). The investigator must determine the direction before acting.
• A spike in 5xx correlates with a deploy. Again, direction matters. The deploy may have caused the failure (the usual case), or the failure may have triggered a rollback that happens to look like a deploy.

The discipline of “form hypothesis, find evidence, test” is what distinguishes correlation from causation. The platform’s role is to make correlated evidence easy to find; the investigator’s role is to test the direction.

What context propagation is

Context propagation is the discipline of carrying a correlation identifier — a trace ID, a request ID — across service boundaries so the same identifier appears in:

• A trace span in service A
• A trace span in service B
• A structured log line in service A
• A structured log line in service B
• A metric exemplar

The trace ID is the canonical correlation primitive. It is a 16-byte globally unique identifier. Distributed tracing systems (Tempo, Jaeger, Zipkin, OpenTelemetry Collector backends) carry it as a span attribute. Structured logging systems (Loki, ELK) carry it as a log field. Metrics that emit exemplars (Prometheus with OpenMetrics exemplar support) carry it as an exemplar reference.

The W3C Trace Context specification defines two HTTP headers — traceparent and tracestate — that services use to propagate the identifier. Service A emits traceparent; service B reads it; service B’s spans inherit the trace ID.

The pivot-across-signal workflow

The same correlation ID appears in all three signals. The investigator pivots across signals using the ID:

Service-level alert (metric)
   ↓
Open dashboard
   ↓
Identify affected service and time window
   ↓
Click an exemplar → opens a trace
   ↓
Read the trace timeline for a slow span
   ↓
Click the span to jump to related logs (filter by trace ID)
   ↓
Logs reveal the dependency / DB query / error message
   ↓
Root cause located

Each pivot relies on the correlation ID surviving. If the ID does not survive a service hop, the investigator is back to flipping between three dashboards manually.

Failure shapes when context is missing

Three failure shapes:

1. Trace ID is not propagated. The trace of one request contains only the spans of one service. Cross-service investigation requires the investigator to match IDs by timestamp manually.
2. Trace ID is propagated, but not into logs. Logs from service B contain no trace ID field. The investigator can filter logs by service and time, but cannot filter by trace ID, so the broad filter returns huge volumes.
3. Trace ID is propagated, but the metric exemplar is missing. The investigator cannot pivot from a metric datapoint to a trace. The metric is independent of the trace stream.

Each failure turns a 30-second pivot into a 30-minute text-search.

Where context propagation breaks in practice

• Cross-language boundaries. A Go service propagates traceparent via a library that handles HTTP headers. A Python service must read the same headers. A Java service with Spring Boot’s tracing support picks them up automatically. A C++ service with no tracing support does not. The boundary is a place where the ID is dropped.
• Asynchronous boundaries. A service enqueues a message into a queue. The consumer reads the message. Trace propagation across the queue requires the producer to write the trace ID into the message and the consumer to extract and reinstate it. Many queues do not propagate by default.
• Internal batch jobs. A cron job runs every 5 minutes and has no inbound request. Trace context is generated internally; the spans share a trace ID with each other but are not traceable to a user-facing request.
• Storage I/O. A SQL query does not carry a trace ID unless the database driver or the application explicitly attaches it as a comment.

The OpenTelemetry SDKs handle propagation when configured. The failures are the gaps where the SDK cannot reach: cron jobs, hand-written HTTP clients, queue producers, and stored procedures.

Engineering discipline for context propagation

The discipline has four parts:

1. Configure the SDK at every service boundary. HTTP clients, HTTP servers, gRPC clients, message producers, message consumers. The SDKs are configured once at startup; propagation is automatic for every request.
2. Log the trace ID. Every structured log line includes trace_id (and ideally span_id). Loki’s LogQL can filter by the field. LogQL can filter by the field.
3. Emit exemplars from histograms. Prometheus histograms can carry exemplars (OpenMetrics format). Tempo picks the exemplars up and links them to traces.
4. Audit the propagation periodically. A scheduled request generates a trace at each hop; the audit checks the trace ID survives end-to-end. An end-to-end trace audit is one of the highest-value platform tests.

Production guidance

• All services should emit a trace for every request. The service-level cost is small; the operational payoff is large.
• All services should log the trace ID. LogQL filters by trace ID are the highest-leverage query pattern.
• All histograms should emit exemplars. The exemplar is the pivot from metric datapoint to trace.
• The trace propagation should be tested regularly. An end-to-end trace test catches regressions early.
• Where propagation is impossible (batch jobs, stored procedures), log a synthetic correlation ID that ties the batch job to its trace spans.

Verification

You should be able to answer:

• What is the operational difference between correlation and causation?
• Why does a correlation ID across signals outperform three separate dashboards?
• Where does trace propagation break in practice, and how do you fix it?
• What is the production discipline for ensuring context propagation is maintained?

Quiz