Correlation IDs in Logs

The incident review asks: “show me every log line for the failed checkout at 03:12:44.” The on-call engineer runs a LogQL query against Loki. The query returns two lines — one from the API gateway and one from the database. The two services that actually decided to fail the request (the inventory service and the pricing service) are silent. The reason is that those two services do not see the request_id that the gateway stamped. They generated new IDs of their own. The story cannot be told.

This is the lesson. A correlation ID is the thread that ties the log lines from every service that touched one request into a single narrative. Without it, the logs are searchable, but not in the way the on-call engineer needs.

What a correlation ID is

A correlation ID is a unique identifier stamped on every log line produced while processing a single unit of work. The convention has three identifiers in modern systems:

• request_id — opaque UUID generated at the system edge (the load balancer or API gateway). Lives for the duration of the request. Propagated across every downstream call.
• trace_id — 128-bit identifier from W3C Trace Context. Generated by the OpenTelemetry SDK at the same point as the request_id. Propagated by the SDK on every outbound call.
• span_id — 64-bit identifier for a single unit of work inside the trace. Every service creates at least one; nested calls create child spans.

The trace_id and request_id are often the same value in simple systems, but the contract is different. The trace_id is owned by the tracing system and follows the W3C propagation rules. The request_id is owned by the application and follows whatever convention the team adopts.

Why a sysadmin cares

Three operational payoffs depend on correlation IDs.

1. Single-query reconstruction. One LogQL query, one ID, every log line from every service. The investigation stops being a cross-tab exercise and starts being a copy-paste.
2. Trace-to-logs pivot. Grafana 11.x supports the pivot natively: a Tempo trace has a button that opens Loki filtered by the trace’s trace_id. The pivot requires the trace ID to appear in the structured log payload.
3. Failure boundary identification. When the ID disappears at a specific hop, the bug is at that hop’s propagation code. The investigation becomes a one-file read instead of a fleet-wide search.

The cost is one header on every outbound call. The discipline is in the propagation library, not in the application code.

How it works — the mental model

Edge
  load balancer --generates request_id--> API gateway
                                          |
                                          +-- X-Request-ID: 7f4a1c
                                          |
                                          v
  service A (inventory)
    reads X-Request-ID from incoming request
    stamps 7f4a1c on every log line it emits
    outbound call to service B --X-Request-ID: 7f4a1c-->
                                          |
                                          v
  service B (pricing)
    reads X-Request-ID from incoming request
    stamps 7f4a1c on every log line it emits
    outbound call to message bus --header: 7f4a1c-->
                                          |
                                          v
  consumer (warehouse picklist)
    reads the header from the message envelope
    stamps 7f4a1c on every log line it emits

The crucial point is that the ID is stamped at the edge and read at every hop. A service that generates its own ID instead of reading the incoming one breaks the chain. The bug is not visible in the service itself — every line still has a correlation ID — it is visible only when the investigator tries to follow the chain across the hop.

How to configure it

The application side — Go with the OpenTelemetry SDK and a structured logger:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    "log/slog"
)

func main() {
    // Honour incoming trace context from any peer.
    otel.SetTextMapPropagator(propagation.TraceContext{})

    // Extract the trace_id from the current span and stamp it on
    // every log line emitted inside the request.
    handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        Level: slog.LevelInfo,
    })
    logger := slog.New(&traceHandler{Handler: handler})
    slog.SetDefault(logger)
}

type traceHandler struct{ slog.Handler }

func (h *traceHandler) Handle(ctx context.Context, r slog.Record) error {
    span := trace.SpanFromContext(ctx)
    if span.SpanContext().IsValid() {
        r.AddAttrs(slog.String("trace_id", span.SpanContext().TraceID().String()))
        r.AddAttrs(slog.String("span_id",  span.SpanContext().SpanID().String()))
    }
    return h.Handler.Handle(ctx, r)
}

The HTTP side — middleware that reads X-Request-ID from the incoming request and stamps it on the response:

func requestID(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID")
        if id == "" {
            id = uuid.NewString()
        }
        w.Header().Set("X-Request-ID", id)
        ctx := withRequestID(r.Context(), id)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

The message-bus side — Kafka headers carry the correlation ID across the async boundary:

producer.Input() <- &sarama.ProducerMessage{
    Topic: "picklist",
    Headers: []sarama.RecordHeader{
        {Key: []byte("X-Request-ID"), Value: []byte(requestIDFromContext(ctx))},
    },
    Value: sarama.ByteEncoder(payload),
}

The consumer side — read the header before invoking the handler:

for msg := range consumer.Messages() {
    var id string
    for _, h := range msg.Headers {
        if string(h.Key) == "X-Request-ID" {
            id = string(h.Value)
        }
    }
    ctx := withRequestID(context.Background(), id)
    handle(ctx, msg)
}

How to validate it

The validation ladder:

# 1. The edge stamps a request ID.
curl -s -D - https://api.example.com/checkout -o /dev/null | grep -i x-request-id
# X-Request-ID: 7f4a1c8e9b1d4e2a

# 2. The same ID appears in every downstream service.
logcli query --since=1h '{service=~"checkout|inventory|pricing"} | json | request_id="7f4a1c8e9b1d4e2a"'
# 2026-01-15T03:12:44.512Z {} service=checkout ... request_id=7f4a1c8e9b1d4e2a
# 2026-01-15T03:12:44.518Z {} service=inventory ... request_id=7f4a1c8e9b1d4e2a
# 2026-01-15T03:12:44.601Z {} service=pricing ... request_id=7f4a1c8e9b1d4e2a

# 3. The trace ID is honoured end to end.
tctl trace show 0af7651916cd43dd8448eb211c80319c
# (Tempo returns the full span tree)

# 4. The trace-to-logs pivot works in Grafana.
# Open the trace in Grafana -> click "Logs for this trace" ->
# the pivot opens Loki filtered by trace_id=0af7651916cd43dd8448eb211c80319c.

# 5. Async hops carry the ID.
logcli query --since=1h '{service="warehouse-picklist"} | json | request_id="7f4a1c8e9b1d4e2a"'
# (the consumer received the ID via the Kafka header)

How it can fail

Six recurring failure modes.

1. The edge does not stamp an ID. The load balancer strips X-Request-ID instead of generating one. Every downstream service generates its own. Symptom: the LogQL query for request_id="X" returns one line per service instead of every line from every service.
2. The header is renamed. Service A expects X-Request-ID; Service B sends X-Correlation-Id. Symptom: the chain breaks at the A-to-B hop; the on-call engineer concludes Service A is the culprit when Service A is innocent.
3. A service generates a new ID instead of reading the incoming one. The middleware is missing or short-circuited. Symptom: every line from Service B has a fresh ID; the chain breaks at B’s ingress.
4. The message bus drops the header. The Kafka producer is not configured to copy the ID into the record headers, or the consumer is not configured to read them. Symptom: the async hop is invisible in Loki; the trace spans say the work happened but the logs do not.
5. The trace_id is not propagated into logs. The OpenTelemetry SDK is configured, but the log handler does not read the trace context. Symptom: Tempo has the trace, Loki has no trace_id field, the Grafana pivot button is greyed out.
6. The cookie or session value carries a different ID. The user-facing session ID is unrelated to the per-request request_id. The team conflates the two and queries the session ID across the logs. Symptom: the LogQL query returns nothing, and the on-call engineer concludes the system is broken.

How to troubleshoot it

The diagnostic order for “I cannot find all the logs for one request”:

1. What ID do you have? Confirm the source. Is it the request_id, the trace_id, the session cookie, or a UI correlation token from a CDN? Mixing them is the most common investigation failure.
2. Does the ID appear in the source service? logcli query '\{service="edge"\} | json | request_id="X"'. If the answer is zero, the ID is not being stamped.
3. At which hop does the ID disappear? Query each service in the path with the same ID. The first service that returns zero lines is the boundary that lost the ID.
4. Is the trace present in Tempo? tctl trace show <id>. If Tempo has it but Loki does not, the trace_id is not being stamped into the log handler.
5. Is the async hop configured? Inspect the producer and consumer code. The fix for a broken Kafka hop is the header propagation, not the application logic.

Security implications

The request_id and trace_id are not sensitive. They are opaque identifiers with no semantic content. They are safe to log, safe to forward, and safe to store at any retention.

The risk is around the values that look like correlation IDs but are not. A session cookie, an OAuth access token, or an Authorization header can be mistaken for a request_id by an analyst running a LogQL query. The convention is to log a hash or a prefix of the session token, not the value, and to keep the request_id as a separate field with a separate name.

The second-order risk is around propagation. A service that echoes the X-Request-ID header back to the client without validation is one step away from a header-injection vulnerability. The remediation is to allow only UUID-shaped values in the header, or to overwrite the header at the edge with a freshly generated UUID.

Performance implications

The cost of a correlation ID is one string per log line and one header per outbound call. The string is roughly 36 bytes (a UUID) or 32 bytes (a hex trace_id). The header is roughly 70 bytes on the wire. At 10 000 requests per second, the header overhead is 700 KB/s of egress — measurable but not material.

The expensive path is the per-line trace_id extraction in the log handler. Reading the current span context, formatting the trace ID, and adding it to the record costs roughly 100 ns per line. At 50 000 lines per second, that is 5 ms of CPU per second — well under 1 percent of a single core. The cost is worth paying.

Production guidance

• Stamp at the edge. The load balancer or API gateway generates the request_id and the trace_id if no incoming traceparent is present.
• Propagate everywhere. Every HTTP client, every gRPC client, every message-bus producer must copy the ID into the outbound envelope.
• Stamp in every log line. The handler reads the active span context and adds the trace_id and span_id to the structured record.
• Pivot in Grafana. Configure the Tempo data source to send trace_id queries to Loki. Configure the Loki data source to accept the pivot. The Grafana docs for trace-to-logs cover the exact wiring.

Verification

You should now be able to answer:

• What is the difference between a request_id and a trace_id?
• Where is the correlation ID generated, and where is it read?
• How is the ID propagated across an asynchronous message-bus hop?
• What does the trace-to-logs pivot in Grafana require?

Quiz