End-to-End Correlation Test

A team ships a new version of an internal SDK that “simplifies” the W3C trace context propagation. The change passes unit tests. The change is deployed on a Tuesday. By Friday the on-call engineer is staring at a Loki line with a trace_id that resolves to an empty span tree in Tempo. The propagator is silently dropping the parent context after the first hop. The unit tests never caught it because the unit tests asserted the SDK’s own span, not the chain.

The fix is not in the unit tests. The fix is the end-to-end correlation test: a scheduled synthetic request that exercises every pivot in the chain — application context, log handler, log shipping, structured metadata promotion, Loki ingest, Tempo ingest, Grafana data link — and asserts every invariant. When the test passes, every pivot the engineer relies on works. When it fails, the team knows exactly which pivot broke and where in the chain.

This lesson is the discipline of writing that test. The four invariants are what make a test end-to-end. The cadence is what makes the test catch regressions before the on-call engineer sees them in production.

What it is

An end-to-end correlation test is a scheduled synthetic request that exercises the full pipeline from the application’s OpenTelemetry context to the Grafana data-link click. The test is not a unit test (which asserts one component in isolation) and not an integration test (which asserts one boundary). The test is the whole chain: span emission, log handler, log shipping, Loki structured metadata, Tempo trace resolution, Grafana data link. The test asserts four invariants:

1. Log line shape. The application emitted a structured trace_id field of exactly 32 lowercase hex characters.
2. Loki structured metadata. The field is queryable as structured metadata, not as a line filter.
3. Tempo resolution. The trace_id resolves to a trace block with at least one span from each service in the request path.
4. Grafana data link. The Loki data source has a derivedFields rule whose datasourceUid points at a real Tempo data source; the Tempo data source has a tracesToLogsV1 block whose datasourceUid points at a real Loki data source.

A test that asserts only one of these is not end-to-end. A test that asserts the whole chain is the production guarantee.

Why a sysadmin cares

Three operational payoffs depend on the test:

• Pre-deployment detection. A test that runs on every CI pipeline catches a broken propagator or a typo’d UID before it reaches production. A test that runs on a 30-min cadence catches a service that degrades over hours.
• Post-incident verification. After a correlation outage, the test is the criterion for “is the platform actually fixed”. A team that “fixed” the regex by hand without re-running the test is guessing.
• Regression memory. A change in OTel SDK auto-config defaults, in Loki’s structured-metadata promotion syntax, or in Grafana’s data source provisioning schema can break pivots silently. The end-to-end test is the only mechanism that detects a silent regression on every release.

How it works

The test is a synthetic request, made by a probe, against the production (or staging) fleet. The probe is a small service that emits a trace, makes a request through the relevant boundary, captures the trace_id from the log line, and asserts the invariants. The schedule is a cron-style run from the operator’s CI or a hosted synthetic probe.

probe (scheduled)
   |
   |  generate root span via OpenTelemetry SDK
   |  trace_id = 4bf92f3577b34da6a3ce929d0e0e4736 (random)
   |
   |  POST /checkout  (with W3C traceparent header)
   v
+--------------------------+
| checkout-api             |
|   reads traceparent      |
|   makes child span       |
|   emits 5 log lines      |
|   each with trace_id     |
+--------------------------+
   |
   |  calls payments-api with traceparent
   v
+--------------------------+
| payments-api             |
|   reads traceparent      |
|   makes child span       |
|   emits 3 log lines      |
+--------------------------+
   |
   |  wall-clock wait (15 s for Tempo to flush + Loki to index)
   v
+--------------------------+
| probe assertions         |
|   1. log line shape      |
|   2. Loki structured     |
|   3. Tempo trace exists  |
|   4. Grafana UID match   |
+--------------------------+

Four observations:

1. The probe is the test. It calls the production endpoint (or staging), not a mocked one. The chain is the production chain; the test runs against the real pipeline.
2. The wall-clock wait is the slowest bottleneck. Tempo flushes traces every 5 s by default; Loki indexes structured metadata within a few seconds; Grafana provisions data sources on provisioning reload. A test that fires “submit + assert” gets a flaky failure; a test that fires “submit + 15 s + assert” gets a stable pass.
3. Each pivot is one assertion. Splitting the chain into assertion units lets the failure point to the broken pivot. A single mega-assertion “the chain works” is correct but uninformative when it fails.
4. The cadence is a sibling of the alert budget. A test that runs every 30 minutes catches a 30-minute regression. The cost is one synthetic request per cycle; on a busy fleet this is a rounding error.

How to configure it

A probe in Go that exercises the chain end-to-end:

// probe/main.go — sends a request and asserts four invariants.
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "regexp"
    "strings"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

var traceIDRe = regexp.MustCompile(`^[0-9a-f]{32}$`)

func main() {
    // 1. Send the request from inside an OpenTelemetry span.
    // The trace_id generated by the SDK is what we assert against.
    tp := sdktrace.NewTracerProvider()
    defer tp.Shutdown(context.Background())
    otel.SetTracerProvider(tp)
    tracer := otel.Tracer("probe")

    ctx, span := tracer.Start(context.Background(), "probe.request")
    defer span.End()

    // Inject the W3C traceparent header onto the outbound request.
    req, _ := http.NewRequestWithContext(ctx, "POST",
        "https://api.internal/checkout", strings.NewReader(`{"probe":true}`))
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

    probeID := fmt.Sprintf("probe-%d", time.Now().Unix())
    req.Header.Set("X-Probe-Id", probeID)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatalf("step 1 fail: request did not complete: %v", err)
    }
    defer resp.Body.Close()

    // The trace_id is on the current span.
    traceID := span.SpanContext().TraceID().String()
    if !traceIDRe.MatchString(traceID) {
        log.Fatalf("step 1 fail: trace_id %q is not 32 lowercase hex", traceID)
    }
    log.Printf("step 1 pass: trace_id=%s", traceID)

    // Wait for Tempo to flush and Loki to index.
    time.Sleep(15 * time.Second)

    // 2. Loki structured metadata: the trace_id must be
    // queryable as metadata, not as a line filter.
    lokiQ := fmt.Sprintf(
        `{service_name="checkout-api"} | trace_id="%s" | probe_id="%s" | head 1`,
        traceID, probeID,
    )
    if !lokiLineMatches(lokiQ) {
        log.Fatalf("step 2 fail: Loki does not have trace_id as structured metadata")
    }
    log.Printf("step 2 pass: Loki has the trace_id field")

    // 3. Tempo resolution: the trace_id must resolve.
    if !tempoHasTrace(traceID) {
        log.Fatalf("step 3 fail: Tempo has no trace for %s", traceID)
    }
    log.Printf("step 3 pass: Tempo resolves the trace_id")

    // 4. Grafana data link: the Loki data source's derived
    // fields must point at a real Tempo UID, and vice versa.
    if !grafanaCorrelationIntact() {
        log.Fatalf("step 4 fail: Grafana data links are misconfigured")
    }
    log.Printf("step 4 pass: Grafana data links are intact")
}

func lokiLineMatches(query string) bool { /* ... */ return true }
func tempoHasTrace(traceID string) bool  { /* ... */ return true }
func grafanaCorrelationIntact() bool     { /* ... */ return true }

The Prometheus blackbox-exporter variant — for a CI/CD-only check, the blackbox-exporter module http_2xx is the coarsest test. The end-to-end check requires custom Go rather than the blackbox-exporter, but the blackbox variant is a useful pre-check:

# /etc/prometheus/blackbox.yml
modules:
  http_correlation:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"probe":true}'
      valid_status_codes: [200]
      fail_if_matches_regexp: ["trace_id=00000000000000000000000000000000"]

The CI integration — run the probe on every merge to main:

# .github/workflows/correlation.yml
name: Correlation E2E
on: [push]
jobs:
  probe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t correlation-probe ./probe
      - name: Send probe and assert
        env:
          LOKI_URL: ${{ secrets.LOKI_URL }}
          TEMPO_URL: ${{ secrets.TEMPO_URL }}
          GRAFANA_URL: ${{ secrets.GRAFANA_URL }}
        run: |
          docker run --rm \
            -e LOKI_URL -e TEMPO_URL -e GRAFANA_URL \
            correlation-probe

How to validate it

# READ-ONLY: run the probe and inspect the output.
go run ./probe
# step 1 pass: trace_id=4bf92f3577b34da6a3ce929d0e0e4736
# step 2 pass: Loki has the trace_id field
# step 3 pass: Tempo resolves the trace_id
# step 4 pass: Grafana data links are intact
#
# Exit status 0 = pass; non-zero exit = fail; the failing
# step is in the "step N fail" line.

# READ-ONLY: confirm the probe is scheduled somewhere.
crontab -l | grep correlation-probe
# */15 * * * * /usr/local/bin/correlation-probe >> /var/log/correlation.log

# READ-ONLY: confirm the recent history of passes and fails.
tail -20 /var/log/correlation.log
# 2026-08-13T14:00:00Z step 1 pass: trace_id=8f1d23a47b1e...
# 2026-08-13T14:15:00Z step 1 pass: trace_id=...
# 2026-08-13T14:30:00Z step 3 fail: Tempo has no trace for ...
# (A step-3 fail means Tempo dropped the trace; check the
# Tempo ingester's queue depth or the network between the
# collector and Tempo.)

# READ-ONLY: confirm the test fires from a synthetic
# monitoring rule that pages on continuous failure.
curl -fsS -u "$GRAFANA_ADMIN" \
  'http://grafana.internal:3000/api/v1/provisioning/contact-points' \
  | jq '.[] | select(.name | test("probe"; "i"))'
# { "name": "probe-pager", ... }
# (The team's pager is wired to fire on three consecutive
# probe failures, not on a single one.)

If step 1 prints “step 1 fail: trace_id is not 32 lowercase hex”, the SDK is mis-configured and the OTel pipeline in the probe is broken. If step 2 fails, the log shipping pipeline is dropping the field. If step 3 fails, Tempo is not receiving or has aged out the trace. If step 4 fails, the Grafana provisioning has a typo’d UID or a removed data source.

How it can fail

1. The probe’s own SDK is uninitialised. A probe that runs without the SDK configured emits the trace with trace_id=0 and the assertions fail at step 1. Symptom: “step 1 fail: trace_id is not 32 lowercase hex”.
2. The probe’s request gets a 5xx. A downstream service is down; the probe’s HTTP call returns 503. The trace_id is generated but no log lines come out. Symptom: “step 1 fail: request did not complete”.
3. Wall-clock too short. A new probe that waits 5 s expects Loki and Tempo to keep up. The test is flaky. Symptom: “step 2 fail” with no Loki errors in the Loki log; the next run passes.
4. Loki structured metadata promotion is broken. A team rolls out an Alloy change that drops the stage.structured_metadata block; Loki lines have no trace_id as metadata. Symptom: “step 2 fail” with the line text containing “trace_id=4bf92f…” (as text) but the structured query returns zero rows.
5. Tempo retention drops the trace between probe cycles. A retention change from 24h to 1h deletes traces faster than the probe cycles. Symptom: “step 3 fail” with the traceID returning 404 against /api/traces/<id>.
6. Grafana provisioning reload not fired. A team renames a UID in the Tempo provisioning file but does not restart Grafana; the old UID is still in the Loki derivedFields. Symptom: “step 4 fail” with the /api/datasources/uid/<uid> API returning 404 against the renamed UID.

How to troubleshoot it

The diagnostic order for “the correlation test fails”:

1. Which step failed? The probe’s exit log says “step N fail”. The first failed step is the broken pivot; the fix is at that pivot.
2. Does the probe’s own SDK produce a valid trace_id? Step 1 is a self-test; a fail here means the probe is unconfigured.
3. Is the request reaching the service? A 5xx from checkout-api means the service is down for a reason unrelated to correlation. Confirm with curl outside the probe.
4. Was the wall-clock enough? Step 2 / step 3 flaky on cold start? Raise the sleep from 15 s to 30 s on the first probe of a cycle.
5. Are the Loki and Tempo APIs reachable? curl /ready against each. A 503 from Tempo means the ingester is overloaded; the test is correct, the platform is the issue.
6. Are Grafana’s data sources provisioned correctly? /api/datasources returns the current state. A failed /api/datasources/uid/<uid> is the source of step-4 fails.

Security implications

The probe is a service that issues real requests against the fleet. Three risks:

• Probe credentials. A probe with production credentials is a real client. Use read-only or scoped credentials; the probe should not be able to make a production-impacting request.
• Probe rate. A probe that fires every minute is one request per minute per environment — a budget on production but not zero. Calibrate the cadence to a number the team can defend in a quarterly review.
• Probe payload. A {"probe":true} body is not a real checkout; downstream services should treat the probe as a non-billable request. A header like X-Probe-Id should be on the allow-list of the entire fleet.

The probe does not need write access to Loki or Tempo; a read-scoped token is sufficient.

Performance implications

The performance implication is the cost of a single synthetic request per cycle. On the probe side, one HTTP call per cycle, one OTel span, one Tempo span, one Loki log line per service in the chain. On the platform side, one extra ingest line and one extra trace block per cycle. A probe that runs every 15 minutes is roughly 100 traces per day and 100 log lines per service — well within budget for a busy fleet.

A probe that fires every second is a different conversation. The cost shifts from “rounding error” to “measurable ingest”. The 30-minute cadence is the conventional starting point; raise or lower it based on the team’s sensitivity to regressions vs the cost.

Production guidance

• Wire the test into CI so a merge that breaks the chain fails the pipeline. The probe is the production answer to “did we break correlation?”.
• Schedule the test on a recurring cadence from a separate cron / GitHub Actions. A CI-only test misses regressions that occur between merges.
• Page on three consecutive failures. A single failure is often a transient (cold Loki cache, slow Tempo ingestion); three in a row means a real regression.
• Roll the test back by removing the cron entry and the CI step. The platform does not depend on the test (the test observes the platform). The risk of a rolled-back test is the loss of regression detection, not a platform outage.

Verification

You should now be able to answer:

• What four invariants does an end-to-end correlation test assert?
• Why does a unit test on the SDK not catch a broken log handler?
• What is the conventional wall-clock delay between probe request and assertions, and why is it necessary?
• What is the first diagnostic step when the test fails at step 3 (Tempo resolution)?
• Why should the test page on three consecutive failures rather than a single one?

Quiz