Correlation Test

The incident post-mortem says “the trace-to-logs pivot was broken for six hours”. The regression was a single YAML line in the Loki data source provisioning. The fix is one line. The detection was an on-call engineer who clicked the pivot and got no rows. The detection was six hours late.

The correlation test is the discipline that catches the regression before the on-call engineer does. The test fires a synthetic request with a known trace_id, then asserts the trace_id is present at every signal: the Prometheus histogram exemplar, the Loki log line, the Tempo trace. The test runs on a schedule. The test failure pages the team. The test catches the regression in five minutes, not six hours.

What it is

A correlation test is a scheduled end-to-end probe that verifies the join key at every signal. The test has four steps.

1. Generate a known trace_id. A fixed value, generated once and stored in the test config. The value is unique to the test run (or to the time bucket) so the assertion is unambiguous.
2. Fire a request with the trace_id. The request is a synthetic transaction that exercises the application code path. The trace_id is injected via the traceparent header. The OpenTelemetry SDK reads the header and uses the trace_id as the active span context.
3. Wait for the telemetry to land. The test sleeps for the longest of the data source ingest latencies (typically 10 seconds for Loki, 15 seconds for Prometheus, 30 seconds for Tempo).
4. Assert the trace_id is present at every signal. The test queries Prometheus, Loki, and Tempo with the trace_id and asserts each query returns at least one row.

The test is a black-box probe. The test does not know about the labels, the log line shape, or the trace structure. The test only knows the trace_id. The black-box shape is what makes the test durable.

  test runner
     |
     | generate trace_id
     v
  +------------------+
  |  fire request    |
  |  trace_id = X    |
  +--------+---------+
           |
           | wait 30s
           v
  +------------------+
  |  assert          |
  |  Prometheus: X   |
  |  Loki: X         |
  |  Tempo: X        |
  +------------------+

Why a sysadmin cares

The correlation is a contract that crosses the boundary between the application and the platform. The contract is set up once and forgotten. The contract is broken every time a refactor changes the propagator, the log handler, or the data source provisioning. The regression is silent. The on-call engineer discovers the regression during the next incident.

Three operational payoffs.

1. Detection before the incident. The test catches the regression in five minutes. The on-call engineer does not discover the regression at 03:00.
2. Owner of the regression. The test failure pages the team that owns the correlation. The team fixes the regression. The next test run passes.
3. Audit trail. The test results are stored as a time series. The post-mortem has evidence of when the regression was introduced and when it was fixed.

The cost is the discipline of writing the test, scheduling it, and routing the failure. The investment is a single shell script. The return is paid on every regression.

How it works — the four-assertion pattern

The classic four-assertion pattern.

# 1. Generate a known trace_id.
TRACE_ID="0123456789abcdef0123456789abcdef"

# 2. Fire a request with the traceparent header.
curl -s -o /dev/null \
  -H "traceparent: 00-${TRACE_ID}-aaaaaaaaaaaaaaaa-01" \
  http://gateway/api/v1/checkout

# 3. Wait for the telemetry to land.
sleep 30

# 4. Assert the trace_id is present at every signal.
# Prometheus exemplar
curl -s 'http://prometheus:9090/api/v1/query?query=http_server_request_duration_seconds_bucket' \
  | jq -e ".data.result[].exemplar.labels.trace_id | select(. == \"${TRACE_ID}\")"

# Loki log line
logcli query --since=2m \
  "{service=\"checkout\"} | json | trace_id=\"${TRACE_ID}\"" \
  | grep -q "${TRACE_ID}"

# Tempo trace
tempo-cli query "{ trace = \"${TRACE_ID}\" }" \
  | grep -q "${TRACE_ID}"

The four assertions are independent. The test passes only when all four are true. The test fails with a specific message that identifies which signal is missing the trace_id.

How to configure it

The test runner is a shell script. The script is wrapped in a CI pipeline or scheduled as a cron job. The exit code is the result.

#!/usr/bin/env bash
# /usr/local/bin/correlation-test.sh
set -euo pipefail

# 1. Generate a known trace_id.
TRACE_ID="$(uuidgen | tr -d '-' | head -c 32)"
SPAN_ID="aaaaaaaaaaaaaaaa"
HEADER="00-${TRACE_ID}-${SPAN_ID}-01"

# 2. Fire a request with the traceparent header.
curl -fsS -o /dev/null \
  -H "traceparent: ${HEADER}" \
  --max-time 10 \
  http://gateway/api/v1/checkout

# 3. Wait for the telemetry to land.
SLEEP_SECONDS="${SLEEP_SECONDS:-30}"
sleep "${SLEEP_SECONDS}"

# 4. Assert the trace_id is present at every signal.
fail=0

# Prometheus exemplar
if ! curl -fsS \
    "http://prometheus:9090/api/v1/query?query=http_server_request_duration_seconds_bucket" \
    | jq -e ".data.result[].exemplar.labels.trace_id | select(. == \"${TRACE_ID}\")" \
    >/dev/null; then
  echo "Prometheus exemplar missing for trace_id=${TRACE_ID}"
  fail=1
fi

# Loki log line
if ! logcli query --since=2m \
    --addr=http://loki:3100 \
    '{service="checkout"} | json | trace_id="'"${TRACE_ID}"'"' \
    | grep -q "${TRACE_ID}"; then
  echo "Loki log line missing for trace_id=${TRACE_ID}"
  fail=1
fi

# Tempo trace
if ! tempo-cli query --addr=http://tempo:3200 \
    "{ trace = \"${TRACE_ID}\" }" \
    | grep -q "${TRACE_ID}"; then
  echo "Tempo trace missing for trace_id=${TRACE_ID}"
  fail=1
fi

exit "${fail}"

The schedule is a cron entry that runs every five minutes:

*/5 * * * * /usr/local/bin/correlation-test.sh \
  || /usr/local/bin/correlation-test-page.sh

The alerting side — the failure pages the team that owns the correlation. The page is a PagerDuty incident that names the broken signal:

#!/usr/bin/env bash
# /usr/local/bin/correlation-test-page.sh
set -euo pipefail

# Read the failure summary from the test output.
SUMMARY="$(cat /tmp/correlation-test-failure)"

# Send to PagerDuty.
curl -fsS -X POST \
  -H "Authorization: Token token=${PAGERDUTY_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"routing_key\":\"${PAGERDUTY_ROUTING_KEY}\",\"event_action\":\"trigger\",\"payload\":{\"summary\":\"Correlation test failed: ${SUMMARY}\",\"source\":\"correlation-test\",\"severity\":\"warning\"}}" \
  https://events.pagerduty.com/v2/enqueue

The metric side — the test result is exposed as a Prometheus metric for graphing:

# Append to the test runner.
if [ "${fail}" -eq 0 ]; then
  RESULT=1
else
  RESULT=0
fi

# Push to a Pushgateway.
echo "correlation_test_succeeded ${RESULT}" \
  | curl -fsS --data-binary @- \
      http://pushgateway:9091/metrics/job/correlation-test

How to validate it

# 1. The test runner is executable.
chmod +x /usr/local/bin/correlation-test.sh
shellcheck /usr/local/bin/correlation-test.sh

# 2. The test runs end to end.
sudo -u monitoring /usr/local/bin/correlation-test.sh
# (no output on success; nonzero exit on failure)

# 3. The test result is exposed as a metric.
curl -s http://pushgateway:9091/metrics | grep correlation_test
# correlation_test_succeeded 1

# 4. The test is scheduled in cron.
crontab -l | grep correlation-test
# */5 * * * * /usr/local/bin/correlation-test.sh ...

# 5. The test failure pages the team.
# (Manual step) Run the test with a broken configuration and
# confirm the page reaches the on-call rotation.

How it can fail

Six recurring failure shapes.

1. The test does not wait for the telemetry to land. The test fires the request and immediately fires the assertions. The Loki query returns zero rows because the line has not arrived yet. Symptom: the test fails on every run. The failure is consistent.
2. The test uses the wrong trace_id. The test fires a request with a trace_id that the application does not respect. The application generates its own trace_id. The assertion queries for the test trace_id and finds zero rows. Symptom: the test fails on every run. The application trace_id is the one in Tempo, not the test’s.
3. The test does not exercise the right code path. The test fires a request to the health endpoint. The health endpoint does not log the trace_id. The Loki assertion returns zero rows. Symptom: the test fails on every run. The endpoint is the wrong target.
4. The test asserts the wrong metric. The test asserts a metric that does not have an exemplar. The Prometheus assertion returns zero rows. Symptom: the test fails on every run. The metric is the wrong target.
5. The test sleeps too long. The test sleeps for five minutes. The schedule runs every five minutes. The test overlaps itself. The second run fires the request, the first run’s assertions are still pending. The first run’s assertions return zero rows because the second run has not yet completed. Symptom: the test fails under load.
6. The test is not owned. The test fails. The failure pages nobody. The test is silent. Symptom: the test fails on every run. The on-call engineer discovers the regression during the next incident.

How to troubleshoot it

The diagnostic order is “is the test firing the right request?”, “is the test waiting long enough?”, “is every assertion right?”, “is the schedule correct?”, “is the failure being paged?”.

1. Is the test firing the right request? curl -sv -H "traceparent: ..." .... The traceparent header is on the request. The response is 200.
2. Is the test waiting long enough? Increase the sleep to 90 seconds. The Prometheus assertion should pass after 15 seconds. The Loki assertion should pass after 10 seconds. The Tempo assertion should pass after 30 seconds.
3. Is every assertion right? Run each assertion by hand. The Prometheus query should return at least one row. The Loki query should return at least one line. The Tempo query should return the trace. The test passes only when all three are true.
4. Is the schedule correct? crontab -l. The cron entry is at the right cadence. The schedule does not overlap itself.
5. Is the failure being paged? Run the test with a broken configuration. The PagerDuty incident should arrive within one minute.

Security implications

The correlation test fires a synthetic request with a known trace_id. The request uses the same authentication as the production traffic. The test is a real transaction.

The risk is around the synthetic payload. A test that exercises a payment endpoint with a real payment is a production incident in disguise. The mitigation is to exercise a no-op endpoint (for example, a health check with the trace_id header) or to use a test tenant that is isolated from the production data.

The second-order risk is around the trace_id exposure. The test trace_id is a real identifier that ends up in the production logs. The trace_id is opaque and safe to log. The risk is that the test trace_id is reused across runs and the assertions become ambiguous. The mitigation is to generate a fresh trace_id per run.

Performance implications

The correlation test fires one synthetic request per run. The request is a no-op endpoint with the trace_id header. The cost is one row in Prometheus, one log line in Loki, one trace in Tempo. The cost is negligible.

The cost to watch is the assertion cost. The Prometheus query is a single API call. The Loki query is a single LogQL query. The Tempo query is a single TraceQL query. The combined cost is sub-second. The test is fast.

Production guidance

• Generate a fresh trace_id per run. The test trace_id must be unique to the run. The assertion is unambiguous. The trace_id is generated by the test runner, not by the application.
• Wait long enough for the telemetry to land. The longest of the three data sources is Tempo at 30 seconds. The test sleeps for 30 seconds. The test is patient.
• Schedule the test at the right cadence. Every five minutes is the right cadence for a correlation test. The test is fast enough to run frequently. The test catches the regression within five minutes.
• Route the failure to the right team. The team that owns the correlation also owns the test failure. The failure is a P1-equivalent event. The test is the only signal that the correlation is broken.

Verification

You should now be able to answer:

• What is the four-assertion pattern for a correlation test?
• Why does the test wait for the telemetry to land before asserting?
• What is the failure shape of a test that does not exercise the right code path?
• How do you schedule the correlation test at the right cadence?
• How do you route the test failure to the right team?

Quiz