Lab: Correlate Metrics, Logs, and Traces

Objective

By the end of this lab you will have a running four-backend stack in which one request identifier — a single 32-hex trace_id — appears on a Prometheus histogram exemplar, on a Loki log line, and on a Tempo span, and you will have proved that join three times from the command line before opening Grafana at all. You will then break one character of one regex and observe the failure shape that this whole discipline exists to catch: a pivot that is present in the configuration, visible in the API, and silently never fires.

Architecture

Five containers on one Docker network. Nothing talks to the outside world and every published port is bound to loopback.

 exposer (nginx)                     you (curl)
 serves an OpenMetrics                    |
 exposition with an              +--------+--------+
 exemplar on one bucket          |                 |
        |                        v                 v
        | scrape            Loki push API     Tempo OTLP/HTTP
        v                   :3100             :4318
 +-------------+          +-------------+   +-------------+
 | prometheus  |          |    loki     |   |    tempo    |
 | :9090       |          |   :3100     |   |   :3200     |
 | exemplar-   |          |  log line   |   |    span     |
 | storage on  |          |  trace_id   |   |  trace_id   |
 +------+------+          +------+------+   +------+------+
        |                        |                 |
        +------------+-----------+--------+--------+
                     |                    |
                     v                    v
                  +--------------------------+
                  |        grafana :3000     |
                  |  exemplarTraceIdDestin.  |  metric -> trace
                  |  derivedFields           |  log    -> trace
                  |  tracesToLogsV2          |  trace  -> log
                  +--------------------------+

The three Grafana keys in that last box are the entire UI half of correlation. The three backends above them are the data half. This lab builds the data half first, on purpose: a pivot configured against a join key that is not present is the most expensive way to discover that the join key is not present.

Requirements

A disposable Linux host. A nested VM is the honest mode: the lab writes files under your home directory, binds six loopback ports (9090, 3100, 3200, 4318, 3000, 8080), and creates four Docker volumes. Nothing here is safe to run on a host that already serves a monitoring stack on any of them.
Docker Engine 28.x with the Compose v2 plugin. docker compose version must work; docker-compose v1 will not parse this file.
curl, jq and openssl on the host. jq is not optional — every assertion in this lab reads a specific field out of a JSON response, and grep on JSON is how people convince themselves a broken pivot works.
About 1.5 GB of free disk for the images and 1 GB of RAM headroom.
GNU date. The lab generates nanosecond timestamps with date +%s%N, which BSD date on macOS does not support.
No out-of-band access requirement. Nothing in this lab touches the host network configuration, the firewall, or SSH. The only host state it creates is one directory and four named Docker volumes, and Cleanup removes both.

Scenario

Checkout is failing for about one request in three, and only at night. The platform has all three signals. It has a Prometheus histogram for request duration, it has application logs in Loki, and it has traces in Tempo. What it does not have is any way to get from one to the next without a human copying a value between two browser tabs.

You have been asked to make the three signals addressable from each other, and — because the last person who did this left behind a “trace to logs” button that opened an empty page for six months — to prove it works with something other than a click.

The trace you will follow is 0af7651916cd43dd8448eb211c80319c. In production the application generates it; here you pin it, because a test that asserts on a known value is unambiguous and a test that asserts “something arrived” is not.

Tasks

Task 1: Create the working tree and check the ports

WORKDIR="$HOME/rb-obs-correlation"
mkdir -p "$WORKDIR"/{exposition,grafana/provisioning/datasources,secrets}
cd "$WORKDIR"

# Nothing may already own the four ports this lab publishes.
for port in 9090 3100 3200 3000 4318 8080; do
  if ss -ltn "sport = :$port" | grep -q LISTEN; then
    echo "PORT $port IS ALREADY IN USE - stop that service or move this lab"
  fi
done

A silent loop means all six ports are free. If any line prints, resolve it now: a port conflict later in the lab looks like a container that starts, exits, and leaves a health check failing for reasons that have nothing to do with correlation.

Generate the Grafana admin password into a file rather than into the compose file, so it never reaches the process list or a docker inspect output:

openssl rand -base64 24 | tr -d '=/+' > "$WORKDIR/secrets/grafana_admin"
chmod 0600 "$WORKDIR/secrets/grafana_admin"

Task 2: Write the three backend configurations

Every file in this task and the next three lives in $WORKDIR, at the path named in the first line of its block. Create them as you go; the compose file in Task 4 mounts each one by that exact name.

prometheus.yml — a 15-second scrape and, critically, exemplar storage. The storage is not on by default in Prometheus 2.55: it needs both a runtime feature flag (Task 4) and a size in the config file. Set only one of the two and you get a Prometheus that parses exemplars off the wire and then discards them.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

storage:
  exemplars:
    # A circular buffer, not a ledger. Once it is full, the oldest exemplar is
    # overwritten - which is why an exemplar is a pointer to a recent example
    # and never an audit trail.
    max_exemplars: 100000

scrape_configs:
  - job_name: checkout-exposition
    static_configs:
      - targets: ['exposer:80']
        labels:
          env: lab

loki.yaml — the single-binary filesystem configuration. auth_enabled: false means Loki accepts a push with no tenant header, which is right for a lab and wrong for anything shared.

# loki.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  allow_structured_metadata: true
  reject_old_samples: false

analytics:
  reporting_enabled: false

tempo.yaml — a local-storage Tempo with the OTLP receivers switched on. The HTTP receiver on 4318 is the one you will push to by hand in Task 6.

# tempo.yaml
server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  # Short blocks so a trace you push is flushed within the life of the lab.
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

Task 3: Write the exposition that carries the exemplar

This is the metric side of the join, and it is worth doing by hand once. An exemplar is not a label and not a separate metric: it is a suffix on a histogram bucket line, legal only in the OpenMetrics text format.

cat > "$WORKDIR/exposition/checkout-metrics" <<'METRICS'
# HELP http_server_request_duration_seconds Duration of checkout HTTP requests.
# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{service="checkout",le="0.1"} 0
http_server_request_duration_seconds_bucket{service="checkout",le="0.5"} 0
http_server_request_duration_seconds_bucket{service="checkout",le="1.0"} 0
http_server_request_duration_seconds_bucket{service="checkout",le="5.0"} 1 # {trace_id="0af7651916cd43dd8448eb211c80319c"} 4.2
http_server_request_duration_seconds_bucket{service="checkout",le="+Inf"} 1
http_server_request_duration_seconds_sum{service="checkout"} 4.2
http_server_request_duration_seconds_count{service="checkout"} 1
# EOF
METRICS

Read the le="5.0" line again. After the bucket value comes a space, a #, a space, the exemplar label set in braces, and the observed value 4.2. That 4.2 is the actual request duration that produced the sample; the trace with that id is the same request. The trailing # EOF is mandatory — the OpenMetrics parser rejects an exposition without it.

The file deliberately has no extension. nginx maps a Content-Type from the extension, finds nothing, and falls back to default_type:

# nginx.conf
server {
    listen 80;
    server_name _;

    location = /metrics {
        # The whole lab turns on this line. Serve the same bytes as
        # text/plain and Prometheus uses the plain-text parser, which has no
        # concept of an exemplar and drops it without a warning.
        default_type "application/openmetrics-text; version=1.0.0; charset=utf-8";
        alias /srv/checkout-metrics;
    }
}

Why the content type decides whether the exemplar exists

Prometheus advertises the formats it can parse in the scrape request’s Accept header, and then parses the response using whatever the target’s Content-Type says it sent. The exemplar syntax — the # suffix on a sample line — is defined by OpenMetrics and is not part of the older Prometheus text exposition format.

So a target that emits perfectly valid exemplars but labels the response text/plain produces a scrape that succeeds, a metric that appears, a graph that renders, and no exemplars anywhere. There is no error, because from the plain-text parser’s point of view nothing went wrong: it read a bucket value and ignored a comment.

This is the first of the three silent failures in this lab, and it is the one most often misdiagnosed as “Prometheus does not support exemplars”.

Task 4: Write the compose file

# docker-compose.yaml
name: rb-obs-correlation

services:
  exposer:
    image: nginx:1.27-alpine
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - ./exposition/checkout-metrics:/srv/checkout-metrics:ro
    ports:
      # Published only so you can read the Content-Type yourself in Task 6.
      - "127.0.0.1:8080:80"

  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      # Half of the exemplar switch. The other half is storage.exemplars
      # in prometheus.yml. Both are required.
      - --enable-feature=exemplar-storage
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"

  loki:
    image: grafana/loki:3.3.0
    command: -config.file=/etc/loki/loki.yaml
    volumes:
      - ./loki.yaml:/etc/loki/loki.yaml:ro
      - loki-data:/loki
    ports:
      - "127.0.0.1:3100:3100"

  # Tempo runs as uid 10001 and its storage paths do not exist in the image,
  # so Docker creates the volume owned by root and Tempo cannot write to it.
  # This one-shot container fixes the ownership before Tempo starts.
  tempo-init:
    image: grafana/tempo:2.6.0
    user: root
    entrypoint: ["/bin/sh", "-c"]
    command: ["chown -R 10001:10001 /var/tempo"]
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:2.6.0
    command: -config.file=/etc/tempo/tempo.yaml
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "127.0.0.1:3200:3200"
      - "127.0.0.1:4318:4318"

  grafana:
    image: grafana/grafana:11.3.0
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_AUTH_ANONYMOUS_ENABLED: "false"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./secrets/grafana_admin:/run/secrets/grafana_admin:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"

volumes:
  prometheus-data:
  loki-data:
  tempo-data:
  grafana-data:

Task 5: Provision the three pivots

One file declares all three data sources and all three directions of travel. Note the UIDs: prom-lab, loki-lab, tempo-lab. Every cross-reference below is by UID, never by name, because a rename must not break a pivot.

# grafana/provisioning/datasources/observability.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom-lab
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      # metric -> trace. `name` is the exemplar's LABEL name, which must
      # match the label in the exposition exactly: trace_id, not traceID.
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo-lab
          urlDisplayLabel: 'Open trace'

  - name: Loki
    uid: loki-lab
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    jsonData:
      maxLines: 1000
      # log -> trace. The regex runs in the browser against the RAW log
      # line, so it has to match the shape the application actually emits.
      # Ours is compact JSON: {"msg":"...","trace_id":"0af7..."}
      derivedFields:
        - name: trace_id
          matcherRegex: '"trace_id":"([a-f0-9]{32})"'
          datasourceUid: tempo-lab
          # With datasourceUid set, `url` is the QUERY sent to that data
          # source. $$ is the provisioning escape for a literal $.
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open trace in Tempo'

  - name: Tempo
    uid: tempo-lab
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: false
    jsonData:
      httpMethod: GET
      # trace -> log. `tags` maps a span attribute onto a Loki stream label:
      # the span carries service.name, the Loki stream carries service_name.
      tracesToLogsV2:
        datasourceUid: loki-lab
        tags:
          - key: 'service.name'
            value: 'service_name'
        filterByTraceID: true
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'

Task 6: Start the stack and confirm each backend is ready

Configuration changelab host— Pulls five images and starts five containers. No host state outside the working directory.

$ cd ~/rb-obs-correlation && docker compose up -d

cd "$HOME/rb-obs-correlation"

# Every service should be running; tempo-init should have exited 0.
docker compose ps -a

# Readiness, one backend at a time. Retry rather than assume: Loki and Tempo
# take a few seconds to open their HTTP listeners.
for i in $(seq 1 30); do
  curl -sf http://127.0.0.1:9090/-/ready >/dev/null &&
  curl -sf http://127.0.0.1:3100/ready   >/dev/null &&
  curl -sf http://127.0.0.1:3200/ready   >/dev/null &&
  curl -sf http://127.0.0.1:3000/api/health >/dev/null && break
  sleep 2
done
echo "waited $i cycle(s)"

Now confirm the two halves of the exemplar switch, because getting this wrong is the difference between the rest of the lab working and the rest of the lab looking broken:

# Half one: the target really is serving OpenMetrics.
curl -sI http://127.0.0.1:8080/metrics | grep -i '^content-type'

# Half two: the feature flag is live in this process. The command-line
# flags endpoint is the authority - it reports what this process was
# started with, not what the config file asks for.
curl -sf http://127.0.0.1:9090/api/v1/status/flags \
  | jq -r '.data["enable-feature"]'

# And the size, which lives in the config file rather than the flags.
curl -sf http://127.0.0.1:9090/api/v1/status/config \
  | jq -r '.data.yaml' | grep -A2 '^storage:'

The first must report application/openmetrics-text. The second must print exemplar-storage; if it prints an empty string, Prometheus is parsing your exemplars and throwing them away, and no amount of Grafana configuration will make a diamond appear.

Task 7: Push the log line and the span

The metric side arrived by scrape. The other two you push by hand, both carrying the same identifier. Set it once:

TRACE_ID=0af7651916cd43dd8448eb211c80319c
SPAN_ID=b7ad6b7169203331

The Loki push API takes a stream label set and a list of [nanosecond-timestamp, line] pairs. Build the JSON with jq rather than by hand — a log line that is itself JSON inside a JSON document is exactly where hand-escaping goes wrong:

LOG_LINE=$(jq -nc --arg tid "$TRACE_ID" \
  '{level:"error", msg:"payment declined", service:"checkout", trace_id:$tid}')

jq -nc --arg ts "$(date +%s%N)" --arg line "$LOG_LINE" \
  '{streams:[{stream:{service_name:"checkout", level:"error"},
              values:[[$ts, $line]]}]}' \
| curl -sf -X POST http://127.0.0.1:3100/loki/api/v1/push \
    -H 'Content-Type: application/json' --data-binary @- \
    -w 'loki push: HTTP %{http_code}\n'

The span goes to Tempo’s OTLP/HTTP receiver as OTLP JSON. Note that the trace and span ids are hex strings and the timestamps are nanoseconds-as-strings — that is the JSON mapping for OTLP’s bytes and uint64 fields, not a stylistic choice. The 4.2-second duration is the same 4.2 that the exemplar recorded:

END_NS=$(( $(date +%s) * 1000000000 ))
START_NS=$(( END_NS - 4200000000 ))

jq -nc \
  --arg tid "$TRACE_ID" --arg sid "$SPAN_ID" \
  --arg start "$START_NS" --arg end "$END_NS" \
  '{resourceSpans:[{
      resource:{attributes:[{key:"service.name",
                             value:{stringValue:"checkout"}}]},
      scopeSpans:[{
        scope:{name:"runbook-academy-lab"},
        spans:[{
          traceId:$tid, spanId:$sid,
          name:"POST /api/v1/checkout", kind:2,
          startTimeUnixNano:$start, endTimeUnixNano:$end,
          attributes:[{key:"http.response.status_code",
                       value:{intValue:"502"}}],
          status:{code:2, message:"payment declined"}
        }]}]}]}' \
| curl -sf -X POST http://127.0.0.1:4318/v1/traces \
    -H 'Content-Type: application/json' --data-binary @- \
    -w '\ntempo push: HTTP %{http_code}\n'

An OTLP endpoint answers a successful push with {"partialSuccess":{}} and HTTP 200. A 400 here is a malformed payload, not a rejected trace — read the body, it names the field.

Task 8: Prove the join three times, from the command line

This is the task that separates this lab from a screenshot. Nothing below opens a browser.

Metric to trace. Ask Prometheus for the exemplars on the histogram, and pull the trace id out of the response:

NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
AGO=$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)

curl -sfG http://127.0.0.1:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_server_request_duration_seconds_bucket' \
  --data-urlencode "start=$AGO" --data-urlencode "end=$NOW" \
  | jq -r '.data[0].exemplars[0].labels.trace_id'

Trace id to trace. Ask Tempo for that id. Look at the response shape first so the assertion you write next is about something real:

curl -sf -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/$TRACE_ID" | jq 'keys'

curl -sf -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/$TRACE_ID" \
  | jq '[.. | .spans? // empty | .[]] | length'

A trace pushed seconds ago is still in the ingester rather than in a flushed block; lookup by id reads both, so it resolves immediately. A 404 here after a successful push means the id you queried is not the id you pushed — compare them character by character before suspecting Tempo.

Trace id to logs. Ask Loki for the same id. This is the query Grafana’s trace-to-logs button builds for you:

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode "query={service_name=\"checkout\"} |= \"$TRACE_ID\"" \
  --data-urlencode 'since=15m' \
  | jq -r '.data.result[0].values[0][1]'

Three commands, three signals, one identifier. That is the whole contract. Any Grafana button you configure from here is a convenience on top of a join that you have already shown exists.

Task 9: Break the derived field and read the symptom

Now inject the regression that this lab exists to teach. Change the Loki derived field from the JSON shape to the logfmt shape, leaving the rule otherwise perfect:

cd "$HOME/rb-obs-correlation"
cp grafana/provisioning/datasources/observability.yaml \
   grafana/provisioning/datasources/observability.yaml.good

sed -i "s|matcherRegex: '\"trace_id\":\"(\[a-f0-9\]{32})\"'|matcherRegex: 'trace_id=([a-f0-9]{32})'|" \
  grafana/provisioning/datasources/observability.yaml

grep -n 'matcherRegex' grafana/provisioning/datasources/observability.yaml

Configuration changelab host— Reloads datasource provisioning without restarting Grafana

$ GF_PW=$(cat ~/rb-obs-correlation/secrets/grafana_admin); curl -sf -X POST -u admin:$GF_PW http://127.0.0.1:3000/api/admin/provisioning/datasources/reload

Now look at what an operator would look at. The data source is healthy:

GF_PW=$(cat "$HOME/rb-obs-correlation/secrets/grafana_admin")

curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/datasources/uid/loki-lab/health | jq -r '.status'

curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/datasources/uid/loki-lab \
  | jq '.jsonData.derivedFields'

Health is fine. The rule is present. Its name is right, its target data source is right, its display label is right. The query still returns the log line. Every check an operator would think to run passes — and the “Open trace in Tempo” link never appears, because the regex matches nothing in the line.

The only honest test is to run the regex against a real line:

LINE=$(curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'since=15m' \
  | jq -r '.data.result[0].values[0][1]')

echo "$LINE"
echo "$LINE" | grep -Eo 'trace_id=[a-f0-9]{32}'   || echo "logfmt regex: NO MATCH"
echo "$LINE" | grep -Eo '"trace_id":"[a-f0-9]{32}"' || echo "json regex: NO MATCH"

Restore the working file and reload:

cd "$HOME/rb-obs-correlation"
mv grafana/provisioning/datasources/observability.yaml.good \
   grafana/provisioning/datasources/observability.yaml

GF_PW=$(cat secrets/grafana_admin)
curl -sf -X POST -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/admin/provisioning/datasources/reload

Validation

Everything above becomes one script with an exit code. This is the deliverable — the thing you would schedule, not the thing you would remember to click.

cat > "$HOME/rb-obs-correlation/correlation-test.sh" <<'TEST'
#!/usr/bin/env bash
# Asserts the join key at all three signals. Non-zero exit = broken pivot.
set -euo pipefail

TRACE_ID="${1:?usage: correlation-test.sh TRACE_ID}"
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
AGO=$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
fail() { echo "FAIL: $1"; exit 1; }

curl -sfG http://127.0.0.1:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_server_request_duration_seconds_bucket' \
  --data-urlencode "start=$AGO" --data-urlencode "end=$NOW" \
  | jq -e --arg t "$TRACE_ID" \
      '[.data[].exemplars[].labels.trace_id] | index($t)' >/dev/null \
  || fail "no Prometheus exemplar carries $TRACE_ID"

curl -sf -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/$TRACE_ID" \
  | jq -e '[.. | .spans? // empty | .[]] | length > 0' >/dev/null \
  || fail "Tempo has no spans for $TRACE_ID"

curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode "query={service_name=\"checkout\"} |= \"$TRACE_ID\"" \
  --data-urlencode 'since=30m' \
  | jq -e '.data.result | length > 0' >/dev/null \
  || fail "Loki has no line carrying $TRACE_ID"

GF_PW=$(cat "$HOME/rb-obs-correlation/secrets/grafana_admin")
LINE=$(curl -sfG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service_name="checkout"}' \
  --data-urlencode 'since=30m' | jq -r '.data.result[0].values[0][1]')
RE=$(curl -sf -u "admin:$GF_PW" \
  http://127.0.0.1:3000/api/datasources/uid/loki-lab \
  | jq -r '.jsonData.derivedFields[0].matcherRegex')
echo "$LINE" | grep -Eq "$RE" \
  || fail "the derived field regex does not match a real log line"

echo "PASS: $TRACE_ID present on all three signals, derived field matches"
TEST

chmod +x "$HOME/rb-obs-correlation/correlation-test.sh"
"$HOME/rb-obs-correlation/correlation-test.sh" 0af7651916cd43dd8448eb211c80319c

The fourth assertion is the one worth arguing about. The first three prove the data-layer join; only the fourth catches the regression you injected in Task 9, because only the fourth compares the configured regex against a line that actually exists.

Finally, confirm the UI half is provisioned as declared:

GF_PW=$(cat "$HOME/rb-obs-correlation/secrets/grafana_admin")
for uid in prom-lab loki-lab tempo-lab; do
  printf '%s health: ' "$uid"
  curl -sf -u "admin:$GF_PW" \
    "http://127.0.0.1:3000/api/datasources/uid/$uid/health" | jq -r '.status'
done

curl -sf -u "admin:$GF_PW" http://127.0.0.1:3000/api/datasources/uid/prom-lab \
  | jq '.jsonData.exemplarTraceIdDestinations'
curl -sf -u "admin:$GF_PW" http://127.0.0.1:3000/api/datasources/uid/tempo-lab \
  | jq '.jsonData.tracesToLogsV2'

Then, optionally, look at it: browse to http://127.0.0.1:3000, log in as admin with the password in secrets/grafana_admin, open Explore against Prometheus, query http_server_request_duration_seconds_bucket, and enable exemplars on the query. The diamond on the 5-second bucket is the pivot you proved by API in Task 8.

Expected Outcome

Five containers running, tempo-init exited 0, no restart loops.
/api/v1/status/runtimeinfo lists exemplar-storage, and the exposer serves application/openmetrics-text.
correlation-test.sh 0af7651916cd43dd8448eb211c80319c prints PASS and exits 0.
All three data sources report a healthy status, and the three pivot keys read back from the Grafana API exactly as written in provisioning.
You can describe, without looking it up, what a broken derived-field regex looks like from the operator’s chair.

Troubleshooting

query_exemplars returns an empty data array. Work the two halves of the switch in order. Check the feature flag in /api/v1/status/runtimeinfo first — it is a per-process fact and the cheapest thing to read. If the flag is present, check the target’s Content-Type; if that is text/plain, nginx matched an extension you did not intend. Only then suspect the exposition itself, and validate it by eye against the sample in Task 3: a missing # EOF makes the whole scrape fail rather than just the exemplar, so if the metric is present the terminator is present too.

Tempo restarts in a loop and logs a permission error on /var/tempo. The tempo-init container did not run, or ran after Tempo. Confirm with docker compose ps -a that it shows Exited (0), and that tempo has the depends_on condition. Re-running docker compose up -d will not re-run a completed init container; use docker compose up -d --force-recreate tempo-init.

The Loki push returns HTTP 400. Two usual causes. A timestamp in seconds rather than nanoseconds — Loki reads the value as a moment in 1970 and rejects it as too old. Or a stream label containing a character outside [a-zA-Z_][a-zA-Z0-9_]*; service.name is a legal OpenTelemetry attribute and an illegal Loki label, which is exactly why the tracesToLogsV2 block has to map one onto the other.

The Tempo push returns HTTP 200 but the trace never appears. Check the trace id length. OTLP/JSON wants 32 hex characters for a trace id and 16 for a span id; a 31-character id is accepted by the receiver and stored under a different key than the one you query. This is the truncation failure mode from the correlation-anatomy lesson, and its symptom — push succeeded, query returns nothing — is why “the trace is missing” is so often the wrong conclusion.

Grafana starts but rejects your password. The admin password is read from the file at first boot only. If you regenerated secrets/grafana_admin after Grafana had already initialised its database, the volume still holds the old hash. Either use the old value or destroy grafana-data and start again.

A container exits immediately with a config error. Read it, do not restart it: docker compose logs --no-log-prefix tempo (or loki, or prometheus). All three parse their configuration at startup and name the offending key and line. A restart loop with no log output is almost always a bind-mount path that does not exist on the host, which makes Docker create a directory where you meant to mount a file.

Cleanup

cd "$HOME/rb-obs-correlation"
docker compose config --volumes     # confirm the four names, then:
docker compose down -v

# Confirm nothing is left behind.
docker compose ps -a
docker volume ls | grep rb-obs-correlation || echo "no lab volumes remain"

# The working tree, including the generated admin password.
cd "$HOME"
rm -rf "$HOME/rb-obs-correlation"

The images stay in the local cache. Remove them with docker image rm grafana/tempo:2.6.0 grafana/loki:3.3.0 prom/prometheus:v2.55.1 grafana/grafana:11.3.0 nginx:1.27-alpine if you want the disk back.

What You Learned

The join key is data, and the pivot is UI. You built the data half first and proved it with three API calls. Every Grafana key you then configured was a shortcut over a join that already existed — which is the only order in which a “broken pivot” is diagnosable.
An exemplar is a format, not a feature. It rides on the OpenMetrics content type and needs both --enable-feature=exemplar-storage and storage.exemplars.max_exemplars. Miss any one of the three and the metric still renders, which is why this failure is so consistently misread.
A configured pivot is not a working pivot. In Task 9 the health check passed, the API returned the rule, and the link never appeared. The only assertion that caught it ran the configured regex against a real log line. Presence is not behaviour.
The three-signal contract is testable in twenty lines of shell. The correlation test is black-box: it knows a trace id and nothing else, so it survives refactors of the label set, the log format and the panel layout, and fails exactly when the contract does.
Static exposition is a lab convenience with a real edge. Because the file never changes, Prometheus stores the same trace id on every scrape. A real application emits a different one per request, so its exemplar buffer is a rolling window of recent examples — never a history you can search.

Taking this to a real change window

Wiring correlation into a live Grafana is a data source provisioning change: low blast radius, instant rollback, no restart needed thanks to the reload endpoint. Treat it as a normal change with two additions.

First, apply it to a disposable Grafana with the production provisioning directory before the window, exactly as this lab does — the regex is the part that breaks, and it breaks against real log lines, not against staging ones.

Second, land the correlation test in the same change, scheduled, alerting to the team that owns the pivot. Without it your detection mechanism for a broken join is an on-call engineer clicking a button during an incident, which is both the worst time to find out and the least likely moment for anyone to file the bug afterwards.

Objectives

Prerequisites