Lab: Instrument and Sample Traces

Objective

By the end of this lab, one HTTP request to a checkout service will produce a single trace containing spans from two processes, joined by a header neither application knows exists. You will get there without editing the application’s request handlers — the agent is in the image and the entry point turns it on.

Then you will do the three things that make instrumentation an operational subject rather than a setup step: break the propagation boundary and count the traces it splits, put a credential in a span attribute and get it back out, and misconfigure a sampler so that most traces lose their downstream half while every dashboard stays green.

Architecture

Three containers and one HTTP call between two of them.

  curl -> 127.0.0.1:8080
             |
   +---------v-----------+    HTTP + traceparent    +---------------------+
   |  checkout-api       | -----------------------> |  payment-svc        |
   |  flask :8080        |                          |  flask :8080        |
   |  opentelemetry-     |                          |  opentelemetry-     |
   |  instrument python  |                          |  instrument python  |
   +---------+-----------+                          +----------+----------+
             |  OTLP/gRPC 4317                                 |
             +------------------+------------------------------+
                                v
                     +---------------------+
                     |  tempo (monolithic) |
                     |  :3200  :4317       |
                     +---------------------+

  What each stage changes:

    stage 0   command = python app.py                 -> no spans at all
    stage 1   command = opentelemetry-instrument ...  -> 4 spans, 1 trace
    stage 2   OTEL_PROPAGATORS=none on checkout       -> 2 traces, split
    stage 3   a manual span inside the handler        -> 5 spans
    stage 4   a redaction hook in before_request      -> same spans, no secret
    stage 5   OTEL_TRACES_SAMPLER on each service     -> traces missing halves

Every stage after the first is a single environment variable or a handful of lines, applied one at a time. That is deliberate: when a trace goes wrong in production, the useful question is “which one thing changed”, and this lab is built so you can always answer it.

Requirements

A disposable Linux host with Docker Engine, the Compose v2 plugin, and outbound HTTPS to PyPI and Docker Hub. Two images are built locally.
About 3 GB of free disk and 2 GB of RAM.
Ports 3200, 4317 and 8080 free on loopback. Task 1 checks.
curl and jq.
No out-of-band access requirement. Nothing here touches SSH, the host firewall or the primary interface, and every published port is bound to 127.0.0.1.

Scenario

Two services, both in Python, both three years old. The team has a metrics dashboard that shows checkout latency rising and payment latency flat, which is arithmetically impossible if checkout’s time is spent waiting on payment — so either the dashboard is wrong or the two services disagree about what they are measuring. Nobody can tell which, because there are no traces.

The ask is “add tracing”, and the version of that request that succeeds has four parts: get spans out of both services without a code change anyone has to review; make sure the two halves join into one trace; make sure the spans do not carry anything that should not leave the process; and choose a sampling rate before the first production rollout rather than after the trace bill arrives.

Tasks

Task 1: Capture the starting state

LAB="$HOME/otel-instrumentation-lab"
mkdir -p "$LAB/checkout" "$LAB/payment"
cd "$LAB"

ss -ltn 2>/dev/null | tee ports.pre-lab
grep -E ':(3200|4317|8080)' ports.pre-lab || echo "all three lab ports are free"

docker image ls --format '{{.Repository}}:{{.Tag}}' | sort | tee images.pre-lab
docker compose version

Task 2: Write the two services and the stack

Two Flask applications, each about twenty lines, with no observability code in them at all. That absence is the starting state, and it is what Task 4 changes without either file being touched. checkout calls payment over HTTP and returns whatever it got back; payment authorises, fails on demand through FAIL_RATE, and echoes the traceparent header it received so that Task 5 can watch the propagation boundary without a packet capture.

cd "$HOME/otel-instrumentation-lab"

cat > checkout/app.py <<'PY'
import os

import requests
from flask import Flask, jsonify, request

PAYMENT_URL = os.environ.get("PAYMENT_URL", "http://payment:8080/authorize")

app = Flask(__name__)


@app.get("/healthz")
def healthz():
    return jsonify(status="ok")


@app.get("/checkout")
def checkout():
    amount = request.args.get("amount", "10.00")
    reply = requests.post(PAYMENT_URL, json={"amount": amount}, timeout=5)
    return jsonify(checkout="done", payment=reply.json()), reply.status_code


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
PY

cat > payment/app.py <<'PY'
import os
import random

from flask import Flask, jsonify, request

FAIL_RATE = float(os.environ.get("FAIL_RATE", "0"))

app = Flask(__name__)


@app.get("/healthz")
def healthz():
    return jsonify(status="ok")


@app.post("/authorize")
def authorize():
    body = request.get_json(silent=True) or {}
    if random.random() < FAIL_RATE:
        return jsonify(error="card declined"), 502
    # Echoing the inbound header is a lab affordance, not a pattern to copy.
    return jsonify(authorized=True, amount=body.get("amount"),
                   traceparent=request.headers.get("traceparent")), 200


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
PY

One Dockerfile, used by both. Read the two RUN lines carefully: the image carries the agent and the instrumentation packages, and the CMD does not use them. Shipping the agent and activating it are separate decisions, and keeping them separate is what lets you roll instrumentation forward and back without rebuilding.

# checkout/Dockerfile and payment/Dockerfile — identical
FROM python:3.12-slim
WORKDIR /app

RUN pip install --no-cache-dir \
      "flask>=3.0,<4.0" "requests>=2.31,<3.0" \
      opentelemetry-distro opentelemetry-exporter-otlp

# Inspects the packages already installed and pulls the matching
# instrumentation libraries — here, flask and requests. Without this the
# launcher starts, finds nothing to patch, and produces no spans.
RUN opentelemetry-bootstrap -a install

COPY app.py .
EXPOSE 8080
CMD ["python", "app.py"]

Write the Dockerfile into both service directories:

cd "$HOME/otel-instrumentation-lab"

cat > checkout/Dockerfile <<'DOCKER'
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir \
      "flask>=3.0,<4.0" "requests>=2.31,<3.0" \
      opentelemetry-distro opentelemetry-exporter-otlp
RUN opentelemetry-bootstrap -a install
COPY app.py .
EXPOSE 8080
CMD ["python", "app.py"]
DOCKER
cp checkout/Dockerfile payment/Dockerfile

The stack. Every knob the later tasks turn is a variable with a default, so a stage change is one line in .env and a restart:

cd "$HOME/otel-instrumentation-lab"

cat > .env <<'ENVEOF'
TEMPO_TAG=2.6.1
APP_CMD=python app.py
CHECKOUT_PROPAGATORS=tracecontext,baggage
CHECKOUT_SAMPLER=parentbased_always_on
CHECKOUT_SAMPLER_ARG=1.0
PAYMENT_SAMPLER=parentbased_always_on
PAYMENT_SAMPLER_ARG=1.0
PAYMENT_FAIL_RATE=0
ENVEOF

cat > tempo.yaml <<'YAML'
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 1m

compactor:
  compaction:
    block_retention: 24h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal
YAML

cat > docker-compose.yaml <<'YAML'
services:
  tempo-init:
    image: grafana/tempo:${TEMPO_TAG}
    user: root
    entrypoint: ["chown", "-R", "10001:10001", "/var/tempo"]
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:${TEMPO_TAG}
    command: ["-config.file=/etc/tempo/tempo.yaml"]
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "127.0.0.1:3200:3200"
      - "127.0.0.1:4317:4317"
    restart: unless-stopped

  payment:
    build: ./payment
    command: ${APP_CMD:-python app.py}
    environment:
      FAIL_RATE: ${PAYMENT_FAIL_RATE:-0}
      OTEL_SERVICE_NAME: payment-svc
      OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4317
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_TRACES_EXPORTER: otlp
      OTEL_METRICS_EXPORTER: none
      OTEL_LOGS_EXPORTER: none
      OTEL_RESOURCE_ATTRIBUTES: deployment.environment=lab,service.version=1.0.0
      OTEL_TRACES_SAMPLER: ${PAYMENT_SAMPLER:-parentbased_always_on}
      OTEL_TRACES_SAMPLER_ARG: ${PAYMENT_SAMPLER_ARG:-1.0}
      OTEL_BSP_SCHEDULE_DELAY: "1000"
    restart: unless-stopped

  checkout:
    build: ./checkout
    command: ${APP_CMD:-python app.py}
    depends_on:
      - payment
    environment:
      PAYMENT_URL: http://payment:8080/authorize
      OTEL_SERVICE_NAME: checkout-api
      OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4317
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_TRACES_EXPORTER: otlp
      OTEL_METRICS_EXPORTER: none
      OTEL_LOGS_EXPORTER: none
      OTEL_RESOURCE_ATTRIBUTES: deployment.environment=lab,service.version=1.0.0
      OTEL_PROPAGATORS: ${CHECKOUT_PROPAGATORS:-tracecontext,baggage}
      OTEL_TRACES_SAMPLER: ${CHECKOUT_SAMPLER:-parentbased_always_on}
      OTEL_TRACES_SAMPLER_ARG: ${CHECKOUT_SAMPLER_ARG:-1.0}
      OTEL_BSP_SCHEDULE_DELAY: "1000"
    ports:
      - "127.0.0.1:8080:8080"
    restart: unless-stopped

volumes:
  tempo-data:
YAML

docker compose config >/dev/null && echo "compose file parses"

OTEL_BSP_SCHEDULE_DELAY is the batch processor’s flush interval in milliseconds. The default is five seconds, which is correct in production and tedious in a lab where you want to see a span within a request or two of sending it. It is also worth knowing the knob exists: when an application exits before the batch drains, this is the number that decides how much you lose.

Task 3: Establish the starting state — no spans at all

Service impact possiblelab host— Builds both images (a few minutes on first run) and starts the stack

$ docker compose up -d --build

for i in $(seq 1 30); do curl -fsS http://127.0.0.1:3200/ready && break; sleep 2; done
curl -fsS http://127.0.0.1:8080/healthz; echo
curl -fsS 'http://127.0.0.1:8080/checkout?amount=10.00'; echo

Set up the query helpers, drive some traffic, and confirm the trace store has nothing for either service:

export TEMPO="http://127.0.0.1:3200"

tq() {
  curl -fsSG "$TEMPO/api/search" \
    --data-urlencode "q=$1" \
    --data-urlencode "start=$(date -d '-20 min' +%s)" \
    --data-urlencode "end=$(date +%s)" \
    --data-urlencode "limit=${2:-200}"
}
n() { tq "$1" "${2:-200}" | jq '.traces | length'; }

# $1 requests, $2 the amount — which doubles as a marker, because the manual
# span in Task 6 records it and that makes one run's traces findable.
hit() {
  for i in $(seq 1 "${1:-20}"); do
    curl -sf -o /dev/null "http://127.0.0.1:8080/checkout?amount=${2:-10.00}" || true
  done
}

# Every stage below recreates a container. Wait for it rather than racing it.
wait_up() {
  for i in $(seq 1 30); do
    curl -fsS -o /dev/null http://127.0.0.1:8080/healthz && return 0
    sleep 2
  done
  echo "checkout did not come back up"; return 1
}

hit 20
sleep 90

n '{}'                                       # expect 0
n '{ resource.service.name = "checkout-api" }'   # expect 0

Zero is the correct answer and it is worth recording. Half the “we added tracing and it did not work” reports are really “we never established what zero looked like”, so the first non-zero number has nothing to be compared against.

Task 4: Turn on instrumentation without touching the code

One line in .env, and the entry point changes from the interpreter to the launcher:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^APP_CMD=.*|APP_CMD=opentelemetry-instrument python app.py|' .env
docker compose up -d
wait_up
docker compose logs --tail 20 checkout

hit 10
sleep 90
n '{ resource.service.name = "checkout-api" }'
n '{ resource.service.name = "payment-svc" }'

Both counts should be non-zero and equal. Now the question that matters — did they join into one trace, or are they two traces that merely happened at the same time?

# Traces where a checkout span has a payment span somewhere beneath it.
n '{ resource.service.name = "checkout-api" } >> { resource.service.name = "payment-svc" }'

# Traces that entered through payment. If propagation works, this is 0:
# payment never starts a trace of its own.
n '{ trace:rootService = "payment-svc" }'

The first count equals the number of requests you sent; the second is zero. Pull one trace apart to see what the agent produced:

TID="$(tq '{ trace:rootService = "checkout-api" }' 1 | jq -r '.traces[0].traceID')"
curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | jq '[.. | objects | select(has("name") and has("spanId")) | {name, kind}]'

Confirm the mechanism as well as the result — the payment service echoes back the header it was given:

curl -sf 'http://127.0.0.1:8080/checkout?amount=10.00' | jq '.payment.traceparent'

Four spans from one curl: the Flask server span on checkout, the outgoing requests client span on checkout, the Flask server span on payment, and — depending on your agent version — an internal span or two. None of them exist in the application source. The agent patched Flask’s WSGI entry and the requests session at import time, and everything else followed.

What the launcher actually does before your code runs

opentelemetry-instrument is not a wrapper that watches your process. It runs first, configures the SDK from the OTEL_* environment, installs a sitecustomize hook, and then execs your program. When Python imports flask, the hook patches it; when it imports requests, the hook patches that too.

Two consequences fall out of that ordering. First, the entry point is the configuration: a container that starts python app.py because somebody “simplified the Dockerfile” produces no telemetry at all, with no error, and the image still contains the agent. Second, trace.get_tracer(...) called at module import time in your own code is safe, because the API hands back a proxy that resolves against the global provider on first use — which is why Task 6’s manual span works without any bootstrap code of its own.

The failure this ordering causes is the reverse case: an application that calls TracerProvider(...) itself and runs under the launcher ends up with two providers, and the spans go to whichever one won the global registration. Pick one mechanism per process.

Task 5: Break the boundary and count what it costs

The trace joined because the requests client injected a traceparent header and the Flask server on the other side extracted it. Turn the propagator off on the calling side only — the SDK still runs, the spans are still produced, and only the header stops:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^CHECKOUT_PROPAGATORS=.*|CHECKOUT_PROPAGATORS=none|' .env
docker compose up -d checkout
wait_up

hit 10
sleep 90

n '{ resource.service.name = "checkout-api" } >> { resource.service.name = "payment-svc" }'
n '{ trace:rootService = "payment-svc" }'

The joined count stops growing and the payment-rooted count starts. Every request now produces two traces: one that begins and ends at checkout, and an orphan that begins at payment with no parent and no idea it was called by anyone.

What makes this failure expensive is what it does not break. Both services still emit spans. Both per-service latency panels are correct. Both error rates are correct. The only thing that is gone is the relationship — and the relationship is the entire reason the trace store exists.

Look at the header itself. The payment service echoes back whatever traceparent it received, so one request shows you the boundary directly:

curl -sf 'http://127.0.0.1:8080/checkout?amount=10.00' | jq '.payment.traceparent'

With the propagator off this is null: the client sent no header, so the server had nothing to extract and opened a root span instead. Turn it back on and the same command returns a four-field hex string: the version, a 32-character trace ID, the caller’s 16-character span ID, and the two-character flag whose low bit is the sampled decision.

Put it back:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^CHECKOUT_PROPAGATORS=.*|CHECKOUT_PROPAGATORS=tracecontext,baggage|' .env
docker compose up -d checkout
wait_up
hit 10
sleep 90
n '{ trace:rootService = "payment-svc" }'   # stops growing again

In production the variable is rarely the cause. The cause is a proxy, an API gateway or a CDN that drops headers it does not recognise, and the symptom is identical to what you just produced. That is why the propagation audit is a single request inspected at every hop, and why it belongs in the runbook for any change to a middlebox on the path.

Task 6: Add a domain span the agent cannot invent

The agent knows about protocols. It does not know that authorising a charge is a thing your business does, or that the amount matters. That is what a manual span is for, and it is the only part of this lab that edits application code.

# checkout/app.py — additions
from opentelemetry import trace

tracer = trace.get_tracer("checkout-api.domain")


@app.get("/checkout")
def checkout():
    amount = request.args.get("amount", "10.00")
    with tracer.start_as_current_span("charge.authorise") as span:
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.method", "card")
        try:
            reply = requests.post(PAYMENT_URL, json={"amount": amount}, timeout=5)
            reply.raise_for_status()
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
            return jsonify(checkout="failed"), 502
        return jsonify(checkout="done", payment=reply.json()), reply.status_code

The with block is not stylistic. If the handler raises before span.end() runs, the SDK never exports the span, and the trace loses exactly the operation that failed — which is the one you needed. A context manager ends the span on every exit path including the exceptional one.

cd "$HOME/otel-instrumentation-lab"

cat > checkout/app.py <<'PY'
import os

import requests
from flask import Flask, jsonify, request
from opentelemetry import trace

PAYMENT_URL = os.environ.get("PAYMENT_URL", "http://payment:8080/authorize")

app = Flask(__name__)
tracer = trace.get_tracer("checkout-api.domain")


@app.get("/healthz")
def healthz():
    return jsonify(status="ok")


@app.get("/checkout")
def checkout():
    amount = request.args.get("amount", "10.00")
    with tracer.start_as_current_span("charge.authorise") as span:
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.method", "card")
        try:
            reply = requests.post(PAYMENT_URL, json={"amount": amount}, timeout=5)
            reply.raise_for_status()
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
            return jsonify(checkout="failed"), 502
        return jsonify(checkout="done", payment=reply.json()), reply.status_code


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
PY

docker compose up -d --build checkout
wait_up
hit 10

# Now make every authorisation fail, so the exception path runs.
sed -i 's|^PAYMENT_FAIL_RATE=.*|PAYMENT_FAIL_RATE=1|' .env
docker compose up -d payment
sleep 8            # payment is not published, so give it a beat to bind
hit 10
sed -i 's|^PAYMENT_FAIL_RATE=.*|PAYMENT_FAIL_RATE=0|' .env
docker compose up -d payment
sleep 8

sleep 90
n '{ span:name = "charge.authorise" }'
n '{ span:name = "charge.authorise" && span:status = error }'
n '{ span.payment.method = "card" }'

The first count covers every request since the rebuild; the second covers only the failing window. Read one of the failures in full — the exception is attached to the span as an event, not as an attribute:

TID="$(tq '{ span:name = "charge.authorise" && span:status = error }' 1 \
  | jq -r '.traces[0].traceID')"
curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | jq '[.. | objects | select(has("name") and has("spanId"))
         | {name, status: (.status.code // 0),
            events: [(.events // [])[] | .name]}]'

An event named exception on the charge.authorise span, carrying the type, the message and the stack trace, is what record_exception produced. That is where the exception message lives — which matters for the next task, because an exception message is one of the two places a credential most often ends up in a trace.

Task 7: Leak a credential, then get it back out

Send something that looks like a secret in the query string. This is not a contrived accident; it is what a redirect URL, a signed download link or a legacy API key parameter looks like in real traffic:

The amount is a marker: payment.amount is on the manual span from Task 6, so it identifies exactly this request instead of whichever trace the search happens to return first.

curl -sf -o /dev/null 'http://127.0.0.1:8080/checkout?amount=99.99&token=hunter2'
sleep 90

TID="$(tq '{ span.payment.amount = "99.99" }' 1 | jq -r '.traces[0].traceID')"

# Which attributes does this agent actually set? Read them; do not guess.
curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | jq -r '[.. | objects | select(has("key") and has("value"))
            | "\(.key) = \(.value | to_entries[0].value)"] | .[]' \
  | sort -u

Look for the URL attribute — depending on the agent version and the semantic convention it follows, it is http.url, url.full, or both — and the token is in it, in full, in a store with a retention measured in weeks and a search box that anyone with a Grafana login can use.

curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | grep -c hunter2 || echo "not present"

Now redact it. The Flask instrumentation sets its URL attributes when the server span starts, which is before any before_request handler runs, so a handler can overwrite them:

# checkout/app.py — additions
@app.before_request
def redact_url_attributes():
    span = trace.get_current_span()
    if not span.is_recording():
        return
    # Set both spellings. Whichever one this agent version uses gets
    # corrected; the other is simply added, and is accurate either way.
    span.set_attribute("http.url", request.base_url)
    span.set_attribute("url.full", request.base_url)
    span.set_attribute("http.target", request.path)
    span.set_attribute("url.query", "[redacted]")

cd "$HOME/otel-instrumentation-lab"

cat > checkout/app.py <<'PY'
import os

import requests
from flask import Flask, jsonify, request
from opentelemetry import trace

PAYMENT_URL = os.environ.get("PAYMENT_URL", "http://payment:8080/authorize")

app = Flask(__name__)
tracer = trace.get_tracer("checkout-api.domain")


@app.before_request
def redact_url_attributes():
    span = trace.get_current_span()
    if not span.is_recording():
        return
    span.set_attribute("http.url", request.base_url)
    span.set_attribute("url.full", request.base_url)
    span.set_attribute("http.target", request.path)
    span.set_attribute("url.query", "[redacted]")


@app.get("/healthz")
def healthz():
    return jsonify(status="ok")


@app.get("/checkout")
def checkout():
    amount = request.args.get("amount", "10.00")
    with tracer.start_as_current_span("charge.authorise") as span:
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.method", "card")
        try:
            reply = requests.post(PAYMENT_URL, json={"amount": amount}, timeout=5)
            reply.raise_for_status()
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
            return jsonify(checkout="failed"), 502
        return jsonify(checkout="done", payment=reply.json()), reply.status_code


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
PY

docker compose up -d --build checkout
wait_up
curl -sf -o /dev/null 'http://127.0.0.1:8080/checkout?amount=88.88&token=hunter2'
sleep 90

TID="$(tq '{ span.payment.amount = "88.88" }' 1 | jq -r '.traces[0].traceID')"
curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | grep -c hunter2 || echo "redacted: token not present in the trace"

The deeper answer is that the value should not have been in the URL. Redaction at the telemetry boundary is a backstop for the traffic you do not control, not a licence for the traffic you do — and every backstop has a version where the attribute name changed underneath it.

Task 8: Choose a sampler, then choose the wrong one

Sampling is the difference between a trace bill you planned and one you discover. Start with a head sample at ten percent on the entry service:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^CHECKOUT_SAMPLER=.*|CHECKOUT_SAMPLER=parentbased_traceidratio|' .env
sed -i 's|^CHECKOUT_SAMPLER_ARG=.*|CHECKOUT_SAMPLER_ARG=0.1|' .env
docker compose up -d checkout
wait_up

hit 100 71.01
sleep 90
n '{ span.payment.amount = "71.01" }'

The marker is doing real work here. Counting all traces before and after would force you to trust that nothing else wrote to Tempo in the meantime and that the result limit did not truncate the answer. Counting the traces that carry payment.amount = "71.01" counts exactly this run — and it only works because Task 6 put a domain attribute on the span, which is the argument for domain attributes in one line.

Expect roughly ten, and expect it to vary: the sampler decides per trace ID, so a hundred requests is a hundred coin flips at p=0.1. Anything from about four to sixteen is an ordinary result, and treating a single run as a measurement is how teams conclude their sampler is broken. Run it three times before believing any number.

Now the misconfiguration. Put the entry service back to sampling everything and give the downstream service a bare ratio sampler — the one that makes its own decision instead of honouring the caller’s:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^CHECKOUT_SAMPLER=.*|CHECKOUT_SAMPLER=parentbased_always_on|' .env
sed -i 's|^CHECKOUT_SAMPLER_ARG=.*|CHECKOUT_SAMPLER_ARG=1.0|' .env
sed -i 's|^PAYMENT_SAMPLER=.*|PAYMENT_SAMPLER=traceidratio|' .env
sed -i 's|^PAYMENT_SAMPLER_ARG=.*|PAYMENT_SAMPLER_ARG=0.1|' .env
docker compose up -d checkout payment
wait_up

hit 60 72.02
sleep 90

# Every request from this run that produced a trace at all.
n '{ span.payment.amount = "72.02" }'

# Of those, the ones that also contain a payment-svc span.
n '{ span.payment.amount = "72.02" } && { resource.service.name = "payment-svc" }'

Sixty traces, roughly six of them complete. The other fifty-four have a checkout server span, a client span that says it called payment, and nothing on the other end — so the trace shows the request leaving and never shows it arriving. The obvious reading of that shape is “payment is dropping requests”, and a team can spend a long morning on that reading.

Restore the parent-based form and confirm the join rate returns to one:

cd "$HOME/otel-instrumentation-lab"
sed -i 's|^PAYMENT_SAMPLER=.*|PAYMENT_SAMPLER=parentbased_traceidratio|' .env
docker compose up -d payment
wait_up

hit 60 73.03
sleep 90
n '{ span.payment.amount = "73.03" }'
n '{ span.payment.amount = "73.03" } && { resource.service.name = "payment-svc" }'

The two numbers now match. parentbased_traceidratio applies its ratio only when there is no parent decision to honour; with a parent, it does what the parent did. That is the whole reason the production default is the parent-based form, and it is why the rate belongs on the service that starts traces rather than on every service in the fleet.

Task 9: What this lab deliberately does not cover

Tail sampling. Every decision here is a head decision: made at the root, before anyone knows whether the request was slow or failed. Keeping all errors and all slow traces while dropping most of the healthy ones requires buffering whole traces somewhere central, which means a collector with the tail-sampling processor between the services and Tempo. That is a different topology and it belongs to the collector-pipeline lab; the honest summary here is that head sampling is cheap and blind, and that the two are usually layered rather than chosen between.

Fleet configuration. Setting these variables per service in a compose file is fine for two services and is exactly the mechanism that drifts across thirty. The shared-library or operator-injection patterns from the SDK deployment lesson exist because the configuration that goes wrong is the same in every service.

Validation

cd "$HOME/otel-instrumentation-lab"
export TEMPO="http://127.0.0.1:3200"

tq() {
  curl -fsSG "$TEMPO/api/search" \
    --data-urlencode "q=$1" \
    --data-urlencode "start=$(date -d '-20 min' +%s)" \
    --data-urlencode "end=$(date +%s)" \
    --data-urlencode "limit=${2:-200}"
}
n() { tq "$1" "${2:-200}" | jq '.traces | length'; }

echo "== the launcher is the entry point"
docker compose exec checkout sh -c "tr '\\0' ' ' < /proc/1/cmdline"; echo

echo "== ten requests produce ten joined two-service traces"
for i in $(seq 1 10); do
  curl -sf -o /dev/null "http://127.0.0.1:8080/checkout?amount=77.77" || true
done
sleep 90
echo "  traces from this run: $(n '{ span.payment.amount = "77.77" }')   (expect 10)"
echo "  of those, joined:     $(n '{ span.payment.amount = "77.77" } && { resource.service.name = "payment-svc" }')   (expect 10)"

echo "== payment never roots a trace"
n '{ trace:rootService = "payment-svc" }'

echo "== the manual domain span exists and carries its attribute"
n '{ span:name = "charge.authorise" }'
n '{ span.payment.method = "card" }'

echo "== the redaction holds on the post-redaction request"
TID="$(tq '{ span.payment.amount = "88.88" }' 1 | jq -r '.traces[0].traceID')"
curl -fsS -H 'Accept: application/json' "$TEMPO/api/traces/$TID" \
  | grep -c hunter2 || echo "  no credential in the trace"

The payment-rooted count is the one to watch. It is non-zero only for the window in which propagation was broken, and it never grows again afterwards — which is the cleanest possible signal that the boundary is intact.

Expected Outcome

With python app.py as the entry point, both services run and Tempo holds zero traces for them.
With opentelemetry-instrument python app.py, one request produces one trace containing spans from both services, and trace:rootService = "payment-svc" is zero.
With OTEL_PROPAGATORS=none on checkout, the same request produces two traces, and the payment-rooted count starts growing.
{ span:name = "charge.authorise" } finds the manual span; the failing ones carry an event named exception.
The token sent alongside amount=99.99 is present in a span attribute; the same token sent alongside amount=88.88 after the hook is not — confirmed by re-reading both traces rather than by trusting the hook.
A ratio of 0.1 on the entry service produces roughly ten traces per hundred requests, identified by their marker amount rather than by arithmetic on a running total.
A bare traceidratio on the downstream service leaves about one trace in ten complete while the entry service still records every one of them; restoring parentbased_traceidratio brings the two counts back into line.

Troubleshooting

The build fails resolving the OpenTelemetry packages. The distro and the instrumentation libraries are pre-release versioned and move together. Install them in one pip install invocation, as the Dockerfile does, so the resolver sees them at once; installing them in separate layers is how the versions end up mismatched.

Instrumentation is on but no spans appear. In order: is the entry point the launcher — read it from the process table with docker compose exec checkout sh -c "tr '\\0' ' ' < /proc/1/cmdline", which must show opentelemetry-instrument; did opentelemetry-bootstrap -a install run at build time, so the Flask and requests instrumentations are present; can the container reach the collector endpoint at all; and has one max_block_duration passed, since search needs a block to have been cut.

Spans appear under unknown_service. OTEL_SERVICE_NAME did not reach the process. Check docker compose exec checkout env | grep OTEL_. The service name is a resource attribute and nothing else supplies it, which is why the SDK falls back to a placeholder rather than failing.

The two services produce separate traces even with propagation on. Confirm the header on the wire with the docker compose exec snippet in Task 5. If traceparent is absent, the requests instrumentation is not loaded — go back to the previous item. If it is present, the receiving side is not extracting it, which usually means the Flask instrumentation is missing on payment.

grep -c hunter2 returns a count after the redaction. Read the attribute dump again: the agent is using a key spelling the hook does not set. Add it. This is the failure the callout warns about, and finding it here is the point.

The sampled counts look wrong. Sampling is probabilistic and a hundred requests is a small sample. Run each measurement three times. If every run returns zero, check that OTEL_TRACES_SAMPLER_ARG is a decimal fraction — 10 means ten, not ten percent, and is clamped to “sample everything”.

Nothing changes after editing .env. Compose reads .env when it builds the configuration, so the container has to be recreated. Naming the service on docker compose up -d re-creates it; docker compose restart reuses the container it already has, environment and all.

Cleanup

cd "$HOME/otel-instrumentation-lab"

# 1. Stop everything and destroy the trace store.
docker compose down -v --rmi local --remove-orphans
docker volume ls | grep tempo-data || echo "volume gone"

# 2. The Tempo image was pulled, not built. Remove it only if it was not
#    already on the host before this lab.
grep -E 'grafana/tempo' images.pre-lab || echo "tempo image was not present before"
# docker image rm "grafana/tempo:2.6.1"

# 3. The source tree, the Dockerfiles and .env.
cd "$HOME"
rm -rf "$HOME/otel-instrumentation-lab"

# 4. Confirm the host is as you found it.
ss -ltn 2>/dev/null | grep -E ':(3200|4317|8080)' \
  || echo "all three lab ports are free again"
unset TEMPO

--rmi local removes the two images built from these Dockerfiles and leaves anything pulled from a registry alone, which is why step 2 is separate.

What You Learned

Shipping the agent and activating it are two decisions. The image carried opentelemetry-distro from the first build and produced nothing until the entry point changed. That separation is what makes instrumentation a deployment-time toggle instead of a rebuild, and it is also the failure: a Dockerfile “cleanup” that restores python app.py silences telemetry with no error anywhere.
A joined trace is a claim you can test. { A } >> { B } counts the traces where the caller and the callee are actually related, and trace:rootService = "payment-svc" counts the ones that broke. Two queries turn “is propagation working” from an argument into a number.
The boundary fails without breaking anything else. With the propagator off, both services emitted correct spans, correct latencies and correct error rates. Only the relationship disappeared. In production the cause is usually a middlebox rather than a variable, and the symptom is identical — which is why the propagation audit belongs in the change runbook for every proxy on the path.
The agent instruments protocols; you instrument the domain. A span named charge.authorise, carrying an amount and a method, is one no agent could have invented, and the with block is what guarantees it is exported on the failure path — which is the path you needed it on.
A redaction you did not re-read is a hope. You put a token in a URL, found it in a span attribute, wrote a hook, and then went back and looked for the token again. The last step is the one that distinguishes a control from a comment.
parentbased_ is not decoration. A bare traceidratio on a downstream service ignores the caller’s decision and deletes most of the second half of every trace, producing a shape that reads as “the callee is dropping requests”. Head sampling belongs on the service that starts traces; everything downstream honours the parent.

Taking this to a real rollout

Sequence the rollout so each stage has its own rollback. Build: add the agent to the image and change nothing else — no telemetry, no risk, and the image is now ready. Activate on one service, at a low sample rate, and watch that service’s own latency and memory rather than the traces: the batch processor’s queue is bounded and the overhead is small, but “small” is a number you should measure on your workload rather than accept from a lab. Activate the callee, and only then does the join rate become meaningful. Raise the rate last, because that is the step that changes the bill.

Two things this lab simplifies. It uses Flask’s development server, which is a single process; a real deployment runs a WSGI server with workers and forks after the SDK initialises, which is its own failure mode and its own paragraph in the deployment lesson. And it exports straight from the application to Tempo. In production the export goes to a collector, because that is where sampling policy, redaction and retries belong — the SDK is inside somebody’s request path and should be doing as little as possible.

Finally, write down the sample rate and who owns it. It is the single number that decides both the trace bill and whether the trace you need at 03:00 exists at all, and it is the number most likely to have been set once by someone who has since left.

Objectives

Prerequisites