Lab: Deploy Tempo

Objective

By the end of this lab you will have a Tempo running every role in one process, accepting OTLP on both the gRPC and the HTTP port, writing a write-ahead log to a volume it can actually write to, keeping blocks in an S3-compatible bucket, and deleting them on a retention you chose. You will push a trace whose ID you generated yourself and get that exact ID back out of the query API, which is the only evidence that the whole write-then-read path works.

Then you will break it twice — once at the receiver, once at the bucket — and watch which checks notice and which stay green.

Architecture

One host, one Docker network, two long-running containers.

   your shell
     |  curl -X POST /v1/traces   (OTLP/HTTP, JSON encoding)
     |  curl -G   /api/search     (TraceQL)
     |  curl      /api/traces/ID  (trace by id)
     v
  +---------------------------------------------------------+
  | host (disposable)                                        |
  |                                                          |
  |   127.0.0.1:3200  Tempo HTTP  (/ready /metrics /api/...) |
  |   127.0.0.1:4317  OTLP gRPC                              |
  |   127.0.0.1:4318  OTLP HTTP                              |
  |        |                                                 |
  |   +----v-------------------------------+                 |
  |   | tempo (one process, all roles)     |                 |
  |   |  distributor -> ingester -> WAL    |                 |
  |   |       querier <- compactor         |                 |
  |   |  volume tempo-data:/var/tempo      |                 |
  |   +----------------+-------------------+                 |
  |                    | S3 API (stage 2)                    |
  |   +----------------v-------------------+                 |
  |   | minio  :9000 api  :9001 console    |                 |
  |   |  volume minio-data:/data           |                 |
  |   +------------------------------------+                 |
  +---------------------------------------------------------+

The lab runs in two stages on purpose. Stage 1 puts the blocks on the container filesystem so the first round trip has exactly one moving part. Stage 2 moves the same Tempo onto a bucket without touching anything else, so when something breaks you know which change broke it. That is also how a real migration should be sequenced, and it is why the storage lesson insists the bucket is a first-class production dependency rather than a detail of the config file.

Requirements

A disposable Linux host with Docker Engine and the Compose v2 plugin (docker compose version must answer). The lab writes only inside a directory you create and inside two named Docker volumes.
About 3 GB of free disk for the two images and the volumes.
curl, jq and od. od is in coreutils and is used to generate trace IDs without pulling in another dependency.
Ports 3200, 4317, 4318, 9000 and 9001 free on loopback. Task 1 checks.
GNU date. The OTLP payload needs nanosecond timestamps, and date +%s%N is a GNU extension. On macOS or BSD it returns a literal N, and every span you push either gets rejected or lands in 1970.
No out-of-band access requirement. Nothing here touches SSH, the host firewall, or the primary interface. Every published port is bound to 127.0.0.1, so nothing this lab starts is reachable from the network.

Scenario

Your platform has metrics and logs. Traces are the missing third, and the proposal on the table is “run the Tempo container from the quickstart and point the services at it”. That quickstart deliberately optimises for a first trace in ninety seconds: the blocks go on the container filesystem, retention is unset, and the receiver endpoints come from defaults that moved under a version bump.

You are building the version that gets reviewed. The target is explicit receiver endpoints, a WAL on a volume with the right ownership, blocks in a bucket, a retention someone signed off, and a validation that distinguishes “the process is up” from “a trace survives the round trip”. Monolithic mode is the right shape for this stage — one team, one host, well under the ceiling where the deployment-modes lesson says to move to simple scalable — and the point of writing it down now is that the migration later starts from something readable.

Tasks

Task 1: Capture the starting state

LAB="$HOME/tempo-deploy-lab"
mkdir -p "$LAB"
cd "$LAB"

# Every listener on the host, so Cleanup can compare against it.
ss -ltn 2>/dev/null | tee ports.pre-lab
grep -E ':(3200|4317|4318|9000|9001)' ports.pre-lab \
  || echo "all five lab ports are free"

# Which images are already local? Cleanup reads this before removing any.
docker image ls --format '{{.Repository}}:{{.Tag}}' | sort | tee images.pre-lab

docker compose version

If the grep printed a listener, stop and resolve it. A port conflict here surfaces three tasks later as a container that restarts in a loop with a message about an address, and by then you have changed four other things.

Pin an image tag rather than tracking latest. Find out what the registry actually offers instead of assuming a version string — the same discipline as apt-cache madison in the Grafana lab, and the reason a “reproducible” stack stops being reproducible the first time somebody deletes the tag to make the pull work:

curl -s 'https://hub.docker.com/v2/repositories/grafana/tempo/tags?page_size=25' \
  | jq -r '.results[].name' | grep -E '^[0-9]+\.[0-9]+' | head

# Substitute a tag the command above printed:
TEMPO_TAG=2.6.1
echo "TEMPO_TAG=$TEMPO_TAG" > .env

Everything below refers to that variable through the .env file, so the tag lives in one place and the compose file is identical on any host.

Task 2: Write the stack

Two files. The compose file first:

# docker-compose.yaml
services:
  # Tempo runs as uid 10001 inside the image, and a named volume that has
  # never been written to is created root-owned. Without this one-shot the
  # ingester starts, fails to open the WAL, and exits on a permission error
  # that reads exactly like a wrong path.
  tempo-init:
    image: grafana/tempo:${TEMPO_TAG}
    user: root
    entrypoint: ["chown", "-R", "10001:10001", "/var/tempo"]
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:${TEMPO_TAG}
    command:
      - "-config.file=/etc/tempo/tempo.yaml"
      - "-config.expand-env=true"
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      # Loopback only. Nothing this lab starts is reachable from the network.
      - "127.0.0.1:3200:3200"
      - "127.0.0.1:4317:4317"
      - "127.0.0.1:4318:4318"
    restart: unless-stopped

volumes:
  tempo-data:

Then the Tempo configuration. Every key below is one this lab or a later task depends on; there is nothing in it for decoration:

# tempo.yaml — stage 1, blocks on the local filesystem backend
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        # Both endpoints are written out in full. Leaving them empty makes
        # them inherit whatever the embedded OTLP receiver defaults to in
        # this build, and that default moved from 0.0.0.0 to localhost in
        # recent collector releases. Inside a container, localhost means
        # "not reachable from the published port".
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  # How long a block stays open before it is cut and flushed. The production
  # default is far larger; one minute here means the storage tasks produce a
  # visible object inside the lab session rather than half an hour after you
  # have stopped watching.
  max_block_duration: 1m

compactor:
  compaction:
    # The default is "keep forever", which is the single most expensive
    # inherited default in Tempo. Choose a number even in a lab.
    block_retention: 24h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal

Write both files:

cd "$HOME/tempo-deploy-lab"

cat > docker-compose.yaml <<'YAML'
services:
  tempo-init:
    image: grafana/tempo:${TEMPO_TAG}
    user: root
    entrypoint: ["chown", "-R", "10001:10001", "/var/tempo"]
    volumes:
      - tempo-data:/var/tempo

  tempo:
    image: grafana/tempo:${TEMPO_TAG}
    command:
      - "-config.file=/etc/tempo/tempo.yaml"
      - "-config.expand-env=true"
    depends_on:
      tempo-init:
        condition: service_completed_successfully
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "127.0.0.1:3200:3200"
      - "127.0.0.1:4317:4317"
      - "127.0.0.1:4318:4318"
    restart: unless-stopped

volumes:
  tempo-data:
YAML

cat > tempo.yaml <<'YAML'
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 1m

compactor:
  compaction:
    block_retention: 24h

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/blocks
    wal:
      path: /var/tempo/wal
YAML

docker compose config >/dev/null && echo "compose file parses"

Why the chown container exists at all

The grafana/tempo image drops to an unprivileged uid. A Docker named volume that has never been written to is created empty and owned by root, and the kernel does not care that the process inside the container is called tempo. So the ingester opens the WAL directory, gets EACCES, and the process exits during start-up.

The fix is not to run Tempo as root. It is to make the volume’s ownership match the uid the image runs as, once, before the first start. service_completed_successfully is what sequences it: the init container has to exit zero before Tempo is allowed to start, so a chown that fails stops the stack rather than producing a Tempo that half-works.

The same shape appears in Kubernetes as fsGroup on the pod security context. Different mechanism, identical failure, and in both cases the log message points at the path rather than at the ownership.

Task 3: Verify the config before starting anything

The validation lesson puts static config checking first for a reason: there is no point testing readiness on a config that does not parse.

cd "$HOME/tempo-deploy-lab"
docker compose run --rm --no-deps tempo \
  -config.file=/etc/tempo/tempo.yaml -config.verify

Exit zero means the file parsed and every key resolved to something the binary knows. It does not mean the bucket is reachable, the credential is valid, or the port is free — the verifier makes no network calls at all. That distinction is the entire reason the later tasks exist.

If your tag does not recognise the flag it will say so on the first line. Fall back to the same check with more cleanup: run docker compose up tempo in the foreground, read the first twenty log lines, and interrupt it. A config error is fatal at start-up either way; the flag only saves you the volume write.

Task 4: Start it, and run the two cheap checks

Service impact possiblelab host— Pulls the image on first run, chowns the volume, then starts Tempo

$ docker compose up -d

docker compose ps
docker compose logs tempo | head -30

The log’s first lines name the version and the modules the process registered. Read them now rather than after something breaks: this is the only place the running build identifies itself, and “which Tempo am I actually running” is the first question in every Tempo incident.

Layer 1 is the process. Layer 2 is the local HTTP probe:

for i in $(seq 1 30); do
  curl -fsS http://127.0.0.1:3200/ready && break
  sleep 2
done

curl -fsS http://127.0.0.1:3200/api/echo
curl -fsS -o /dev/null -w 'metrics: HTTP %{http_code}\n' \
  http://127.0.0.1:3200/metrics

/ready answers when the process has registered its roles. /api/echo is the cheapest possible proof that the HTTP router is serving. Neither has looked at a receiver port, a bucket, or a span — a fact Task 6 makes uncomfortably concrete.

Confirm the receiver ports are actually bound, which is the check /ready cannot make for you:

# Published ports, from the host's point of view.
docker compose port tempo 4317
docker compose port tempo 4318

# A real request. 4318 speaks HTTP, so an empty POST is answered rather than
# ignored, and any HTTP status at all proves the listener exists.
curl -s -o /dev/null -w 'otlp/http: HTTP %{http_code}\n' \
  -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' --data '{}'

Task 5: Push a trace whose ID you chose, and read it back

Every “is tracing working?” conversation that ends in an argument does so because nobody controlled the trace ID. Generate it yourself and the round trip becomes a yes-or-no question.

OTLP over HTTP accepts a JSON encoding, so no SDK, agent or extra image is needed — the payload below is the OTLP trace structure written out by hand. Save the generator:

cd "$HOME/tempo-deploy-lab"

cat > push-span.sh <<'SH'
#!/usr/bin/env bash
# Push one OTLP/JSON span into Tempo. Prints the trace ID it used.
set -euo pipefail

OTLP_HTTP="${OTLP_HTTP:-http://127.0.0.1:4318}"
SERVICE="${SERVICE:-checkout-api}"
TRACE_ID="${1:-$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')}"
SPAN_ID="$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n')"

END_NS="$(date +%s%N)"
START_NS=$(( END_NS - 250000000 ))   # a 250 ms span

resp="$(mktemp)"
payload="$(cat <<JSON
{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        { "key": "service.name",
          "value": { "stringValue": "${SERVICE}" } },
        { "key": "deployment.environment",
          "value": { "stringValue": "lab" } }
      ]
    },
    "scopeSpans": [{
      "scope": { "name": "runbook-academy.tempo-deploy-lab" },
      "spans": [{
        "traceId": "${TRACE_ID}",
        "spanId": "${SPAN_ID}",
        "name": "POST /checkout",
        "kind": 2,
        "startTimeUnixNano": "${START_NS}",
        "endTimeUnixNano": "${END_NS}",
        "attributes": [
          { "key": "http.request.method",
            "value": { "stringValue": "POST" } },
          { "key": "http.response.status_code",
            "value": { "intValue": "200" } }
        ],
        "status": { "code": 1 }
      }]
    }]
  }]
}
JSON
)"

code="$(curl -s -o "$resp" -w '%{http_code}' \
  -X POST "${OTLP_HTTP}/v1/traces" \
  -H 'Content-Type: application/json' \
  --data-binary "$payload")"

printf 'http=%s trace_id=%s span_id=%s\n' "$code" "$TRACE_ID" "$SPAN_ID"
cat "$resp"; echo
rm -f "$resp"
SH

chmod +x push-span.sh

Two details in that payload repay reading rather than pasting. kind: 2 is SPAN_KIND_SERVER — the span kind is a number on the wire, and getting it wrong is how the entry point of a request ends up filed as an internal operation and disappears from every server-side query. status.code: 1 is OK; 2 is ERROR and 0 is UNSET, which is what an uninstrumented handler leaves behind, and why unset is not a synonym for “succeeded”.

Push a span and keep the ID:

cd "$HOME/tempo-deploy-lab"
TRACE_ID="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"
./push-span.sh "$TRACE_ID"

An HTTP 200 means the receiver accepted the request. It does not mean the span was stored: OTLP reports per-span rejections inside a 200 as a partial success in the response body, which is why the script prints the body.

Now read it back. Tempo answers a by-ID lookup from the ingester before the block is ever flushed, so this works immediately:

curl -fsS -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/${TRACE_ID}" | jq 'keys'

Ask for JSON explicitly. The trace-by-ID endpoint returns protobuf unless the Accept header says otherwise, and a jq failure on binary output is the least informative error in this lab. The top-level key is batches on some versions and resourceSpans on others; the keys call tells you which, and then:

curl -fsS -H 'Accept: application/json' \
  "http://127.0.0.1:3200/api/traces/${TRACE_ID}" \
  | jq '[.. | objects | select(has("name") and has("spanId")) | .name]'

That filter walks whatever shape came back and pulls out the span names, so it survives the difference. The name you pushed is the proof the write path works.

Search is the other half, and it is not immediate. A search has to find the span in a block, so wait out max_block_duration before expecting a hit:

sleep 90

START="$(date -d '-15 min' +%s)"
END="$(date +%s)"

curl -fsSG http://127.0.0.1:3200/api/search \
  --data-urlencode 'q={ resource.service.name = "checkout-api" }' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode 'limit=5' \
  | jq '.traces | length'

Task 6: Unbind the receiver, and watch /ready stay green

Delete the HTTP protocol block — the same shape as the typo the validation lesson opens with, where a receiver key was accepted and bound nothing:

cd "$HOME/tempo-deploy-lab"
cp tempo.yaml tempo.yaml.working

# Remove the `http:` key and the endpoint line under it.
sed -i '/^        http:$/,+1d' tempo.yaml
sed -n '/receivers:/,/^ingester:/p' tempo.yaml

docker compose restart tempo
sleep 8

Now run the checks in the order most monitoring systems run them:

# Layer 1: the container.
docker compose ps --format '{{.Service}} {{.State}}'

# Layer 2: readiness. Still 200.
curl -s -o /dev/null -w 'ready: HTTP %{http_code}\n' http://127.0.0.1:3200/ready

# Layer 3: the receiver. The first check that notices.
curl -s -o /dev/null -w 'otlp/http: HTTP %{http_code}\n' \
  --max-time 5 -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' --data '{}'

The container is running, /ready is 200, and the OTLP HTTP endpoint returns 000 — curl’s code for “no HTTP response at all”. Every client pointed at 4318 is now failing and nothing in layers 1 and 2 has changed colour. The gRPC receiver on 4317 is untouched, which is what makes this failure so durable in the wild: half the fleet keeps working, the platform dashboard stays green, and one language’s services quietly stop producing traces.

Put it back:

cd "$HOME/tempo-deploy-lab"
mv tempo.yaml.working tempo.yaml
docker compose restart tempo
sleep 8
curl -s -o /dev/null -w 'otlp/http: HTTP %{http_code}\n' \
  -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' --data '{}'

Task 7: Move the trace store onto a bucket

Nothing about Tempo changes here except the storage stanza. Add MinIO and a throwaway mc client to the stack:

cd "$HOME/tempo-deploy-lab"

# A password for this lab only, kept in the compose env file rather than in
# tempo.yaml. Tempo expands ${...} because of -config.expand-env=true.
MINIO_PASSWORD="$(od -An -N18 -tx1 /dev/urandom | tr -d ' \n')"
cat >> .env <<ENVEOF
MINIO_ROOT_USER=tempo-lab
MINIO_ROOT_PASSWORD=$MINIO_PASSWORD
TEMPO_S3_ACCESS_KEY=tempo-lab
TEMPO_S3_SECRET_KEY=$MINIO_PASSWORD
ENVEOF
chmod 0600 .env

The two new services, and the environment block that hands Tempo the credential through the process environment instead of through the file in your editor and your git history:

# docker-compose.yaml — additions
  tempo:
    environment:
      TEMPO_S3_ACCESS_KEY: ${TEMPO_S3_ACCESS_KEY}
      TEMPO_S3_SECRET_KEY: ${TEMPO_S3_SECRET_KEY}

  minio:
    image: minio/minio
    command: ["server", "/data", "--console-address", ":9001"]
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "127.0.0.1:9000:9000"
      - "127.0.0.1:9001:9001"
    restart: unless-stopped

  # Not started by `up`; run on demand with `docker compose run --rm mc`.
  mc:
    image: minio/mc
    profiles: ["tools"]
    entrypoint: ["sh"]
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}

volumes:
  minio-data:

Apply that by rewriting the file’s tail, then check it still parses:

cd "$HOME/tempo-deploy-lab"

# Drop the old volumes: block, append the new services, then one volumes:
# block covering both. Compose is strict about structure.
sed -i '/^volumes:$/,$d' docker-compose.yaml

cat >> docker-compose.yaml <<'YAML'
    environment:
      TEMPO_S3_ACCESS_KEY: ${TEMPO_S3_ACCESS_KEY}
      TEMPO_S3_SECRET_KEY: ${TEMPO_S3_SECRET_KEY}

  minio:
    image: minio/minio
    command: ["server", "/data", "--console-address", ":9001"]
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "127.0.0.1:9000:9000"
      - "127.0.0.1:9001:9001"
    restart: unless-stopped

  mc:
    image: minio/mc
    profiles: ["tools"]
    entrypoint: ["sh"]
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}

volumes:
  tempo-data:
  minio-data:
YAML

docker compose config >/dev/null && echo "compose file still parses"

If docker compose config complains, open the file: the environment: block has to land inside the tempo: service, indented four spaces, above the blank line before minio:. Fix it there rather than re-running the sed.

docker compose up -d minio

# Create the bucket. `mc alias set` writes its config inside the throwaway
# container, so no credential is left on your host filesystem.
docker compose run --rm mc -c '
  mc alias set lab http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" &&
  mc mb --ignore-existing lab/tempo-traces &&
  mc ls lab'

Now the storage stanza:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
      # MinIO is plain HTTP inside this network. In production this is false
      # and the endpoint is TLS. `insecure` is not a MinIO requirement; it is
      # a statement about this lab's network.
      insecure: true
      # Mandatory for MinIO and for most S3-compatible stores: they do not
      # serve virtual-hosted-style bucket names, so the bucket has to go in
      # the path. Getting this wrong produces redirects the SDK mishandles,
      # and the symptom is writes that work small and fail big.
      forcepathstyle: true
    wal:
      path: /var/tempo/wal

cd "$HOME/tempo-deploy-lab"
sed -i '/^storage:$/,$d' tempo.yaml

cat >> tempo.yaml <<'YAML'
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
      insecure: true
      forcepathstyle: true
    wal:
      path: /var/tempo/wal
YAML

docker compose up -d tempo
sleep 8
curl -fsS http://127.0.0.1:3200/ready

Push spans and wait for a block to be cut and uploaded:

cd "$HOME/tempo-deploy-lab"
for i in $(seq 1 20); do ./push-span.sh >/dev/null; done
sleep 90

docker compose run --rm mc -c '
  mc alias set lab http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null &&
  mc ls --recursive lab/tempo-traces'

Objects under the bucket are the proof. A block is a small directory of files — metadata, the span data, and the bloom filter that lets a querier skip the block without reading it — and that layout is exactly what the storage lesson means by index-less. There is no separate index to operate, so the bucket listing is the index.

Task 8: Break the credential, and find the metric that says so

This is the failure the storage lesson opens with, and it is worth producing yourself once, because the shape is so counter-intuitive: writes keep being accepted while nothing is being stored.

cd "$HOME/tempo-deploy-lab"
cp .env .env.working
sed -i 's/^TEMPO_S3_SECRET_KEY=.*/TEMPO_S3_SECRET_KEY=not-the-password/' .env
docker compose up -d tempo
sleep 8

# The receiver still accepts spans. Every one of these returns 200.
for i in $(seq 1 20); do ./push-span.sh >/dev/null; done
sleep 90

Do not guess the metric name — ask the process which ones it has:

curl -s http://127.0.0.1:3200/metrics \
  | grep -E '^tempo_ingester_[a-z_]*(flush|fail)[a-z_]*' | grep -v '^#'

docker compose logs --tail 40 tempo | grep -iE 'access|denied|flush|error'

The flush-failure series is non-zero and rising, the log carries the object store’s own rejection, and the bucket has gained nothing:

docker compose run --rm mc -c '
  mc alias set lab http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null &&
  mc ls --recursive lab/tempo-traces | wc -l'

Nothing upstream noticed. /ready is 200, the container is up, the OTLP endpoint returns 200 to every client, and the only signal is a counter nobody is scraping. That is why the meta-monitoring rule for a trace store is to alert on the flush-failure counter and on “no new objects in the bucket for N minutes”, never on the readiness probe.

Restore the credential:

cd "$HOME/tempo-deploy-lab"
mv .env.working .env
docker compose up -d tempo
sleep 8
for i in $(seq 1 5); do ./push-span.sh >/dev/null; done

Task 9: Read the retention back out of the running process

You set block_retention: 24h in Task 2. Confirm the process agrees, because a value in a file you edited is a hypothesis until the running binary repeats it back:

curl -fsS http://127.0.0.1:3200/status/config | grep -A4 -i 'compaction' \
  || curl -fsS http://127.0.0.1:3200/config | grep -A4 -i 'compaction'

What you cannot prove in a ninety-minute lab is the deletion itself: no block here is twenty-four hours old. Say that plainly rather than writing a check that pretends otherwise. What you can establish is that the compactor role is running in this process and that its counters exist to be alerted on:

curl -s http://127.0.0.1:3200/metrics | grep -E '^tempo_compact' \
  | grep -v '^#' | head

In production the proof of retention is a non-zero deletion counter over a window longer than the retention, plus a bucket whose size stops growing. Both are alerts, not lab steps — and a compactor that has never deleted anything in thirty days is the most common cause of a Tempo bill nobody budgeted for.

Validation

One transcript, run top to bottom.

cd "$HOME/tempo-deploy-lab"
T=http://127.0.0.1:3200

echo "== layer 1: process"
docker compose ps --format '{{.Service}} {{.State}}'

echo "== layer 2: readiness and router"
curl -fsS "$T/ready"
curl -fsS "$T/api/echo"

echo "== layer 3: both receivers are bound"
curl -s -o /dev/null -w '  otlp/http 4318: HTTP %{http_code}\n' --max-time 5 \
  -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' --data '{}'
docker compose port tempo 4317

echo "== layer 4: a trace you chose survives the round trip"
TRACE_ID="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"
./push-span.sh "$TRACE_ID" >/dev/null
sleep 3
curl -fsS -o /dev/null -w "  by-id lookup: HTTP %{http_code}\n" \
  -H 'Accept: application/json' "$T/api/traces/$TRACE_ID"

echo "== layer 5: storage is actually receiving blocks"
docker compose run --rm mc -c '
  mc alias set lab http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null &&
  mc ls --recursive lab/tempo-traces | wc -l'

echo "== no flush is failing right now"
curl -s "$T/metrics" | grep -E '^tempo_ingester_[a-z_]*fail[a-z_]*' | grep -v '^#'

Layer 5 is what separates this from the quickstart. A count that grows between two runs of this block, minutes apart, is the evidence that data is landing somewhere durable rather than accumulating in a WAL that a restart will discard.

Expected Outcome

docker compose ps shows tempo and minio running, and tempo-init exited zero.
/ready returns 200 and both 4317 and 4318 accept a connection.
A trace ID you generated comes back from the by-ID endpoint within seconds of the push, and a TraceQL search finds the service within two block durations.
mc ls --recursive lab/tempo-traces lists objects, and the count grows over time.
The flush-failure counter is zero with the correct credential and non-zero with the wrong one — and you watched /ready stay at 200 through both.
tempo.yaml contains no inherited default for the receiver endpoints, the WAL path, or block_retention.

Troubleshooting

Tempo restarts in a loop and the log mentions permission. The tempo-init container did not run, or the volume was created by an earlier attempt with different ownership. docker compose down -v and start again; the init step runs before Tempo on every up.

Start-up fails on an unknown or invalid key in tempo.yaml. Configuration keys move between Tempo minor versions. The configuration reference for your tag is the authority — not this lab, and not a blog post. Remove the key it names, re-run the Task 3 verification, and write the change down: a key that moved is exactly what an upgrade runbook needs to record.

The OTLP push returns HTTP 200 but nothing is retrievable. Read the response body the script prints; OTLP reports per-span rejections as a partial success inside a 200. The usual cause is a malformed timestamp — check that date +%s%N prints digits and not a trailing letter N.

The by-ID lookup returns 404 immediately after a push. Give it a couple of seconds; the distributor forwards asynchronously. If it is still 404 after ten, confirm the ID in the URL is the one the script printed. A trace ID is 32 hex characters, and a 31-character ID from a truncated copy is rejected without explanation.

jq fails with a parse error on the by-ID response. The Accept: application/json header is missing, so you are piping protobuf into jq.

The search returns zero while the by-ID lookup works. Almost always the time window. start and end are unix seconds and the default window is narrow; widen it before suspecting anything else. If it is still zero after max_block_duration plus a minute, the block has not been cut yet.

mc cannot reach MinIO. docker compose run attaches the container to the project network, so the hostname is minio and not localhost. A connection refused from inside that container means MinIO is not up yet — check docker compose ps before suspecting the credentials.

MinIO writes work for a while and then fail on larger objects. forcepathstyle is missing or false. Virtual-hosted-style addressing turns the bucket name into a subdomain, which MinIO answers with a redirect the SDK does not follow the way you expect. This is the failure that looks intermittent and is not.

Cleanup

cd "$HOME/tempo-deploy-lab"

# 1. Stop the stack and destroy its volumes and network.
docker compose down -v --remove-orphans
docker volume ls | grep -E 'tempo-data|minio-data' || echo "volumes gone"

# 2. Remove only the images this lab pulled. Read the Task 1 capture first
#    and drop from the list anything that was already present.
grep -E 'grafana/tempo|minio/' images.pre-lab || echo "none were present before"
# docker image rm "grafana/tempo:$TEMPO_TAG" minio/minio minio/mc

# 3. The lab directory, including .env with the MinIO password.
cd "$HOME"
rm -rf "$HOME/tempo-deploy-lab"

# 4. Confirm the host is as you found it.
ss -ltn 2>/dev/null | grep -E ':(3200|4317|4318|9000|9001)' \
  || echo "all five lab ports are free again"

If a port is still bound, a container survived: docker ps -a and remove it by name.

What You Learned

A default endpoint is a decision somebody else made, in a release you did not read. Writing 0.0.0.0:4317 and 0.0.0.0:4318 in full is three seconds of typing that survives the version bump where the embedded receiver’s default bind address changed. Every receiver in the stack deserves the same treatment.
/ready answers a question about roles, not about ports. You removed a receiver and watched readiness stay at 200 while every HTTP client failed. Any smoke test that stops at layer 2 reports that Tempo as healthy for as long as nobody opens a trace.
Controlling the trace ID turns a debate into a check. Because you generated the ID, the by-ID lookup is proof rather than coincidence — and it separates “the write path works” from “a block was cut and flushed”, which are different failures with different fixes and very different waits.
The bucket is the trace store; Tempo is the query engine on top of it. With a wrong secret key the receiver kept returning 200, the process stayed ready, and nothing was stored. The only signal was a counter. That is the shape of most “traces are missing” incidents, and it is why the alert belongs on the flush-failure series and on bucket growth.
Retention that was never chosen is retention set to forever. You put block_retention in the config and read it back out of the running process, and you saw that a lab cannot prove the deletion — which is itself the point: in production, the proof is a non-zero deletion counter over a window longer than the retention.
Ownership of a fresh volume is a start-up dependency, not a detail. The init container exists because a named volume is root-owned until something writes to it, and the resulting error names the path rather than the uid.

Taking this to a real change window

This deploy maps onto four blocks with four different rollbacks. Images and compose: rollback is down, no user impact, nothing has been written. Configuration: rollback is the previous tempo.yaml plus a restart, and it is fast — but note that Tempo reads its configuration at start-up and has no runtime reload, so every config change in production is a restart and should be scheduled like one. Storage backend: this is the one-way door, because blocks written to the old backend are not visible from the new one. Run both for one retention period or accept a gap in query coverage, and decide which before the change rather than during it. Retention: shortening block_retention deletes data on the compactor’s next cycle, so treat it as a destructive change with a named approver.

Three things this lab simplifies that production does not. MinIO’s root credential is Tempo’s credential here; in production Tempo gets a scoped identity — workload identity on a managed cluster, or at minimum a user restricted to one bucket prefix — because a leaked key with read access to the trace bucket exposes every request detail anyone ever put in a span attribute. insecure: true is a statement about this lab’s network and belongs nowhere near a real one. And monolithic mode has a ceiling: when ingest outgrows one host the fix is a deployment-mode change, not a bigger host, and it is far cheaper to plan that migration while the current mode still works than after the first out-of-memory kill.

Objectives

Prerequisites