Distributed tracing with OpenTelemetry

Logs and metrics both stop at the container boundary. A log line belongs to one container; a metric is scraped from one target. When a request touches five containers you get five disconnected views and a timestamp to glue them with.

A trace is the one signal built to cross that boundary. It follows a single request through every hop and records what each one spent, and it does that by carrying a small piece of state — the trace context — in the request itself.

Everything difficult about tracing on a Docker host is about that state: making sure it is created, that it survives each hop, and that the resulting spans reach somewhere you can read them.

The span model

A trace is a tree of spans. A span is one unit of work with a start time and a duration, and it knows its parent.

flowchart TD
  A["SERVER span: GET /orders/42<br/>api · 412ms"] --> B["CLIENT span: SELECT orders<br/>api · 340ms"]
  A --> C["CLIENT span: GET /inventory<br/>api · 58ms"]
  C --> D["SERVER span: GET /inventory<br/>inventory · 51ms"]
  D --> E["CLIENT span: GET items<br/>inventory · 44ms"]

Reading that tree answers the question metrics cannot: the 412 ms was not distributed across the request, it was 340 ms in one database query, and the inventory call everyone suspected cost 58 ms.

Each span carries:

Kind is the field that gets ignored and should not be. A CLIENT span and the SERVER span it produced are the two ends of the same network call, recorded by two different processes. The gap between them — CLIENT duration minus SERVER duration — is network plus queueing plus TLS, and it is the only place that number is visible anywhere in your stack.

Context propagation across the container boundary

This is the mechanism, and it is smaller than people expect.

When the API calls the inventory service, its HTTP client adds a header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Four dash-separated fields, defined by W3C Trace Context:

An all-zero trace ID or span ID is invalid and must be rejected. A companion tracestate header carries vendor-specific data alongside it.

The receiving service’s instrumentation reads traceparent, uses that trace ID rather than generating one, and sets its new SERVER span’s parent to the incoming span ID. That is the whole of distributed tracing. The header is the only thing that crosses the container boundary; everything else is reconstructed in the backend by joining on trace ID.

Instrumenting a containerised application

There are two routes and you will use both.

Zero-code (auto) instrumentation wraps the runtime and hooks known libraries — the web framework, the HTTP client, the database driver. It produces the tree above with no application changes, and it is where every service should start.

Manual instrumentation adds spans for work the libraries cannot see: a business operation, a loop over a batch, a call into something exotic.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("reprice-basket") as span:
    span.set_attribute("basket.item_count", len(items))
    for item in items:
        with tracer.start_as_current_span("reprice-item"):
            reprice(item)
    span.set_status(trace.StatusCode.OK)

The Docker-specific part is how the agent gets into the image. It differs per runtime and the shape is always the same — wrap the entrypoint:

FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir opentelemetry-distro opentelemetry-exporter-otlp \
&& opentelemetry-bootstrap --action=install

COPY . .

# The wrapper, not the application, is PID 1.
ENTRYPOINT ["opentelemetry-instrument", "python", "-m", "myapp"]

Go is the exception worth knowing about before you plan a rollout: it has no runtime hook point, so every Go service needs a code change.

Configuration: environment variables, not code

Every OpenTelemetry SDK reads the same environment variables, which means the entire configuration belongs in Compose rather than in the image.

services:
api:
  image: example/api:2.4.1
  environment:
    # Names the service in every trace. Without it: unknown_service.
    OTEL_SERVICE_NAME: "api"
    # Container-level identity, so you can tell replicas apart.
    OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=prod,service.version=2.4.1"
    # The collector by SERVICE NAME. Not localhost. See below.
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
    OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
    OTEL_TRACES_SAMPLER: "parentbased_traceidratio"
    OTEL_TRACES_SAMPLER_ARG: "0.1"
  networks: [observability, edge]

otel-collector:
  image: otel/opentelemetry-collector-contrib:0.140.0
  command: ["--config=/etc/otel/config.yaml"]
  volumes:
    - ./otel-collector.yaml:/etc/otel/config.yaml:ro
  networks: [observability]

networks:
edge:
observability:

The defaults are worth committing to memory because they explain most first-run failures:

One subtlety that costs an afternoon: with the generic OTEL_EXPORTER_OTLP_ENDPOINT, the SDK appends the signal path — so http://otel-collector:4318 becomes http://otel-collector:4318/v1/traces. With the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, the URL “MUST be used as-is without any modification”, so you must include /v1/traces yourself. Setting the signal-specific variable to a bare host and port produces a 404 from the collector and no traces, with an error message that mentions neither.

Short-lived containers lose their spans

The SDK batches spans and exports them on a timer. A container that exits between two exports takes the unexported batch with it.

That is invisible for a long-running service and total for a CLI task, a migration job, or a cron container — exactly the workloads where a trace would be most useful, because you cannot attach to them afterwards.

Three requirements, all of which must hold:

• The SDK’s shutdown must run, which means the process must handle SIGTERM rather than being killed. Auto-instrumentation wrappers register a shutdown hook; a process that ignores SIGTERM and is SIGKILLed at the stop timeout never reaches it.
• The wrapper must be PID 1, or the signal never arrives at it. ENTRYPOINT ["opentelemetry-instrument", ...] in exec form does this; ENTRYPOINT opentelemetry-instrument ... in shell form puts /bin/sh at PID 1 and the signal stops there.
• --stop-timeout must be long enough for the final flush to complete against a collector that might be slow.

For a job whose whole point is one trace, the robust answer is a SimpleSpanProcessor — export each span as it ends, accepting the latency cost — rather than the batching processor that is correct for a server.

Verification that can fail

Start at the collector and work backwards; it is far faster than starting at the UI.

COLLECTOR=otel-collector
APP=api

# 1. Did the collector accept spans, and did the exporter refuse any?
docker compose logs --tail 50 "$COLLECTOR" | grep -iE 'traces|refused|error'

# 2. The collector's own metrics: accepted vs refused, by signal.
docker compose exec -T "$COLLECTOR" wget -qO- http://localhost:8888/metrics | grep -E 'receiver_accepted_spans|receiver_refused_spans|exporter_send_failed'

# 3. Can the app container reach the collector at all?
docker compose exec -T "$APP" wget -qO- --timeout=3 --post-data='' http://otel-collector:4318/v1/traces ; echo "exit=$?"

# 4. Is the app configured the way you think?
docker compose exec -T "$APP" printenv | grep '^OTEL_'

receiver_accepted_spans at zero while the application is serving traffic means nothing is arriving — a client-side problem, so go to step 3 and 4. A non-zero receiver_accepted_spans with a non-zero exporter_send_failed_spans means the collector is receiving and failing to forward, which is a backend problem and a completely different investigation.

$ docker compose logs --since 2m api inventory | grep -o 'traceparent=00-[0-9a-f]*' | sort | uniq -c | sort -rn | head

      2 traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736
    2 traceparent=00-7c1a9e0b41d8f2635a0c8e1147bb90de
    1 traceparent=00-e3f0a2c88b174d6590aa2c1f3d7e4501

A trace ID appearing exactly once across two services that should both handle it is a broken hop. Two occurrences of the same ID is the context arriving intact.

Knowledge check

Where next

A trace tells you which hop was slow. It does not tell you what that hop logged. The next lesson connects the two, so a trace ID found in a slow span becomes a log query across every container that request touched.