Docker Traces

A checkout service is slow. The on-call engineer opens Tempo, looks up a recent trace by ID, and sees a single span: the incoming HTTP request from the load balancer. The downstream calls to the payment service, the database, and the cache are all missing. The trace was supposed to be a complete picture of the request; it is not. The container’s environment variables were set at the wrong level; the OpenTelemetry SDK inside the container never received the OTLP endpoint and never initialised a span exporter. The trace was generated and discarded.

This lesson is about the OTLP exporter path from inside a container to Tempo: what the SDK inside the container does, what the agent on the host does, and the right pattern per workload.

What it is

A trace from a Docker container starts with the OpenTelemetry SDK inside the container. The SDK instruments the application’s HTTP client, gRPC client, database driver, and other I/O libraries; generates spans; and ships them to a collector. The collector batches and forwards them to Tempo.

    application process (inside container)
        |
        |  OpenTelemetry SDK
        |  - auto-instrumentation
        |  - manual instrumentation
        |  - batch span processor
        v
    OTLP exporter
        |  gRPC :4317 or HTTP :4318
        +--------------+--------------+
                       |
              +--------+--------+
              |                 |
         agent on host    sidecar per container
         (Alloy or OTel    (Alloy or OTel
          Collector)         Collector)
              |                 |
              +--------+--------+
                       |
                       v
                     Tempo

The two deployment patterns are:

• Agent on the host. A single collector runs as a long-lived process on the host (or as a host-network container). Every application container exports to it via the host network. The collector forwards to Tempo.
• Sidecar per container. Every application container has its own collector sidecar. The sidecar is on a dedicated network namespace with the application container and forwards to Tempo.

Both patterns reach the same destination; the trade is in operational cost, isolation, and protocol compatibility.

Why a sysadmin cares

The right pattern for a given workload is the difference between “every container ships spans to Tempo” and “we cannot figure out which container’s traces are missing”. A team that picks the wrong pattern finds out at the first multi-tenant incident.

Three production shapes put the pattern choice on the critical path:

• A polyglot fleet. Different runtimes (Java, Python, Node, Go) have different auto-instrumentation quirks. A sidecar pattern keeps the SDK inside the container; the collector is language-agnostic and the application configures only the OTLP endpoint.
• A multi-tenant host. Two teams sharing a Docker host must not be able to read each other’s traces before the collector forwards them. The agent-on-host pattern is single-tenant by default; the sidecar pattern needs additional isolation.
• A trace budget. Tempo’s storage cost is bounded by the sampling rate. An agent-on-host enforces sampling centrally; per-container sampling is harder to reason about.

How it works

The OpenTelemetry SDK inside the container does three things:

1. Auto-instrumentation. The SDK patches the application’s HTTP client, gRPC client, database driver, and other I/O libraries at startup. Each patched library generates a span for the work it does. The patches are language-specific; Java uses a Java agent, Python uses a sitecustomize.py, Node uses a require hook, Go uses the otelhttp middleware.
2. Span context propagation. The SDK reads the W3C traceparent header on incoming requests and writes it on outgoing requests. This is how a trace from the load balancer is connected to a span in the payment service.
3. Export. The SDK batches spans and exports them via OTLP (gRPC on port 4317, HTTP on port 4318). The batch span processor handles retries, queue size, and timeout.

The collector on the other side receives the spans, applies sampling and redaction, and forwards to Tempo. The collector is the place where sampling decisions are enforced centrally and where batched exports to Tempo happen.

    container A                      container B
    +-------------------+            +-------------------+
    | app + OTel SDK    |            | app + OTel SDK    |
    | spans -- OTLP --> |            | spans -- OTLP --> |
    +--------- ---------+            +--------- ---------+
              |                                |
              +---------------+----------------+
                              |
                              v
                  collector (Alloy or OTel)
                              |
                              v
                            Tempo

How to configure it

Three configuration shapes cover most production needs.

Pattern A: agent on the host

# docker-compose.yml
services:
  app:
    image: registry.example.com/checkout:v3.4.1
    environment:
      OTEL_SERVICE_NAME: checkout
      OTEL_EXPORTER_OTLP_ENDPOINT: http://host.docker.internal:4317
      OTEL_RESOURCE_ATTRIBUTES: service.name=checkout,deployment.environment=prod,service.version=3.4.1
      OTEL_TRACES_SAMPLER: parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG: "0.1"
    depends_on:
      - collector
    network_mode: bridge

  collector:
    image: grafana/alloy:latest
    volumes:
      - ./alloy.river:/etc/alloy/config.river:ro
    network_mode: host
    restart: unless-stopped

# alloy.river
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.internal:4317"
  }
}

Severity: CONFIGURATION. Compose up to apply.

The agent runs on the host network; the application container exports to the host’s loopback. The collector is the single choke point for sampling and redaction.

Pattern B: sidecar per container

# docker-compose.yml
services:
  app:
    image: registry.example.com/checkout:v3.4.1
    environment:
      OTEL_SERVICE_NAME: checkout
      OTEL_EXPORTER_OTLP_ENDPOINT: http://collector-sidecar:4317
    networks:
      - app-net

  collector-sidecar:
    image: grafana/alloy:latest
    volumes:
      - ./alloy-sidecar.river:/etc/alloy/config.river:ro
    networks:
      - app-net
    restart: unless-stopped

# alloy-sidecar.river
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }

  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.internal:4317"
  }
}

Severity: CONFIGURATION. Compose up to apply.

The sidecar runs on the same network as the application. The OTLP endpoint is http://collector-sidecar:4317. The sidecar pattern keeps every container’s network traffic on a dedicated network namespace.

Pattern C: SDK inside the container, direct OTLP to Tempo

# docker-compose.yml
services:
  app:
    image: registry.example.com/checkout:v3.4.1
    environment:
      OTEL_SERVICE_NAME: checkout
      OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo.internal:4317
      OTEL_RESOURCE_ATTRIBUTES: service.name=checkout,deployment.environment=prod

Severity: CONFIGURATION. Compose up to apply.

The SDK ships directly to Tempo. This is the simplest pattern and is appropriate when Tempo is reachable, the network policy permits the connection, and there is no need for centralised sampling or redaction at a collector.

Java auto-instrumentation

# Build the image with the OpenTelemetry Java agent.
docker build -t checkout:v3.4.1 \
  --build-arg OTEL_AGENT_VERSION=2.10.0 \
  --build-arg OTEL_SERVICE_NAME=checkout .

docker run -d \
  --name checkout \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317 \
  -e OTEL_SERVICE_NAME=checkout \
  -e OTEL_RESOURCE_ATTRIBUTES=service.name=checkout,deployment.environment=prod \
  checkout:v3.4.1

Severity: CONFIGURATION. Image build and container start.

The OpenTelemetry Java agent is loaded via -javaagent:/path/to/opentelemetry-javaagent.jar. It patches the JVM at startup and instruments every supported library without code changes. The agent’s behaviour is controlled by the same OTEL_* environment variables as the SDK.

Python auto-instrumentation

docker run -d \
  --name checkout \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317 \
  -e OTEL_SERVICE_NAME=checkout \
  -e OTEL_RESOURCE_ATTRIBUTES=service.name=checkout,deployment.environment=prod \
  -e OTEL_PYTHON_AUTO_INSTRUMENTATION_ENABLED=true \
  registry.example.com/checkout:v3.4.1 \
  opentelemetry-instrument python -m checkout

Severity: CONFIGURATION. Container start with the opentelemetry-instrument launcher.

The Python launcher activates auto-instrumentation via the OTEL_PYTHON_AUTO_INSTRUMENTATION_ENABLED flag and the opentelemetry-instrument command. The launcher patches supported libraries at import time.

How to validate it

Five checks, cheapest first.

# Substitute your own value before running:
TRACE_ID=4bf92f3577b34da6a3ce929d0e0e4736

# READ-ONLY: the container exports to the expected endpoint.
docker exec checkout \
  sh -c 'echo $OTEL_EXPORTER_OTLP_ENDPOINT'
# http://host.docker.internal:4317

# READ-ONLY: the SDK has initialised.
docker logs checkout 2>&1 | grep -i 'opentelemetry\|otel'
# OpenTelemetry: Java agent v2.10.0 attached
# OpenTelemetry: default exporter is OTLP

# READ-ONLY: a synthetic span from the SDK.
docker exec checkout \
  curl -fsS http://localhost:8080/healthz

# READ-ONLY: the collector received the spans.
curl -fsS http://host.docker.internal:8889/metrics \
  | grep '^otelcol_exporter_sent_spans'
# otelcol_exporter_sent_spans{exporter="otlp/tempo"} 42

# READ-ONLY: Tempo has the trace.
curl -fsS "http://tempo.internal:3200/api/traces/$TRACE_ID" \
  | jq -r '.resourceSpans[0].scopeSpans[0].spans[0].name'
# "GET /checkout"

A clean validation: the container’s environment points at the right endpoint, the SDK has initialised, a synthetic span is generated, the collector has forwarded it, and Tempo has the trace. Each failure mode below maps to one of these signals failing.

How it can fail

1. The OTLP endpoint points at the wrong host. Cause: OTEL_EXPORTER_OTLP_ENDPOINT is set to a host the container cannot reach (often localhost inside the container’s network namespace). The SDK logs a connection refused on every batch. Detection: otelcol_exporter_send_failed_spans rises; the container’s own logs show OTLP errors.
2. The service name is auto-detected as the binary name. Cause: OTEL_SERVICE_NAME is not set, and the SDK falls back to the binary name or unknown_service. Every container’s spans arrive grouped under the wrong service. Detection: Tempo’s service dropdown shows unknown_service or python3.
3. Spans are lost on container shutdown. Cause: the application does not call the SDK’s shutdown hook on SIGTERM. The batch span processor’s queue is flushed only on graceful shutdown. Detection: a batch job with OTEL_BSP_SCHEDULE_DELAY=5000 and no SIGTERM handler loses up to 5 seconds of spans per exit.
4. Trace context propagation is broken at the load balancer. Cause: the load balancer strips the W3C traceparent header. The trace is broken at the first hop. Detection: every trace in Tempo has a single span from the load balancer’s vantage point; downstream spans arrive as separate traces with no parent.
5. The sidecar collector is down. Cause: the sidecar container has crashed or been restarted. The application cannot reach the sidecar. Detection: the application’s logs show OTLP connection refused.
6. Sampling drops all traces. Cause: the sampler is configured with a ratio of 0 for one environment. The collector receives spans but drops them. Detection: otelcol_processor_tail_sampling_traces_dropped rises without a corresponding rise in otelcol_exporter_sent_spans.

How to troubleshoot it

1. Is the OTLP endpoint reachable from the container? docker exec <container> curl -v $OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces -d '\{\}'. A 200 from the collector confirms the path.
2. Did the SDK initialise? docker logs <container> | grep -i 'opentelemetry\|otel'. A clean initialisation logs the version and the configured endpoint.
3. Are spans reaching the collector? The collector’s /metrics endpoint exposes otelcol_exporter_sent_spans. A non-zero count confirms the path.
4. Are spans reaching Tempo? Tempo’s /api/search returns traces by service name and tags. A query for the service.name returns recent traces if the path is intact.
5. Is trace context propagation intact? Open a request from a known trace and follow the traceparent header through the load balancer to the application. A stripped header produces a broken trace; the fix is at the load balancer.

Security implications

• The collector is an unauthenticated receiver by default. Anyone who can reach port 4317 can send spans. The mitigation is network isolation, not authentication at the receiver.
• Span attributes can carry sensitive values. The application emits attributes such as customer IDs, payment amounts, and database query parameters. Tempo stores the attributes verbatim. Audit the application’s instrumentation.
• Resource attributes reveal topology. The service.namespace and deployment.environment attributes can be inferred from the labels; a careless service name leaks the deployment shape.
• Sidecar collectors share fate with the application. A sidecar that fails restarts with the application; a sidecar that is compromised exposes the application. Treat the sidecar as a privileged neighbour.

Performance implications

• SDK overhead. The auto-instrumentation adds a span per I/O operation. A busy service can generate tens of thousands of spans per second. The cost is CPU on the application container.
• Network cost. OTLP/gRPC compresses spans; OTLP/HTTP does not. For high-volume services, gRPC is materially cheaper.
• Sampling. A 10 percent sampler drops 90 percent of spans at the collector. The cost of storage is paid on the 10 percent that survive. Set the sampling rate based on the budget.

Production guidance

• Choose the agent-on-host pattern for a single-tenant host with a small number of services. The operational cost is one collector per host.
• Choose the sidecar pattern for a multi-tenant host or a host with strict isolation requirements. The cost is one collector per application container.
• Set OTEL_SERVICE_NAME explicitly per container. Never rely on auto-detection.
• Set OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG deliberately. A 10 percent ratio is a reasonable starting point; adjust based on the storage budget.
• Configure a preStop hook or a SIGTERM handler that flushes the batch span processor on shutdown. The default behaviour loses spans.
• Validate the OTLP endpoint at startup. A Prometheus alert on otelcol_exporter_send_failed_spans > 0 catches the common failure mode.

Verification

You should now be able to answer:

• What three things does the OpenTelemetry SDK do inside a container?
• What is the operational difference between agent-on-host and sidecar?
• Why must OTEL_SERVICE_NAME be set explicitly?
• What is the most common cause of lost spans on container shutdown?
• How do you verify that spans are reaching Tempo?

Quiz