Running the OpenTelemetry Collector on a Docker host

The previous lesson showed a twenty-line collector config in passing. That config works on a laptop. In production the collector is a service you operate: it has memory limits, a queue that can fill, a backend that can go away for an hour, and a configuration whose order changes its behaviour. This lesson is about running it.

Why a collector at all

An application can export spans straight to Tempo or Jaeger. Most teams start there and then move to a collector for four reasons:

• Credentials. Twelve services exporting directly need twelve copies of the backend token. One collector needs one.
• Backpressure. When the backend is down, a direct exporter either blocks the application or drops on the floor. A collector absorbs the outage in a queue that lives outside your app.
• Reshaping. Renaming attributes, dropping a noisy span, scrubbing a customer email out of a span attribute — all of this belongs in a pipeline you can change without redeploying twelve services.
• Vendor lock. Changing backends becomes an exporter edit, not a fleet-wide dependency bump.

Three topologies

flowchart LR
  subgraph Agent["Agent per host"]
    A1[svc a] --> AC[collector]
    A2[svc b] --> AC
  end
  subgraph Gateway["Gateway"]
    AC --> GW[gateway collector] --> BE[(Tempo)]
  end

Agent — one collector per Docker host, on the host’s network or on a shared Docker network. Every container on that host exports to it. This is the right default for a standalone Docker host: one process, one config, cheap.

Gateway — a central collector (or a small pool behind a load balancer) that receives from every host’s agent. Add it when you want tail sampling, per-tenant routing, or a single egress point through a firewall.

Sidecar — one collector container per application container. Use it when a single noisy service needs its own queue sizing or its own credentials, not as a default. It multiplies the number of things you operate by the number of services you run.

A production-shaped configuration

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Must be first. It reads its own process memory and starts refusing
  # data before the container hits its cgroup limit and gets OOM-killed.
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100

  # Identify where this telemetry came from. Without this, every span
  # from every host looks the same in the backend.
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: host.name
        value: ${env:HOSTNAME}
        action: upsert

  # Must be last before the exporter. Batching after sampling and
  # filtering means you batch only what you are actually sending.
  batch:
    timeout: 5s
    send_batch_size: 512
    send_batch_max_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/tempo]
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

The processor list is a pipeline, and the order in the list is the order data flows. Two rules follow from that and they are the two that people get wrong:

1. memory_limiter goes first. Its job is to reject incoming data when the collector is close to its memory ceiling. A limiter placed after batch is protecting nothing, because the batcher has already allocated the memory.
2. batch goes last. Anything that drops or filters data should run before batching, so you are not paying to assemble batches you then throw away.

Deploying it

# compose.yaml (excerpt)
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.140.0
    command: ['--config=/etc/otel/collector.yaml']
    volumes:
      - ./otel-collector.yaml:/etc/otel/collector.yaml:ro
    environment:
      HOSTNAME: docker-host-01
    networks: [observability, app]
    mem_limit: 512m
    restart: unless-stopped

Two details matter. The mem_limit must be comfortably above limit_mib + spike_limit_mib from the config — the limiter is a soft brake, the cgroup is a wall, and you want the brake to engage first. And the collector needs to be on the same Docker network as the services exporting to it, otherwise otel-collector:4317 will not resolve.

Resource attributes: the part that decides whether this is useful

A trace without a service.name shows up in every backend as unknown_service. That is the single most common reason a freshly deployed tracing stack is useless on day one.

service.name comes from the application, not the collector — it is set by the SDK, usually from OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES:

services:
  api:
    image: example.com/api:1.4.2
    environment:
      OTEL_SERVICE_NAME: api
      OTEL_RESOURCE_ATTRIBUTES: service.version=1.4.2,deployment.environment=production
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

The collector’s resource processor can add attributes the application cannot know — which host it landed on, which datacentre — but it cannot invent a service name it was never told.

Verifying it is actually working

The collector is a pipeline with three places data can vanish: it was never received, it was dropped in a processor, or the export failed. Its own metrics distinguish all three.

$ curl -s http://localhost:8888/metrics | grep -E 'otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)_spans'

otelcol_receiver_accepted_spans_total{receiver="otlp",transport="grpc"} 18422
otelcol_exporter_sent_spans_total{exporter="otlp/tempo"} 18422
otelcol_exporter_send_failed_spans_total{exporter="otlp/tempo"} 0

Read it as a balance sheet. accepted climbing with sent flat means the exporter is stuck. send_failed climbing means the backend is rejecting or unreachable. accepted flat at zero means nothing is arriving and the problem is upstream — network, endpoint, or port.

The health-check extension answers the cruder question of whether the process is up at all, which is what a Docker HEALTHCHECK should use:

$ curl -fsS http://localhost:13133/

{"status":"Server available","upSince":"2026-08-11T09:14:22.108Z","uptime":"41m18.9s"}

When you need to see the actual spans rather than counts of them, add a debug exporter to the pipeline temporarily:

exporters:
  debug:
    verbosity: detailed

Then put debug alongside otlp/tempo in the pipeline’s exporter list and read docker logs otel-collector. Take it out again afterwards — at detailed verbosity it prints every span, which on a busy service is a second, unbounded logging problem.

A diagnostic order that works

1. Is the collector process up? curl the health-check endpoint.
2. Is anything arriving? otelcol_receiver_accepted_spans_total — if zero, the problem is the application, the network, or the port.
3. Is anything leaving? Compare exporter_sent with exporter_send_failed.
4. Is the backend seeing it under a name you recognise? If everything says unknown_service, the application never set OTEL_SERVICE_NAME.

Knowledge check

Where next

The next lesson deploys Tempo — the backend this collector has been exporting to — and shows what querying stored traces actually looks like.