Collector Deployment Patterns

A platform team rolls out the OpenTelemetry Collector to a 200-host fleet. They choose the agent topology: one collector per host. Six months later, the gateway is added at the cluster edge because the team needs central transformation that the agents cannot do without duplicating configuration. The migration is two weeks of work because the per-host systemd units were never designed to be replaced.

This lesson is the deployment shape of the collector: how it runs on a host (systemd, Docker, docker-compose), how it runs in Kubernetes (DaemonSet, sidecar), and how the readiness probe tells the supervisor whether the collector is doing what it is supposed to do.

What it is

The deployment pattern is the runtime shape of the collector: how many processes run, where they run, and how they are managed. The collector ships two operational modes that compose.

• Agent mode — one collector process per host or per pod. The agent accepts telemetry from local sources (OTLP from applications, filelog from /var/log, journald from systemd), batches it, and forwards it to a backend (or to a gateway). Agent mode is the on-host pattern.
• Gateway mode — one or more collector processes per cluster, receiving from many agents. The gateway is the central pipeline; it batches, filters, samples, and ships to the backends. Gateway mode is the multi-tenant pattern.

   Agent (host)             Agent (host)             Agent (host)
   otlp :4317               otlp :4317               otlp :4317
   filelog                  filelog                  filelog
        \                        |                        /
         \                       |                       /
          +--- OTLP/HTTP --- Gateway (cluster) --- OTLP/HTTP ---+
                                     |
                              +------+------+------+
                              |      |      |
                              v      v      v
                            Loki  Tempo  Mimir

The two modes compose. An agent fans in to a gateway; the gateway fans out to the backends. The gateway is a single point of failure unless it is deployed as a StatefulSet with at least two replicas behind a load balancer.

Why a sysadmin cares

The deployment shape is the operational signature of the fleet. Three properties make it so.

1. Failure isolation. An agent that fails loses the telemetry from one host; a gateway that fails loses the telemetry from the whole fleet. The two failure shapes have different blast radii and different response playbooks.
2. Configuration surface. An agent has a per-host configuration; a gateway has a single configuration for the whole fleet. The cost of a config change is the cost of the deploy surface. A change to a gateway hits every host that forwards to it.
3. Resource ceiling. The collector on a 4 GiB host cannot run a tail_sampling processor on 50,000 traces per second. The right topology is the topology that puts the expensive work on the side that has the memory.

The wrong topology for the failure shape is the wrong answer.

How it works

Agent mode on a host

The agent runs as a systemd service. The systemd unit declares the restart policy, the resource limits, and the environment.

# /etc/systemd/system/otelcol.service

[Unit]
Description=OpenTelemetry Collector
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=otelcol
Group=otelcol
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol/config.yaml
Restart=always
RestartSec=5s

# Resource ceiling: 512 MiB RAM, 500 millicores CPU.
MemoryMax=512M
CPUQuota=50%

# Hardening: no new privileges, private /tmp, private network.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/otelcol

# Capabilities: none required for the agent; bind to
# privileged ports is not needed.
CapabilityBoundingSet=

[Install]
WantedBy=multi-user.target

The agent listens on localhost:4317 and localhost:4318 for OTLP; the local applications connect to the agent. The ReadWritePaths directive limits the writable filesystem to the storage path used by the file_storage extension.

Agent mode in Docker

The agent runs as a Docker container. The image is the official otel/opentelemetry-collector-contrib. The configuration is mounted as a read-only bind mount.

# Dockerfile for the agent
FROM otel/opentelemetry-collector-contrib:0.110.0
COPY config.yaml /etc/otelcol/config.yaml
USER 10001:10001
EXPOSE 4317 4318 8888 13133

# SERVICE-IMPACT: run the agent as a background process.
docker run -d \
  --name otelcol \
  --restart unless-stopped \
  --network host \
  --memory 512m \
  --cpus 0.5 \
  --read-only \
  --tmpfs /var/lib/otelcol:rw,size=64m \
  -v /etc/otelcol/config.yaml:/etc/otelcol/config.yaml:ro \
  -v /var/log/app:/var/log/app:ro \
  otelcol:latest

The --network host is the simplest deployment but the most permissive; for production, use a user-defined bridge network and expose only the ports the local applications need.

Docker Compose

The collector on a single host or a small fleet is well-suited to docker-compose. The compose file declares the collector and its volume mounts; the host’s log paths and config path are bind-mounted into the container.

# docker-compose.yaml

services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.110.0
    command: ["--config=/etc/otelcol/config.yaml"]
    restart: unless-stopped
    network_mode: host
    read_only: true
    tmpfs:
      - /var/lib/otelcol:rw,size=64m
    volumes:
      - ./config.yaml:/etc/otelcol/config.yaml:ro
      - /var/log/app:/var/log/app:ro
      - /var/log/journal:/var/log/journal:ro
    environment:
      - LOKI_BASIC_AUTH=${LOKI_BASIC_AUTH}
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '0.5'
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider",
             "http://localhost:13133/"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 10s

The healthcheck calls the health_check extension on localhost:13133. The compose file is the right place for the bind mounts and the environment; the Dockerfile is the right place for the user, the image, and the exposed ports.

Gateway mode

The gateway runs as a small number of replicas behind a load balancer. The agent fans in to the gateway over OTLP; the gateway fans out to the backends.

# docker-compose.yaml for the gateway

services:
  otelcol-gateway:
    image: otel/opentelemetry-collector-contrib:0.110.0
    command: ["--config=/etc/otelcol/config.yaml"]
    restart: unless-stopped
    ports:
      - "4317:4317"   # OTLP gRPC from agents
      - "4318:4318"   # OTLP HTTP from agents
    read_only: true
    tmpfs:
      - /var/lib/otelcol:rw,size=256m
    volumes:
      - ./gateway.yaml:/etc/otelcol/config.yaml:ro
    environment:
      - LOKI_BASIC_AUTH=${LOKI_BASIC_AUTH}
      - MIMIR_TOKEN=${MIMIR_TOKEN}
    deploy:
      replicas: 2
      resources:
        limits:
          memory: 2G
          cpus: '1.0'
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider",
             "http://localhost:13133/"]
      interval: 10s
      timeout: 3s
      retries: 3

The gateway has a different resource ceiling than the agent because the gateway holds the queues and the transformations. The right sizing is the sizing that matches the per-second volume and the worst-case backend outage.

Kubernetes

The Kubernetes pattern is a DaemonSet for the agent and a Deployment (with at least two replicas) for the gateway. The patterns are out of scope for this lesson; the discipline is the same.

• Agent DaemonSet. One collector per node. The agent receives from local sources; the local sources forward to the agent via the OTLP SDK or via the filelog receiver.
• Gateway Deployment. Two or more collectors behind a Service. The agents forward to the gateway via OTLP; the gateway fans out to the backends.

The Kubernetes configuration is owned by the platform team; the operator’s job is to hand off the configuration and the resource limits.

How to configure it

A complete annotated otelcol.yaml for an agent that fans in to a gateway.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  filelog:
    include:
      - /var/log/app/*.log
    start_at: end

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
      - key: host.name
        from_attribute: host.name
        action: insert
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlp/gateway:
    endpoint: otelcol-gateway.internal.example.com:4317
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    headers:
      X-Scope-OrgID: prod
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

extensions:
  health_check:
    endpoint: localhost:13133
  file_storage:
    directory: /var/lib/otelcol/storage
    timeout: 10s

service:
  extensions: [health_check, file_storage]
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/gateway]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/gateway]
    logs:
      receivers:  [otlp, filelog]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/gateway]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info

The agent ships everything to the gateway over OTLP. The gateway is the place that fans out to Loki, Tempo, and Mimir; the agent is the place that buffers for backend outages and applies per-host labels.

How to validate it

Validation is a parse-check plus a runtime check of the collector self-metrics and the health endpoint.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm the health endpoint is up.
curl -s http://localhost:13133/ -w "%{http_code}\n"

200

# READ-ONLY: confirm the receivers are accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="otlp"} 421
otelcol_receiver_accepted_log_records{receiver="filelog"} 1872

# READ-ONLY: confirm the agent is shipping to the gateway.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="otlp/gateway"} 2293

# READ-ONLY: confirm the process is running.
systemctl is-active otelcol

active

A complete deployment validation has four checks: the configuration parses, the process is running, the health endpoint returns 200, and the self-metrics show data flowing from receivers through processors to exporters. A failure at any of the four checks is the place to start the diagnostic.

How it can fail

Six failure modes specific to deployment.

1. The systemd unit that ignored SIGHUP. A collector process that ignores SIGHUP (some container distributions do). Symptom: systemctl reload otelcol succeeds; the agent log shows no reload entry; the running config is unchanged. The fix is systemctl restart otelcol.
2. The Docker container that filled the read-only root filesystem. The filelog receiver’s file_storage extension writes to /var/lib/otelcol/storage. The container was started without a tmpfs for that path. Symptom: the container restarts; the agent log shows read-only-filesystem errors; the receiver never recovers.
3. The docker-compose that exposed the debug endpoint. The compose file did not restrict the 8889 port; the collector’s /debug endpoint is reachable from the cluster network. Symptom: an attacker reads the in-flight pdata; the security audit flags the exposure.
4. The gateway that ran as a single replica. The docker-compose file declared replicas: 1. Symptom: a routine restart drops every in-flight batch; the agents back-pressure; the applications retry.
5. The Kubernetes DaemonSet that scheduled on control-plane nodes. The DaemonSet had no nodeSelector. Symptom: the collector runs on the control-plane nodes; it scrapes the kubelet metrics and the serviceaccount/token volume; the security audit flags the over-scope.
6. The readiness probe that polled the wrong endpoint. The compose healthcheck polled the metrics endpoint on localhost:8888. The metrics endpoint does not return a process-health signal; it returns Prometheus metrics text. Symptom: the probe returns 200 only on the format of the response; the supervisor never sees a Start failure.

How to troubleshoot it

When the deployment is not behaving, the diagnostic order matters.

1. Confirm the process is running. systemctl is-active otelcol or docker inspect --format '\{\{.State.Running\}\}' otelcol. A stopped process is a Start failure; read the agent log.
2. Confirm the health endpoint is up. curl http://localhost:13133/. A non-200 response means the collector is starting or has a Start failure.
3. Confirm the receivers are bound. ss -tlnp. A receiver on localhost is invisible from outside the host; a receiver on 0.0.0.0 is visible but may be blocked by the firewall.
4. Confirm the exporters are reaching the backend. Check the backend’s own metrics; check the tenant; check the TLS chain.
5. Tail the agent log. The first error after a Start or a reload is usually the only one.
6. Check the resource limits. MemoryMax in the systemd unit, memory: 512M in the compose file, resources.limits in the Kubernetes manifest. A collector that is killed by the OOM killer is a resource-limit problem, not a code problem.

Security implications

The deployment shape has security implications.

• Bind to localhost in agent mode. The OTLP, Zipkin, and Jaeger receivers default to 0.0.0.0. In agent mode, change the bind to localhost. In gateway mode, restrict the listener with a NetworkPolicy.
• Run as a dedicated user. The systemd unit and the Docker container both run as a non-root user. The user has read access to the intended paths and no more.
• Read-only root filesystem. The Docker container runs with --read-only and a tmpfs for the storage path. The collector cannot write outside the storage path.
• Drop capabilities. The systemd unit and the Docker container drop all capabilities. The collector does not need any to bind to unprivileged ports or to read the filesystem.
• Restrict the debug endpoint. The pprof and zpages extensions bind to localhost by default. Expose them on the cluster network only with authentication.

Performance implications

The deployment shape has performance implications.

• Agent sizing. A modest agent (10 hosts, 1000 log lines per second) needs 256 MiB RAM and 100 millicores CPU. A busy agent (200 hosts, 100,000 log lines per second) needs 512 MiB RAM and 500 millicores CPU.
• Gateway sizing. A modest gateway (10 agents, 10,000 log lines per second) needs 512 MiB RAM and 250 millicores CPU. A busy gateway (200 agents, 1,000,000 log lines per second) needs 4 GiB RAM and 2 cores CPU.
• Queue depth. The sending_queue on the agent’s exporter to the gateway bounds the data on a gateway outage. A queue of 5000 entries at 1 KiB per entry holds 5 MiB; a queue of 50,000 entries holds 50 MiB.
• Batching. The batch processor on the agent coalesces for the gateway; the batch processor on the gateway coalesces for the backends. The right batching on the agent is small (timeout: 1s, send_batch_size: 1024); the right batching on the gateway is large (timeout: 5s, send_batch_size: 8192).

Production guidance

• Bind to localhost in agent mode. The default 0.0.0.0 is too permissive for a production host.
• Run as a dedicated user. Never run as root.
• Read-only root filesystem. The Docker container runs with --read-only and a tmpfs for the storage path.
• Wire the readiness probe to the health endpoint. The metrics endpoint does not return a process-health signal.
• Smoke test after every config change. Ship a known record with a unique UUID and confirm it arrives in the right backend within ten seconds.

Verification

You should now be able to answer:

• What is the difference between agent mode and gateway mode, and how do they compose?
• Where should the memory_limiter and the batch processors sit in the chain on the agent and on the gateway?
• What endpoint should the readiness probe poll, and why is the metrics endpoint the wrong choice?
• What does a non-zero otelcol_receiver_refused_* counter tell you, and what is the first diagnostic step?
• Why is SIGHUP not sufficient for a config change on a containerised collector?

Quiz