The Collector Strategy

A team runs 500 production hosts, each with an OpenTelemetry SDK exporting to a single Collector on port 4317. The Collector is a StatefulSet with two replicas behind a load balancer. The backends (Tempo, Loki, Prometheus) live behind the Collector. One morning the load balancer’s idle-connection timeout fires. Every gRPC connection between the SDKs and the Collector is closed by the load balancer; the SDKs retry; the retries swamp the Collector; the Collector starts returning RESOURCE_EXHAUSTED; the SDKs slow down. The fix is not a code change. The fix is a topology choice.

The collector strategy is the answer to two questions: where does the collection-tier process run, and what does it forward to? The answers determine the failure domains, the network topology, and the operational budget.

What the collector strategy is

The collector strategy is the placement and wiring of the OpenTelemetry Collector (or its Grafana distribution, Alloy) in the topology. There are two patterns, and they compose.

Agent mode

In agent mode, the Collector runs one process per host (bare-metal, VM) or per pod (Kubernetes). The agent receives from local sources — the SDK on the same host, the filelog receiver for log files, the hostmetrics receiver for host metrics — and either exports directly to the backends or forwards to a gateway.

The agent is the per-host pattern. The host is the failure boundary: an agent crash affects only the host it runs on.

Gateway mode

In gateway mode, the Collector runs one or more processes per cluster, receiving telemetry from many agents. The gateway is the central pipeline; it batches, filters, and ships to the backends.

The gateway is the central pattern. The gateway is the failure boundary: a gateway outage affects every host behind it.

The composition

The two patterns compose. An agent fans in to a gateway; the gateway fans out to the backends. The agent owns the host-local sources and the per-host batching; the gateway owns the queue, the rate-limiting, the tail sampling, and the fan-out.

    +---------+   +---------+   +---------+
    | Host A  |   | Host B  |   | Host C  |
    | (agent) |   | (agent) |   | (agent) |
    +----+----+   +----+----+   +----+----+
         |             |             |
         +-------------+-------------+
                       |
                       v
               +---------------+
               |   Gateway(s)  |
               | (StatefulSet) |
               +-------+-------+
                       |
         +-------------+-------------+
         |             |             |
    +----v----+  +-----v----+  +-----v----+
    |  Mimir  |  |   Loki   |  |   Tempo  |
    | or Prom |  +----------+  +----------+
    +---------+

The gateway in the diagram is a single logical entity backed by two or more replicas behind a load balancer. The fan-in boundary is the load balancer; the fan-out boundary is each backend’s OTLP receiver.

Why a sysadmin cares

Three failure shapes appear when the strategy is the wrong shape for the fleet.

1. The single Collector that became a single point of failure. A team deploys one Collector as a gateway. The Collector restarts for a config change. The SDKs retry; the application thread pool stalls on the synchronous exporter; the team blames the SDK. The fix is at least two gateway replicas behind a load balancer with sticky gRPC connections and a pre-stop hook that drains the queue.
2. The agent that became a network bottleneck. A team deploys 500 agents that all export directly to the backends. The fan-out is N x M connections (N hosts, M backends). The backends hit their connection limits; connection refused fills the agent logs. The fix is a gateway tier between the agents and the backends.
3. The agent that retried into the gateway during an outage. A gateway outage caused every agent’s sending_queue to fill. When the gateway returned, every agent drained at once. The gateway hit its max_recv_msg_size limit; new records were rejected. The fix is back-off jitter on the agent’s sending_queue and a per-consumer limit on the gateway.

The right strategy is the one that places the failure domain at the right boundary. A 5-host fleet does not need a gateway; a 5,000-host fleet does. The shape that is right for a fleet of 50 hosts is a judgement call, and the rule of thumb is below.

How it works

Each OTel Collector is a pipeline of receivers, processors, and exporters. The strategy is the placement of the pipeline processes.

Agent mode in detail

The agent runs alongside the workload it observes. In Kubernetes this is typically a sidecar or a DaemonSet. The agent receives on localhost:4317 (gRPC) and localhost:4318 (HTTP) and forwards to the gateway on the cluster network.

The agent’s processors are typically limited to resource detection, batching, and redaction. Tail sampling and rate-limiting belong on the gateway, not on the agent, because those decisions require fleet-wide context.

The agent’s exporters are typically one: an otlp exporter to the gateway. The agent does not write to the backends directly in a gateway-tier topology.

Gateway mode in detail

The gateway runs centrally. It receives on the cluster network port (typically 4317 gRPC and 4318 HTTP) and forwards to the backends.

The gateway’s processors carry the fleet-wide concerns: tail-based sampling, drop policies, rate limiting, attribute redaction, and the final batch before the fan-out. The gateway’s exporters are typically one per backend, one per signal type, with sending_queue and file_storage to absorb backend outages.

Decision: agent-only, gateway-only, or both

The decision is a function of fleet size, network shape, and failure tolerance.

• Agent-only (no gateway). Appropriate for a small fleet (less than 25 hosts), a single-VM dev environment, or a single-namespace Kubernetes cluster. Each agent exports directly to the backends over the cluster network. The topology is flat; the failure domain is the agent.
• Gateway-only (no agent). Appropriate when the workloads cannot or should not run a Collector. Embedded devices, serverless functions, and Edge nodes are the canonical cases. The SDK exports to a central gateway over the WAN; the gateway fans out. The failure domain is the gateway.
• Both (the canonical pattern). Appropriate for any production fleet of 50+ hosts. Agents absorb host-local failures and queue during gateway outages; the gateway absorbs backend outages and provides the fleet-wide processing pipeline.

The split is not symmetrical. A team can run an agent-only fleet and a gateway-only fleet in the same organisation without conflict. The decision is per-environment, and the topology document should record which pattern each environment uses.

How to configure it

The two patterns share most of their configuration. The difference is the receiver and exporter wiring.

# /etc/otelcol/agent.yaml  -- per-host agent
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 127.0.0.1:4317 }
      http: { endpoint: 127.0.0.1:4318 }
  hostmetrics:
    collection_interval: 15s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}
  filelog:
    include: [/var/log/app/*.log]
    operators:
      - type: json_parser

processors:
  batch: { timeout: 5s, send_batch_size: 8192 }
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  resourcedetection:
    detectors: [system, env]

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: false
      cert_file: /etc/otelcol/certs/client.crt
      key_file: /etc/otelcol/certs/client.key
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      storage: file_storage

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

service:
  extensions: [file_storage]
  pipelines:
    metrics: { receivers: [otlp, hostmetrics], processors: [memory_limiter, batch, resourcedetection], exporters: [otlp/gateway] }
    traces:  { receivers: [otlp], processors: [memory_limiter, batch, resourcedetection], exporters: [otlp/gateway] }
    logs:    { receivers: [otlp, filelog], processors: [memory_limiter, batch, resourcedetection], exporters: [otlp/gateway] }

The agent’s memory_limiter rejects new data before the queue grows without bound; the sending_queue with file_storage absorbs gateway outages of minutes, not seconds.

# /etc/otelcol/gateway.yaml  -- central gateway
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch: { timeout: 5s, send_batch_size: 16384 }
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 500 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  attributes/pii:
    actions:
      - key: user.email
        action: delete
      - key: http.request.header.authorization
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls: { insecure: false }
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 20000
      storage: file_storage
  otlp/loki:
    endpoint: loki.observability.svc:3100
    tls: { insecure: false }
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces: { receivers: [otlp], processors: [memory_limiter, tail_sampling, attributes/pii, batch], exporters: [otlp/tempo] }
    logs:   { receivers: [otlp], processors: [memory_limiter, attributes/pii, batch], exporters: [otlp/loki] }
    metrics:{ receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }

The gateway owns tail sampling (which requires fleet-wide context), PII redaction (which is a one-place-to-rule-them-all policy), and the per-backend sending queues (which absorb backend outages independently).

How to validate it

# READ-ONLY: the agent is up and bound.
curl -fsS http://127.0.0.1:8888/metrics | grep otelcol_process_uptime
# otelcol_process_uptime 4217

# CONFIGURATION: the agent config is valid.
otelcol-contrib validate --config=/etc/otelcol/agent.yaml

# READ-ONLY: the gateway is reachable from the agent.
ssh host-a 'curl -fsS http://otel-gateway:13133/ 2>&1 || echo refused'
# {"status":"Server available","upSince":"2026-08-13T08:22:04Z"}

# READ-ONLY: traces reach the gateway.
ssh host-a 'curl -fsS http://127.0.0.1:8888/metrics | grep exporter_sent_spans'
# exporter_sent_spans{ exporter="otlp/gateway" } 4218

# READ-ONLY: traces reach Tempo via the gateway.
curl -fsS 'http://tempo:3200/api/search?tags=service.name=checkout&limit=1' | jq '.traces | length'
# 14

# CONFIGURATION: reload each tier without losing the queue.
ssh host-a 'systemctl reload otelcol'
ssh gateway-host 'systemctl reload otelcol'

A clean validation: the agent’s exporter_sent_spans is monotonically increasing, the gateway’s receiver_accepted_spans matches the sum of agents, and the backend’s search returns recent traces. The memory_limiter metric (otelcol_processor_accepted/refused) should show refusals as zero except during overload.

How it can fail

The most expensive collector-strategy failure modes from real incidents.

1. Single gateway replica. The gateway StatefulSet has one replica for “simplicity.” The replica OOMs during a deploy. Every agent’s sending queue fills to its disk limit; agents start dropping at the memory_limiter. Symptom: a 30-minute telemetry blackout for traces while metrics and logs continue (they have different paths).
2. Load balancer closing long-lived gRPC. The load balancer idle timeout is 60s; the gRPC keepalive is 90s. The LB closes the connection; the agent retries. Symptom: a sawtooth in otelcol_exporter_sent_failed correlated with the LB timeout, increasing tail latency on every span export.
3. Tail sampling on the agent. A team puts tail sampling on each agent with a 5% probabilistic policy. Every agent decides independently; the result is 5% per agent, not 5% fleet-wide. Symptom: tail-sampled trace rates look 100x what they should; storage cost explodes.
4. Gateway sends to backends over the public network. A gateway is deployed in the wrong VPC and reaches Tempo over a public hostname with self-signed certificates. Symptom: intermittent TLS failures during certificate rotation; authentication latency dominates the fan-out.
5. Queue persistence without disk. An agent has sending_queue enabled but no file_storage extension configured. The “queue” is in memory. A restart loses queued data. Symptom: gaps in every graph that fall on agent restart timestamps.
6. Cardinality label added at the agent. A team adds a host.ip label at the agent. Every series in the backend multiplies by the host count. Symptom: backend ingest latency rises; rule evaluation times out; alerts evaluate late.

How to troubleshoot it

The diagnostic order is “is each tier up?”, “is each tier forwarding?”, “is the backend accepting?”, and the answer reveals which tier is at fault.

1. Start at the agent. curl http://127.0.0.1:8888/metrics and look at otelcol_exporter_sent_spans / _sent_metrics. If the metric is flat, the agent is not exporting; the problem is at the agent.
2. Move to the gateway. From the gateway host, curl http://localhost:8889/metrics | grep receiver_accepted_spans. If the gateway is not receiving what the agent sent, the network is the problem.
3. Move to the backend. From the gateway host, curl http://tempo:3200/ready. If Tempo is not ready, the backend is the problem; the gateway queue absorbs it.
4. Check the queues. otelcol_exporter_queue_size on each tier tells you how full the queue is. A persistently full queue means the next tier is the bottleneck; a queue that drains means the next tier is healthy.
5. Reproduce at the lowest tier first. A missing trace: confirm the application SDK is configured to export (OTEL_TRACES_EXPORTER=otlp), then confirm the agent’s OTLP listener is bound, then confirm the gateway can be reached, then confirm Tempo accepts the export. Walk up from the bottom.

Security implications

• The agent is on every host. A vulnerability in the agent is a vulnerability on every host. Pin the agent version and update on the same cadence as the OS.
• The gateway is on the cluster network. A compromised gateway can inject telemetry into the backends with arbitrary labels and arbitrary trace IDs. mTLS between agents and the gateway, and between the gateway and the backends, is the baseline; bearer tokens are not a substitute for mTLS.
• PII redaction belongs on the gateway. A field removed at the gateway is gone from every backend; a field removed at every agent is a constant maintenance burden.
• Secrets live in a vault. The agent’s TLS client certificate and the gateway’s mTLS certificate come from a secret manager (HashiCorp Vault, the cloud provider’s secret store). A certificate rotated by hand is a certificate that forgets to be rotated.

Performance implications

• Agent CPU is per host. A 4-core agent can saturate at 10,000 spans/s. Beyond that, the agent queue fills; drop metrics climb; SDKs slow down. The agent should be sized to the highest expected host span-rate with 50% headroom.
• Gateway memory is the bottleneck. Tail sampling requires the gateway to retain in-flight traces for decision_wait seconds. With num_traces: 50000 and 10 s decision wait, the gateway memory budget is in the multi-GB range. The right memory_limiter is the right gateway size.
• Network bandwidth at the fan-out. The gateway fans N signal types to M backends. A gateway that fronts a 500-host fleet with three backends has 1500 outbound connections. The TLS handshake cost dominates startup; keepalive must be tuned to the LB idle timeout.

Production guidance

• Agent on every host. Gateway central. This is the canonical pattern and the right default for any fleet larger than 50 hosts.
• Tune the agent’s queue first, the gateway’s second. The agent queue absorbs gateway outages; the gateway queue absorbs backend outages. Without a queue on the agent, a gateway restart becomes a fleet-wide data gap.
• Pre-stop hooks on the gateway. A graceful shutdown of the gateway drains the in-memory queue to file_storage before the process exits. Without the hook, a rolling restart loses the queued traces.
• mTLS everywhere. The agent-to-gateway link and the gateway-to-backend link are both mTLS. Bearer tokens are for development.

Verification

You should now be able to answer:

• What does the agent own that the gateway should not, and vice versa?
• Why is two-tier queueing (agent queue + gateway queue) necessary for survival of a backend outage?
• Which fleet sizes warrant a gateway, and which do not?
• Where in the topology does tail sampling belong, and why?

Quiz