Collector vs Agent Patterns

A team runs 500 production hosts, each with an OTel SDK exporting to a single Collector on port 4317. The Collector is a StatefulSet with two replicas behind a load balancer. The backends (Tempo, Mimir, Loki) live behind the Collector. One morning the load balancer’s idle-connection timeout fires. Every gRPC connection between the SDKs and the Collector is closed by the load balancer; the SDKs retry; the retries swamp the Collector; the Collector starts returning RESOURCE_EXHAUSTED; the SDKs slow down. The fix is not a code change — it is a topology choice.

OpenTelemetry supports two patterns for collector deployment: agent (per-host or per-pod) and gateway (central). The patterns can compose: an agent fans in to a gateway; the gateway fans out to the backends. The lesson that follows names each pattern, names the failure domain each one owns, and walks the right choice for a given fleet.

What it is

The OTel Collector deployment mode is a topology decision, not a configuration choice. The same binary runs in either mode; the difference is the receiver / exporter wiring and the placement of the process.

Agent mode

In agent mode, the OTel Collector runs one process per host (bare-metal / VM) or per pod (Kubernetes). The agent receives telemetry from local sources — the SDK on the same host, the filelog receiver for log files, the hostmetrics receiver for host metrics — and either exports directly to the backends or forwards to a gateway.

Agent mode is the per-host pattern. The host is the failure boundary: an agent crash affects only the host it runs on.

Gateway mode

In gateway mode, the OTel Collector runs one or more processes per cluster, receiving telemetry from many agents. The gateway is the central pipeline; it batches, filters, and ships to the backends.

Gateway mode is the central pattern. The gateway is the failure boundary: a gateway outage affects every host behind it.

The composition

The two patterns can compose. An agent fans in to a gateway; the gateway fans out to the backends. The agent owns the host-local sources; the gateway owns the queue, the rate- limiting, the tail sampling, and the fan-out.

   +---------+   +---------+   +---------+
   | Host A  |   | Host B  |   | Host C  |
   | (agent) |   | (agent) |   | (agent) |
   +----+----+   +----+----+   +----+----+
        |             |             |
        +-------------+-------------+
                      |
                      v
              +---------------+
              |   Gateway(s)  |
              | (StatefulSet) |
              +-------+-------+
                      |
        +-------------+-------------+
        |             |             |
   +----v----+  +-----v----+  +-----v----+
   |  Mimir  |  |   Loki   |  |   Tempo  |
   +---------+  +----------+  +----------+

The gateway in the diagram is a single logical entity backed by two or more replicas behind a load balancer. The fan-in boundary is the load balancer; the fan-out boundary is the backend’s OTLP receiver.

Why a sysadmin cares

Three failure shapes appear when the topology is the wrong shape for the fleet.

1. The single Collector that became a single point of failure. A team deploys one Collector as a gateway. The Collector restarts for a config change. The SDKs retry; the application thread pool stalls; the team blames the SDK. The fix is at least two gateway replicas behind a load balancer with sticky gRPC connections.
2. The agent that became a network bottleneck. A team deploys 500 agents that all export directly to the backends. The fan-out is N x M connections (N hosts, M backends). The backends hit their connection limits. The fix is a gateway tier between the agents and the backends.
3. The agent that retried into the gateway during an outage. A gateway outage caused every agent’s sending_queue to fill. When the gateway returned, every agent drained at once. The gateway hit its max_recv_msg_size limit; new records were rejected. The fix is back-off jitter on the agent’s sending_queue and a per-consumer limit on the gateway.

The right topology is the one that places the failure domain at the right boundary. A 5-host fleet does not need a gateway; a 5,000-host fleet does.

How it works

The mental model. Each OTel Collector is a pipeline (receivers, processors, exporters). The topology is the placement of the pipeline processes.

Agent mode in detail

The agent runs alongside the workload it observes. In Kubernetes this is typically a sidecar or a DaemonSet. The agent receives on localhost:4317 (gRPC) and localhost:4318 (HTTP) and forwards to the gateway on the cluster network.

The agent’s processors are typically limited to resource detection, batching, and redaction. Tail sampling and rate-limiting belong on the gateway, not on the agent, because those decisions require fleet-wide context.

The agent’s exporters are typically one: an otlp exporter to the gateway. The agent does not write to the backends directly in a gateway-tier topology.

Gateway mode in detail

The gateway runs centrally. It receives on the cluster network port (typically 4317 gRPC and 4318 HTTP) and forwards to the backends.

The gateway’s processors carry the fleet-wide concerns: tail-based sampling, drop policies, rate limiting, attribute redaction, and the final batch before the fan-out. The gateway’s exporters are typically one per backend, one per signal type, with sending_queue and file_storage to survive downstream outages.

The gateway is the single fan-in point in the topology. It is also the single point of failure unless it is deployed as a StatefulSet with at least two replicas behind a load balancer that supports sticky gRPC connections.

Topology trade-offs

   Direct (no gateway)             Agent + gateway
   --------------------            --------------------
   +------+   +------+             +------+   +------+
   | host |---| Mimir|             | host |---| agent|
   +------+   +------+             +------+   +------+
   +------+   +------+             +------+   +------+
   | host |---| Loki |             | host |---| agent|
   +------+   +------+             +------+   +------+
   +------+   +------+                    |
   | host |---| Tempo|                    v
   +------+   +------+              +---------+
                                    | gateway |
   blast radius: a single host      +---------+
   cost per host: direct conn            |
   config drift: per-host           +----+----+----+
                                    | Mimir Loki Tempo
                                    |
   blast radius: the gateway
   cost per host: 1 conn
   config drift: centralised at gateway

The threshold is not hard. A 50-host fleet running three backends with no tail sampling can run direct. A 200-host fleet with tail sampling needs the gateway tier to centralise the policy.

How to configure it

Two configurations: the agent (per-host) and the gateway (central). Both ship the same binary; the difference is the receiver / exporter wiring.

Agent (per-host)

# /etc/otelcol/agent.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      disk:
      filesystem:
      network:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors: [system, env]
    timeout: 2s
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/gateway:
    endpoint: otelcol-gateway.observability.svc:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 1000
      retry_initial_interval: 5s
      retry_max_interval: 30s
      retry_max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info
  pipelines:
    metrics:
      receivers:  [otlp, hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/gateway]
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/gateway]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters:  [otlp/gateway]

The agent binds to localhost so the SDK can reach it without exposing the OTLP receiver to the network. The agent’s only exporter is the gateway. The sending_queue has back-off to absorb gateway restarts.

Gateway (central)

# /etc/otelcol/gateway.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size: 16777216
        max_concurrent_streams: 100
        keepalive:
          server_parameters:
            min_time_between_pings: 10s
            permit_without_stream: true
        tls:
          cert_file: /etc/otelcol/tls/server.crt
          key_file: /etc/otelcol/tls/server.key
          client_ca_file: /etc/otelcol/tls/ca.crt
      http:
        endpoint: 0.0.0.0:4318
        max_request_size: 16777216

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # Tail sampling happens on the gateway; agents don't have
  # the fleet-wide context.
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/mimir:
    endpoint: mimir.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      retry_initial_interval: 5s
      retry_max_interval: 30s
      retry_max_elapsed_time: 300s
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/ca.pem
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
  otlp/loki:
    endpoint: https://loki.internal.example.com/otlp
    headers:
      X-Scope-OrgID: prod
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    timeout: 10s

service:
  extensions: [file_storage/queue]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info
  pipelines:
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/mimir]
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [otlp/tempo]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/loki]

The gateway binds on the cluster network with mTLS. The tail_sampling processor runs on the traces pipeline only — agents do not have the fleet-wide context needed for the decision. Each backend exporter has a sending_queue with back-off.

How to validate it

Validate end-to-end from the SDK through the agent through the gateway to the backend.

# CONFIGURATION: parse-check both configs.
otelcol validate --config=/etc/otelcol/agent.yaml
otelcol validate --config=/etc/otelcol/gateway.yaml

# READ-ONLY: confirm the agent is accepting on localhost.
curl -s http://agent-host:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_metric_points{receiver="otlp",signal="metrics"} 4096
otelcol_receiver_accepted_spans{receiver="otlp",signal="traces"} 1024

# READ-ONLY: confirm the agent is shipping to the gateway.
curl -s http://agent-host:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_metric_points{exporter="otlp/gateway"} 4096
otelcol_exporter_sent_spans{exporter="otlp/gateway"} 1024

# READ-ONLY: confirm the gateway is receiving from the agents.
kubectl exec otelcol-gateway-0 -- curl -s http://localhost:8888/metrics | \
  grep otelcol_receiver_accepted

otelcol_receiver_accepted_spans{receiver="otlp",signal="traces"} 1024

# READ-ONLY: confirm the gateway is exporting to the backends.
kubectl exec otelcol-gateway-0 -- curl -s http://localhost:8888/metrics | \
  grep otelcol_exporter_sent

otelcol_exporter_sent_spans{exporter="otlp/tempo"} 1024
otelcol_exporter_sent_metric_points{exporter="otlp/mimir"} 4096

A discrepancy between the agent’s exporter counter and the gateway’s receiver counter means the gateway is dropping or the load balancer is mis-routing. A discrepancy between the gateway’s exporter counter and the destination backend’s span count means the backend is rejecting.

How it can fail

Five failure modes that arise from topology choices.

1. The single gateway that became a single point of failure. The team deploys one gateway replica. The gateway restarts for a config change; the SDKs retry; the agents’ queues fill. Symptom: otelcol_exporter_failed_* spikes on every agent; the application’s thread pool stalls. The fix is at least two gateway replicas behind a load balancer with connection draining.
2. The agent that retried into the gateway during an outage. A gateway outage caused every agent’s sending_queue to fill. When the gateway returned, every agent drained at once. Symptom: the gateway’s max_recv_msg_size was hit; new records were rejected. The fix is retry_max_elapsed_time to bound the total retry window and retry_max_interval to spread the retries.
3. The fan-out that overwhelmed the backends. 500 agents export directly to the backends; the backends hit their connection limits. Symptom: backend receiver logs show too many open connections. The fix is a gateway tier between the agents and the backends.
4. The mixed-mode configuration drift. The team runs agents in some clusters and direct exports in others. The resource attributes differ between the two topologies. The dashboards that group by service.name see different shapes per cluster. Symptom: every dashboard sums to a different total per cluster. The fix is to standardise on one topology per fleet or to reconcile the resource attributes in the Collector.
5. The tail sampling that ran on the agent. The team configures tail sampling on the agent to save bandwidth. The agent sees only its own traces, not the fleet-wide trace. Tail sampling requires a fleet-wide view. Symptom: tail sampling decisions are wrong; traces that should be kept are dropped. The fix is to move tail sampling to the gateway.

How to troubleshoot it

When the topology is not behaving as expected, the diagnostic order is from the SDK forward.

1. Is the SDK exporting? otelcol_receiver_accepted_* on the agent. Zero means the SDK is not configured for the expected signal type.
2. Is the agent shipping? otelcol_exporter_sent_* on the agent. Zero with non-zero receivers means the agent’s gateway endpoint is wrong or unreachable.
3. Is the gateway receiving? otelcol_receiver_accepted_* on the gateway. Zero with non-zero agent exporters means the load balancer is dropping the connections.
4. Is the gateway exporting? otelcol_exporter_sent_* on the gateway. Zero with non-zero receivers means a backend is unreachable.
5. Is the backend ingesting? The backend’s own metrics (e.g. cortex_distributor_received_samples_total). Zero with non-zero gateway exporters means the backend is rejecting the batch.

Security implications

The topology crosses three trust boundaries: SDK to agent, agent to gateway, gateway to backend. Each crossing is a chance for exposure.

• SDK to agent. In agent mode, the SDK exports to localhost on a loopback interface. No TLS is needed at the SDK-agent boundary because the network is local. In a sidecar topology, the boundary is a shared network namespace; the same loopback assumption applies.
• Agent to gateway. In gateway mode, the agent exports to the cluster network. mTLS is the right answer; the tls.client_ca_file on the receiver enforces it.
• Gateway to backend. The gateway exports to the backends over the cluster network or over a private link. TLS with a CA bundle is mandatory; the bundle must be rotated before the receiver’s certificate expires.
• The LB boundary. A load balancer in front of the gateway may terminate TLS or may pass through. The decision depends on the LB. The discipline is to enforce mTLS end-to-end when possible.

Performance implications

The topology determines where the cost is paid.

• Agent memory and CPU. Each agent carries the memory_limiter, the batch processor, and the OTLP exporter. 100-150 MiB RAM and 50-100 millicores CPU at modest line rates. Linear with the line rate of the host.
• Gateway memory and CPU. Each gateway carries the same processors plus the tail sampling decision logic. The tail sampling processor is the dominant cost; it holds a fraction of in-flight traces in memory while it waits for the decision window. 2-4 GiB RAM is a typical starting point for a gateway serving 1000 spans per second.
• Network. The agent-gateway hop adds a network round trip. On a fast LAN, this is negligible. On a metered link or a WAN, compression and batching become the optimisation surface.
• Disk queue. The gateway’s file_storage extension persists the sending queue to disk. SSD-backed storage is appropriate.

Production guidance

• Pick one topology per fleet. Mixed topologies cause resource-attribute drift. Standardise on agent-plus-gateway for fleets above 100 hosts; standardise on direct for fleets below 10 hosts.
• Run at least two gateway replicas behind a load balancer. A single gateway is a single point of failure. The LB must support sticky gRPC connections and connection draining on restart.
• Configure the agent’s sending_queue with back-off. A retry storm into the gateway is the worst-case failure shape. retry_initial_interval, retry_max_interval, and retry_max_elapsed_time are the surface.
• Place tail sampling on the gateway. Agents do not have the fleet-wide context to make the decision.
• Smoke test after every topology change. A known trace should reach the backend within ten seconds through every hop.

Verification

You should now be able to answer:

• What is the difference between agent mode and gateway mode in an OTel Collector deployment?
• When does the agent-plus-gateway topology pay for itself?
• Which processors belong on the agent, and which belong on the gateway?
• Why is a single gateway replica a single point of failure?
• What is the failure shape when an aggressive load-balancer idle-connection timeout resets every gRPC stream?

Quiz