OTel Collector Pressure

Spans are dropping. The Collector has restarted twice in twenty minutes. Trace coverage has dropped from 100 per cent to 40 per cent. Metrics and logs continue to flow, but traces have a 60 per cent gap. The OTel Collector is the platform’s central pipeline for telemetry, and it is the platform’s single point of failure when one exporter under-performs.

This lesson is about OTel Collector pressure: the memory and queue dynamics that determine whether the Collector drops data or processes it, and the configuration that makes the difference predictable.

What it is

OTel Collector pressure is the condition where the Collector cannot keep up with the incoming volume. Most commonly the memory_limiter processor sheds load to prevent an out-of-memory kill, and the cost of shedding is data loss. Less commonly a single slow exporter backpressures the entire pipeline and the receivers themselves begin to refuse connections.

The Collector is built from four kinds of components: receivers, processors, exporters, and extensions. The pipeline runs left to right. Data flows from a receiver through a series of processors to one or more exporters. Every exporter has a bounded queue in front of it; a queue that fills up propagates backpressure through the processors to the receivers.

Why a sysadmin cares

Three production pains concentrate in Collector pressure:

1. Silent data loss. When the Collector drops data, the symptom is invisible until an investigator needs it. A 30-minute-old gap in the trace data is discovered only when the team tries to reconstruct what happened during an incident.
2. Inconsistent pipelines. Metrics, logs and traces share the same Collector process but flow through different pipelines. A pressure event on the traces pipeline can starve the metrics pipeline for CPU if the pipelines share goroutines, or vice versa.
3. The Collector is a single point of failure. A Collector that restarts is a Collector that drops the data that arrived during the restart. A platform with one Collector has no redundancy for the central pipeline.

The lesson is that the Collector is a critical component in any modern observability stack. Its memory and queue behaviour must be sized for the platform’s load and monitored with the same care as the rest of the stack.

How it works

The Collector pipeline is a graph of components connected by bounded queues:

   receivers (otlp, prometheus, jaeger, ...)
        |
        v
   +-----------------+
   |  memory_limiter |  --  refuses data when RSS exceeds limit
   +-----------------+
        |
        v
   +-----------------+
   |  batch          |  --  coalesces items up to send_batch_size
   +-----------------+     or send_batch_max_size, whichever
        |                  comes first
        v
   +-----------------+
   |  sending_queue  |  --  bounded buffer between processor and exporter
   +-----------------+     default queue_size: 1000
        |
        v
   +-----------------+
   |  exporter       |  --  otlphttp, otlpgrpc, prometheusremotewrite, ...
   +-----------------+     retry_on_failure + retry_on_error
        |
        v
     backend

The memory_limiter processor samples the process RSS every check_interval (default 1s). When the RSS exceeds limit_mib * limit_percentage / 100, the processor refuses data and the receivers begin to drop or refuse connections depending on the receiver’s policy.

The sending_queue is a per-exporter buffer with a configurable size (default 1000) and a configurable number of consumers (default 10). When the queue fills, the exporter’s batch processor holds data in memory; if the queue fills faster than the consumers can drain it, the backpressure propagates backward through the pipeline.

How to configure it

Collector pressure is mitigated at the memory_limiter, the batch processor and the exporter queue. All three must be sized together.

# Severity: CONFIGURATION
# /etc/otelcol/config.yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048          # total memory budget for the Collector
    spike_limit_mib: 512     # headroom above limit before refusal begins

  batch:
    timeout: 200ms           # flush at least every 200 ms
    send_batch_size: 8192    # send when batch reaches this many items
    send_batch_max_size: 10000   # never send more than this many

exporters:
  otlphttp:
    endpoint: http://backend:4318
    timeout: 10s
    sending_queue:
      enabled: true
      num_consumers: 10      # parallel workers per exporter
      queue_size: 5000       # bounded buffer per exporter
    retry_on_failure:
      enabled: true
      initial_interval: 100ms
      max_interval: 30s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Five rules govern these knobs:

1. limit_mib should be 75 per cent of the host’s memory limit. The remaining 25 per cent is for Go’s runtime overhead.
2. spike_limit_mib controls how much headroom above limit_mib is allowed during a spike. Set to roughly 25 per cent of limit_mib.
3. send_batch_size and send_batch_max_size control the batch processor. Larger batches are more efficient; smaller batches reduce per-batch latency.
4. queue_size should be sized for the longest backend outage you intend to survive. A 5000-item queue at 1 KB per item is 5 MB; tune to your item size.
5. num_consumers should match the backend’s parallelism. Setting it higher does not help a backend that can only handle 10 concurrent requests.

How to validate it

Validation is a sequence of read-only checks. Start with the configuration, then move to the live internal metrics and the zPages endpoint:

# Severity: READ-ONLY
otelcol-contrib validate --config=/etc/otelcol/config.yaml

The next step is to inspect the internal metrics. The Collector exposes its own telemetry on the same Prometheus endpoint:

# Severity: READ-ONLY
sum by (signal) (rate(otelcol_exporter_sent[5m]))
sum by (signal) (rate(otelcol_exporter_send_failed[5m]))
sum by (processor) (rate(otelcol_processor_refused[5m]))
sum by (kind) (rate(otelcol_receiver_accepted[5m]))

The expected result, in a healthy Collector, is a non-zero sent rate, a near-zero failed rate, and a zero refused rate. A non-zero refused rate is the canonical sign that the memory_limiter is shedding load.

The zPages extension exposes the pipeline state for ad-hoc diagnostics:

# Severity: READ-ONLY
curl http://otelcol:8888/debug/pipelinez

The expected result is a JSON document listing each pipeline and the queue depth per exporter.

How it can fail

Six failure shapes account for nearly every Collector pressure incident:

1. memory_limiter set too tight. The RSS limit is below the steady-state memory footprint. Symptom: the Collector refuses data continuously even when the backend is healthy.
2. memory_limiter set too loose. The RSS limit is above the host’s actual memory limit. Symptom: the host OOM-kills the Collector before the memory_limiter can react.
3. batch timeout too long. Data sits in the batch for too long waiting to be flushed. Symptom: latency for a single item is the timeout, even when the backend is healthy.
4. batch size too small. Every batch is below send_batch_size, so the batch waits for the timeout. Symptom: high CPU on the batch processor and many small requests to the backend.
5. Single slow exporter backpressures the whole pipeline. One exporter’s queue fills and the backpressure propagates through the processors to the receivers. Symptom: traces stop flowing even though the receivers are healthy.
6. Sending queue too small. The exporter’s queue_size is below the burst size of incoming data. Symptom: otelcol_exporter_send_failed rises during traffic spikes that the queue cannot absorb.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes:

1. Confirm the symptom is Collector pressure. Inspect otelcol_processor_refused and otelcol_exporter_send_failed. Any non-zero rate is a sign that the Collector is shedding load.
2. Identify the failing exporter. The exporter name is a label on the failed metric. Sort by failed rate to identify the worst offender.
3. Inspect the exporter queue. otelcol_exporter_queue_size and otelcol_exporter_queue_capacity show the fill level. A queue that is at capacity is a queue that is backpressuring the pipeline.
4. Inspect the memory_limiter log. The memory_limiter logs every refusal at debug level. Grep for memory_limiter.
5. Inspect the runtime. process.runtime.go.mem.heap.alloc and process.runtime.go.mem.heap.sys show the Go runtime state.
6. Inspect the zPages pipeline view. /debug/pipelinez shows the live state of every pipeline.

Security implications

The Collector exposes several surfaces. Three disciplines matter in production:

1. The receiver endpoints must require TLS. Every receiver that accepts OTLP must use tls settings; insecure: true is acceptable only for development.
2. The authentication extension must be enabled. The auth extension on the receiver enforces authentication; without it any client can push data.
3. The Collector process must run as a non-root user. The Collector binds privileged ports only when run as root; in production run as a dedicated user.

A Collector with insecure: true and no authentication extension is a public endpoint that accepts arbitrary telemetry. Treat it as such.

Performance implications

Collector cost is a function of three variables: ingest volume, batch size and exporter concurrency. The arithmetic is:

collector_memory_peak
    = ingest_volume_per_second * max_latency_seconds
collector_cpu
    = batch_count_per_second * per_batch_cost

A Collector that ingests 10 000 spans per second and has a 5-second queue absorbs 50 000 spans in flight. A 1 KB per span budget is 50 MB of headroom. The memory_limiter must be sized above this; the host’s memory limit must be sized above the memory_limiter.

Verification

You should now be able to answer:

• Which Collector processor enforces a hard backpressure on the pipeline based on RSS?
• What is the difference between send_batch_size and send_batch_max_size in the batch processor?
• Which two internal metrics confirm that the Collector is shedding load under pressure?
• What does an empty sending queue mean, and what does a full one mean?
• How would you size limit_mib relative to the host’s memory limit?

Quiz