Collector Failures

The on-call engineer has a customer-reported incident. The customer support team has the order ID. The on-call engineer checks Tempo. The trace is not there. They check the OpenTelemetry Collector metrics. otelcol_receiver_accepted_spans is climbing steadily. otelcol_exporter_sent_spans is climbing at the same rate. The collector received the spans. The collector sent the spans. Tempo does not have them. The story is missing between the collector and the backend.

This is the lesson. The collector is a middle component. It has its own failure modes. When it accepts spans but does not deliver them, the failure is in the exporter, the queue, or the destination.

What it is

A collector failure is the failure mode where the OpenTelemetry Collector (or Grafana Alloy in the same role) accepts spans from the application but does not deliver them to Tempo. The spans exist in the collector’s queue or were dropped from it. The application is innocent; the backend is innocent; the collector in the middle is the source.

Three failure bands within the collector:

• Exporter queue overflow. The application is producing faster than the collector can drain. The queue fills. New spans are dropped at the queue boundary.
• Retry exhaustion. The destination (Tempo) is unreachable for a window. The collector’s retry policy exhausts. The spans are dropped.
• Destination rejection. Tempo is reachable but returns 5xx or rejects the request. The collector logs the rejection and either retries or drops.

The three bands look identical from the Tempo UI. The collector self-metrics separate them.

Why a sysadmin cares

The collector is the chokepoint of every telemetry signal in production. The same collector pipeline typically carries metrics, logs, and traces. A failure in the trace pipeline is often the visible symptom of a broader telemetry outage.

Three operational payoffs ride on the collector-to-backend path:

1. End-to-end delivery. A trace that does not reach Tempo is useless. The collector is the gate. The gate must hold under load, under partial network failure, and under backend degradation.
2. Backpressure signal. A collector whose queue is filling is the earliest indicator that the trace volume has exceeded the design capacity. The right response is either to scale the collector, to scale the backend, or to revisit the sampling policy.
3. Cost ceiling. The collector’s batch processor and queue are the only thing between an in-process SDK and a backend bill. Tuning the queue sizes is a cost-control lever.

The cost of a collector failure is paid in two places: the missing traces in Tempo (the immediate incident) and the downstream capacity that was consumed before the failure was detected.

How it works — the mental model

The collector pipeline has three components in series. Spans flow from left to right.

Application SDK
  +-- outbound OTLP (gRPC or HTTP)
       |
       v
  OpenTelemetry Collector
    Receiver
      +-- accepts spans from the wire
      +-- bounds: receiver protocol, max receive size
      |
      v
    Processors
      +-- batch, attributes, tail_sampling, etc.
      +-- bounds: timeout, batch size
      |
      v
    Exporter (with sending queue + retry)
      +-- queue holds spans until the destination is reachable
      +-- retry restarts on transient failures
      +-- bounds: queue_size, num_consumers, max_elapsed_time
      |
      v
  Tempo

The receiver and processors are usually reliable. The failure bands are the exporter queue and the destination path.

How to configure it

The minimum production configuration for the OTLP exporter to Tempo:

# /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlp/tempo:
    endpoint: tempo.distribution.svc.cluster.local:4317
    tls:
      insecure: true
    # The sending queue: hold spans until Tempo acknowledges.
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    # The retry policy: back off on transient failure.
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    # The timeout per request: how long to wait for Tempo to ack.
    timeout: 30s

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Three tunables matter most:

• num_consumers — start at 10. Raise to 20 if the queue size metric sits above 50 percent capacity in steady state.
• queue_size — start at 5 000. Raise if the retry window needs to absorb a longer Tempo outage. The trade-off is RAM: roughly 1 KB per queued span.
• max_elapsed_time — start at 300 s. Raise to 600 s or more if Tempo restarts are common. The trade-off is that older spans are less useful by the time they are delivered.

Grafana Alloy equivalent (River configuration):

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.distribution.svc.cluster.local:4317"
    tls { insecure = true }
    timeout = "30s"
  }

  retry_on_failure {
    enabled          = true
    initial_interval = "5s"
    max_interval     = "30s"
    max_elapsed_time = "300s"
  }

  sending_queue {
    enabled       = true
    num_consumers = 10
    queue_size    = 5000
  }

  output { traces = [otelcol.exporter.loadbalancing.tempo.input] }
}

The semantics are identical. The names are different.

How to validate it

The diagnostic ladder:

# 1. Is the collector exporter reaching Tempo?
curl -sf http://alloy:8888/metrics | grep otelcol_exporter_sent_spans
# otelcol_exporter_sent_spans{exporter="otlp/tempo"} 18432
# (numbers climbing = exporter is sending)

# 2. Are spans being dropped at the queue?
curl -sf http://alloy:8888/metrics | grep otelcol_exporter_dropped_spans
# otelcol_exporter_dropped_spans{exporter="otlp/tempo"} 0
# (zero in steady state; non-zero means the queue overflowed)

# 3. Is the retry exhausting?
curl -sf http://alloy:8888/metrics | grep otelcol_exporter_send_failed_spans
# otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} 12
# (small numbers during transient failures are normal;
#  large sustained numbers indicate the destination is rejecting
#  or unreachable)

# 4. Is the queue filling?
curl -sf http://alloy:8888/metrics | grep otelcol_exporter_queue_size
# otelcol_exporter_queue_size{exporter="otlp/tempo"} 1247
# (should sit well below queue_size; sustained values above 80%
#  are a backpressure signal)

# 5. Is Tempo actually receiving?
tctl trace search --service=checkout --since=1h --limit=5
# (Tempo should return traces for the service; an empty result
#  means Tempo rejected or the collector never sent)

# 6. Does the collector's own log show errors?
kubectl logs deploy/alloy -c alloy | grep -iE "error|fail|drop"
# (sustained error lines confirm the destination is unreachable)

# 7. What is the Tempo ingester's view?
curl -sf http://tempo:3200/ready
# (200 = ready, 503 = not ready, non-200 = misconfigured)

The fourth metric is the structural answer. A queue that sits above 80 percent capacity is a collector that cannot keep up with its inputs.

How it can fail

Six recurring failure modes.

1. Retry disabled. The exporter is configured without retry_on_failure. One transient network blip drops the batch. Symptom: the collector log shows a single failed send; the spans are gone; Tempo does not have them.
2. Queue size too small. queue_size: 1000 for a service that emits 8 000 spans per second. The queue fills in 125 ms. Every subsequent span is dropped. Symptom: otelcol_exporter_dropped_spans climbs steadily; otelcol_exporter_queue_size sits at the cap.
3. Tempo is rejecting every batch. Tempo is up but the collector is sending to the wrong endpoint, with the wrong auth header, or to a tenant that does not exist. Tempo returns 4xx for every batch. The collector retries (because 4xx is a failure), the queue fills, the failure cascades. Symptom: otelcol_exporter_send_failed_spans increments on every batch; the queue fills; the application keeps producing.
4. Tempo is down. The Tempo ingester pod was restarted. The network path is open but Tempo returns 503 for 6 minutes. max_elapsed_time is 300 s. Every batch tried during the outage is dropped at the retry-exhausted boundary. Symptom: send_failed_spans spikes during the outage then drops to zero; dropped_spans spikes at the same time.
5. The collector itself is OOM. The queue holds 5 000 spans per exporter, the collector has 4 exporters, and the pod has a 256 MB limit. The pod is killed by the kubelet. The restart drops everything in memory. Symptom: the collector’s restart count climbs; spans are missing for the restart window.
6. TLS mismatch. The collector was configured with TLS enabled but the certificate on the Tempo side expired or the SAN does not match the endpoint hostname. The TCP connection succeeds, the TLS handshake fails, every send is rejected. Symptom: the collector log shows TLS errors; send_failed_spans increments on every batch.

How to troubleshoot it

The diagnostic order:

1. Is the destination reachable? curl -v https://tempo:4317. A TCP connection that succeeds is the first prerequisite.
2. Is the exporter sending? Scrape otelcol_exporter_sent_spans. A non-zero rate means the exporter is reaching Tempo and Tempo is acknowledging.
3. Is the queue filling? Scrape otelcol_exporter_queue_size. A sustained high value is a backpressure signal.
4. Are spans being dropped? Scrape otelcol_exporter_dropped_spans. Non-zero means the system is losing evidence.
5. Is Tempo receiving? Use tctl trace search or query the Tempo HTTP API. If Tempo has no traces for the service in question, the collector never sent them or Tempo rejected them.
6. Is the collector itself healthy? Inspect the pod’s memory and CPU. An OOM-killed collector drops everything in memory on restart.

Security implications

The collector’s OTLP receiver and exporter both speak gRPC or HTTP on the wire. Without TLS, span payloads are visible to any party on the network path. The remediation is the same as for any in-cluster traffic: mTLS between the SDK and the collector, mTLS between the collector and Tempo.

The exporter’s auth header (typically a Bearer token) is the authentication for Tempo’s write endpoint. A collector with a weak or expired credential is rejected by Tempo. The remediation is rotation: the credential should rotate on a schedule shorter than the certificate’s natural lifetime.

The third risk is around the metrics endpoint. The collector exposes its own metrics on :8888 by default. The metrics include queue sizes, drop counts, and send-failure counts. An attacker with access to the metrics endpoint can infer the trace volume, the failure rate, and the operational health of the platform. Bind the metrics endpoint to the cluster network only, and protect it with basic auth or a network policy.

Performance implications

The exporter queue and retry are not free. The costs:

• Memory. The queue holds spans in memory until they are acknowledged. Roughly 1 KB per queued span. A 5 000 span queue with 4 exporters is 20 MB.
• CPU. The consumers drain the queue in parallel. The cost per consumer is small (serialise, compress, send) but non-zero.
• Network. The retry policy generates repeated sends during failures. A 5-minute outage with initial_interval: 5s and max_interval: 30s generates roughly 15 retries per batch. The network cost is bounded by the queue size.

The performance-relevant failure mode is the queue overflow. The collector drops spans to protect itself. The system prioritises availability over evidence. The cost is paid in incident response.

Production guidance

• Enable retry on every exporter. retry_on_failure.enabled: true. The default in newer OTel Collector versions is true, but older configurations may have it disabled.
• Right-size the queue. queue_size should be larger than the expected volume during the retry window. A 300 s retry window at 1 000 spans per second needs a 300 000 span queue.
• Alert on dropped_spans. The threshold is zero in steady state. A non-zero counter means the system is losing evidence.
• Alert on queue_size relative to capacity. A sustained value above 80 percent is a backpressure signal. Scale the collector or the destination.
• Verify Tempo readiness before debugging. curl http://tempo:3200/ready returns 200 when Tempo is accepting traces. A 503 is a Tempo-side problem, not a collector-side problem.

Verification

You should now be able to answer:

• What is the difference between a queue overflow and a retry exhaustion?
• Which collector self-metric indicates that spans are being dropped at the exporter queue?
• What is the most common production cause of “spans reach the collector but not Tempo”?

Quiz