Runbook: Investigate OTel Collector Failure

1 · Prerequisites

Confirm every item is in place before any state change.

• Collector Anatomy
• Collector Down
• Collector Failures
• OTel Collector Pressure
• OTel Collector Configuration
• Read access to the collector self-telemetry endpoint (service::telemetry::metrics, localhost:8888 by default)
• A way to read the collector process logs, including the log of the PREVIOUS container or unit start
• The running configuration file, and the repository commit it is supposed to match
• Knowledge of which signals (traces, metrics, logs) share this collector - the pipelines are separate but the process is not

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · The affected signal is named: traces, metrics or logs. A collector usually runs several pipelines in one process and only one of them may be failing; "telemetry is broken" is not yet a diagnosis.
• · The blast radius is known: one collector, one deployment, or every collector at once. All of them at once points at a shared configuration push or a shared destination, not at a host.
• · The restart count has been read BEFORE anything is restarted. A collector that has restarted three times in twenty minutes is a different incident from one that has been up for a week and stopped exporting.
• · The counters are read before any restart. otelcol_receiver_accepted_* and otelcol_exporter_sent_* are cumulative from process start; a restart zeroes the evidence that would localise the failure.
• · Whether the exporter queue is backed by disk (file_storage extension) is established. Without it, everything buffered in memory is lost the moment the process restarts, which changes whether a restart is cheap.
• · The time the symptom started is pinned against the deploy or config-push timeline for BOTH the collector and the backend it exports to.
• · It is established whether the collector is the only one in the path. An agent-plus-gateway topology has two collectors and the counters must be read at both, or the investigation attributes a gateway failure to an agent.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Establish the process state before touching it. systemctl status otelcol or kubectl get pods -l app=otel-collector. A restart loop is a START failure and is diagnosed from the previous log, not from the counters - the counters never got a chance to move.
2. 2On a restart loop, read the PREVIOUS log. journalctl -u otelcol -n 100 or kubectl logs deploy/otel-collector --previous. A start failure names the exact component: an unknown component key, an invalid field value, or a pipeline that references something the receivers block does not declare. The error is the answer; there is nothing to infer.
3. 3On a running collector, read the two counters that define the quadrant. curl -s localhost:8888/metrics | grep -E "otelcol_(receiver_accepted|exporter_sent)". Nothing accepted is an inbound problem; accepted but not sent is an outbound problem; both climbing is a problem downstream of the collector. Do not skip this because the answer feels obvious.
4. 4If nothing is being accepted: prove the port is bound before blaming the network. ss -lntp | grep -E "4317|4318" on the collector host. A bound port with no traffic is a network or client problem; an unbound port on a healthy process is a receiver that was never built - the classic typo in the receivers block, which starts cleanly and reports healthy because no receiver is unhealthy, it simply is not there.
5. 5Confirm the pipeline really contains the receiver. A component declared under receivers: but never named in service::pipelines is loaded and unused. The zpages extension at /debug/pipelinez shows the pipelines the runtime actually built, which is the only authority on this.
6. 6If accepted is climbing but sent is not: go to the exporter, not the receiver. Read otelcol_exporter_send_failed_*, otelcol_exporter_queue_size and otelcol_exporter_queue_capacity together. A queue at capacity with failures climbing is the signature; the queue depth alone tells you nothing without the capacity beside it.
7. 7Separate the three exporter bands, because they have three different fixes. Queue overflow (the destination keeps up but not fast enough), retry exhaustion (the destination was unreachable for longer than max_elapsed_time), and permanent rejection (the destination is reachable and refusing - wrong auth, wrong tenant header, wrong endpoint). The exporter log line names which one.
8. 8Test the destination from the collector host, not from your laptop. Reachability from the collector network namespace is the only reachability that matters, and a service mesh or network policy makes the two answers differ.
9. 9**If otelcol_processor_refused_* is climbing, the memory limiter is shedding load - find out why before raising the limit.** A collector under memory pressure is usually a collector with a slow exporter behind it. Raising limit_mib in that state moves the failure from controlled shedding to an OOM kill, which loses the queue as well.
10. 10Make one change, at the edge the counters named. Validate first: otelcol validate --config=/etc/otelcol/config.yaml. Then apply. A configuration reload is not universally supported by container images, so confirm the log records the reload rather than assuming it.
11. 11Verify by direction, not by absence of errors. otelcol_receiver_accepted_* and otelcol_exporter_sent_* must both be rising, at rates in the same order of magnitude, with send_failed and refused flat. Then query the backend for data written AFTER the fix.
12. 12Record the gap and close it out. Everything dropped at the receiver, everything shed by the memory limiter and everything that exhausted its retries is gone. Write down the window and the affected signals; the investigation that needs those minutes will happen later, by someone else.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓otelcol_receiver_accepted_* for the affected signal is climbing at roughly the rate the sources produce, not merely non-zero.
• ✓otelcol_exporter_sent_* is climbing in the same order of magnitude as accepted. A large and persistent gap between the two is a pipeline that is still losing data, whatever the logs say.
• ✓rate(otelcol_exporter_send_failed_*) and rate(otelcol_processor_refused_*) are at zero over a window that is longer than one batch timeout.
• ✓otelcol_exporter_queue_size sits well below otelcol_exporter_queue_capacity in steady state. A queue that is merely draining is not yet a queue that is healthy - re-check after one full batch interval.
• ✓The health check extension returns success on its configured port (13133 by default). Note this is a weak check on its own: a collector with no receiver built also reports healthy.
• ✓The destination actually holds the data: query Tempo, Loki or Prometheus for something written after the fix landed, by trace ID, stream selector or metric name. The exporter counter proves a send, not an ingest.
• ✓The process has not restarted since the change: an incrementing restart count with healthy counters means the collector is being killed and recovering between reads.
• ✓The gap window is documented with a start time, an end time and the list of signals affected.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Every diagnostic step up to the configuration change is read-only and has nothing to undo.
• ↶To undo a configuration change: restore the previous file, run otelcol validate against it, and restart. The restart costs whatever is in the in-memory sending queue - the same cost as the change that is being reverted, paid twice.
• ↶If limit_mib was raised and the process is now being OOM-killed by the kernel or the container runtime, put it back immediately. Controlled shedding drops some data; an OOM kill drops the queue as well and takes the process with it.
• ↶If num_consumers was raised and the backend degraded further, revert it. More parallel senders against a struggling destination is a load test aimed at the thing that is already failing.
• ↶If max_elapsed_time or queue_size was raised, the memory footprint rose with it. Revert on the same reasoning as above, and treat the sizing as a capacity change to be made in daylight rather than an incident action.
• ↶There is no rollback for dropped telemetry. Spans refused at the receiver, records shed by the memory limiter, and batches that exhausted their retries are gone permanently; the only recoverable case is a disk-backed queue that survived the restart.
• ↶Any change merged under pressure is temporary until it exists in the repository with a reviewer on it - a collector whose running config differs from the repository is the next incident.

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate to the backend owner as soon as the failure is shown to be permanent rejection: a 4xx from Tempo, Loki or the remote-write endpoint is a contract problem (auth, tenant, path) that no collector-side tuning fixes.
• · Escalate to the network or platform team when the port is bound on the collector and unreachable from the client namespace. That is a policy, mesh or DNS problem and the collector configuration is not where it lives.
• · Escalate to the emitting service team when one source is responsible for the volume that pushed the collector into shedding. The collector-side mitigation buys time; the sampling or instrumentation fix belongs to them.
• · Escalate to capacity planning if the collector sheds load at steady-state volume with a healthy destination. That is a sizing conversation - more replicas, or a split of the traces and metrics pipelines into separate processes - and not something to solve with one more restart.
• · Escalate to security if the OTLP receiver is reachable from outside the expected client set, or if the volume spike originates from an unrecognised source. An unauthenticated receiver on a routable network accepts arbitrary telemetry from anyone who finds it.
• · Escalate to incident command as soon as the gap in telemetry becomes long enough to matter to an investigation in progress. A team debugging a production incident needs to be told that the evidence for the last twenty minutes does not exist.

References

1. OpenTelemetry Collector configuration
2. OpenTelemetry Collector internal telemetry
3. OpenTelemetry Collector deployment patterns
4. OTLP exporter: sending queue and retry
5. Batch processor
6. Health check extension
7. Grafana Alloy documentation (the same pipeline model, different syntax)