Break/Fix: OTel Collector Pipeline Broken

Reported symptoms

It is Thursday morning. A support escalation needs the trace for an order placed on Wednesday and there is no trace. Neither is there one for any other request, from any service, since 14:22 on Tuesday. Before 14:22 the estate’s traces are all there and query normally. After it, Tempo has nothing.

The obvious suspects are all clean:

• The three gateway collectors have not restarted since Tuesday’s rollout. Resident memory is flat at about 700 MiB. CPU is roughly a fifth lower than it was last week, which nobody has thought about.
• The health_check extension has returned 200 without interruption. No readiness probe has failed.
• Tempo is healthy. /ready returns 200 and its ingesters are up.
• Metrics and logs are fine. So is the journald stream that Tuesday’s change was made to add - that change worked.
• Every application team says their exporter is succeeding. No export errors, no retries, no queue growth in any SDK.

The two alerts written for this exact failure have both stayed silent, and neither has been touched in months. Meanwhile the RED dashboard’s exemplar links land on “trace not found” for anything recent, and the trace-to-logs pivot dead-ends the same way, so the on-call rotation has quietly stopped using both.

Tuesday’s change was one added logs: pipeline in the Helm values file. The review comment on the pull request reads, in full, “adds journald, plus a reindent of service:”.

Evidence provided

$ curl -s http://otelcol-gateway:8888/metrics | grep otelcol_receiver_accepted_spans

otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 4.71829e+09

$ curl -s http://otelcol-gateway:8888/metrics | grep otelcol_exporter_sent_spans

otelcol_exporter_sent_spans{exporter="debug"} 4.71802e+09

$ curl -s http://otelcol-gateway:8888/metrics \
| grep -E 'otelcol_exporter_(send_failed_spans|queue_size)|otelcol_processor_refused_spans'

otelcol_exporter_send_failed_spans{exporter="loki"} 0
otelcol_exporter_send_failed_spans{exporter="prometheusremotewrite"} 0
otelcol_exporter_queue_size{exporter="loki"} 0
otelcol_exporter_queue_size{exporter="prometheusremotewrite"} 0
otelcol_processor_refused_spans{processor="memory_limiter"} 0

The exporter block in the config that the pod is actually running, unchanged since long before Tuesday:

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability.svc.cluster.local:4317
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    headers:
      X-Scope-OrgID: prod
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
  debug: {}

And the service: block from the same file:

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [debug]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [prometheusremotewrite]
    logs:
      receivers:  [otlp, journald]
      processors: [memory_limiter, resource, batch]
      exporters:  [loki]
  telemetry:
    metrics:
      address: 0.0.0.0:8888

$ otelcol validate --config=/conf/relay.yaml; echo "exit=$?"

exit=0

Finally, the two alerts, neither of which has fired:

- alert: OtelCollectorExporterFailing
  expr: rate(otelcol_exporter_send_failed_spans{exporter="otlp/tempo"}[5m]) > 0
  for: 10m
- alert: TempoIngestionStopped
  expr: rate(otelcol_exporter_sent_spans{exporter="otlp/tempo"}[5m]) == 0
  for: 15m

Work the evidence before reading on

Take the silence of the two alerts as evidence rather than as a fault. They are syntactically fine and semantically fine, and they were never going to fire.

1. otelcol_exporter_sent_spans returns a series for debug, loki and prometheusremotewrite, and none for otlp/tempo. What is the difference between a series whose value is zero and a series that does not exist, and what does each of the two alert expressions do when handed the second one?
2. The otlp/tempo exporter block is present, complete and correct in the running config. Under what circumstance does a correctly configured component produce no self-metrics at all?
3. CPU is a fifth lower than last week and nobody raised it. What work has the process stopped doing?
4. The SDKs report success and hold no backlog. At what point in the path does an OTLP client consider a batch delivered?
5. Metrics and logs are unaffected, including the pipeline the change added. What does that isolate, and what does it rule out?
6. The diff was one added pipeline and a reindent. What kind of change can remove something while appearing in a diff only as whitespace?

Before continuing: name the one line of configuration that is missing, and say why nothing anywhere reported it.

Root cause

The traces pipeline exports to debug, and to nothing else

service.pipelines.traces.exporters reads [debug]. The otlp/tempo exporter is declared, complete and correct, and it is never built - a component that no pipeline references is not instantiated. That is the whole fault, and every strange observation follows from it.

The debug exporter accepts every batch it is handed and cannot fail. So otelcol_exporter_send_failed_spans stays at zero, the sending queue stays empty, the memory limiter never refuses anything, and the process gets cheaper, because the work of framing gRPC, doing TLS and waiting for Tempo to acknowledge has gone. A fifth less CPU was the platform telling the truth about a pipeline that had stopped doing its job.

The applications see success because OTLP acknowledges at the receiver, not at the backend. The batch span processor in each SDK sends a batch, receives an OK from the collector’s OTLP receiver, and releases the spans from its buffer. Everything after that acknowledgement is the collector’s problem, and from the SDK’s point of view this pipeline is delivering perfectly.

Absence is not zero, and both alerts were written against zero

This is the part worth carrying to other systems. TempoIngestionStopped evaluates rate(otelcol_exporter_sent_spans{exporter="otlp/tempo"}[5m]) == 0. When the exporter is not built, that selector matches nothing, rate() over nothing produces nothing, and the comparison has nothing to compare. The rule returns an empty vector every fifteen seconds, forever, and an empty vector is not a firing alert. The same is true of OtelCollectorExporterFailing.

Both alerts encode an assumption that the exporter will exist and misbehave. The failure mode that actually happened is that the exporter ceased to exist, and an alert cannot detect the disappearance of the thing it selects on. The expression that does is absent().

Maps merge, lists replace

The gateway config is rendered by the upstream collector Helm chart. The team’s values are merged over the chart’s own default config, and the two kinds of YAML value merge differently: maps merge key by key, and lists are replaced wholesale.

That is why the exporters: map came through untouched, otlp/tempo and all. The author of Tuesday’s change needed to add a logs: pipeline, started from the chart’s default service: block because that was the nearest complete example, and pasted it in with the new pipeline appended. The chart’s default wires every pipeline to debug - the only defensible default for a chart that cannot know your backend - and that default replaced the team’s list.

The diff really did look like a reindent. The exporters: line under traces changed from one bracketed list to another, on a line whose indentation had also moved, in a file where the reviewer’s attention was on the new pipeline below it.

Resolution

1. State the loss before fixing it. Every span since 14:22 Tuesday is gone rather than delayed, and nothing holds a copy. Put that in the incident channel now, because somebody is about to promise a customer a trace that will never exist.
2. Add otlp/tempo back to service.pipelines.traces.exporters in the values file, and remove debug from the same list in the same change.
3. Validate the rendered output, not the values file: render the chart and read the service: block that will actually ship. otelcol validate on the rendered config confirms it parses, which is necessary and was never the problem.
4. Restart one gateway replica. The collector applies configuration at process start, so this is a restart rather than a reload, and it drops whatever is in the sending queues of the metrics and logs pipelines on that replica - a few seconds of the two signals that are currently healthy.
5. Confirm on that replica before touching the next one: otelcol_exporter_sent_spans{exporter="otlp/tempo"} exists and is climbing at roughly the receiver rate.
6. Roll the remaining replicas one at a time. A rollout that half-applies leaves a third of spans going to debug, which presents as sampling and is much harder to diagnose than a total outage.
7. Send a synthetic span with a service name nobody else uses and query Tempo for it. This is the first check in the whole incident that crosses the entire path.
8. Open the follow-up before closing: the absent() alerts and the end-to-end probe. The config fix leaves the estate exactly one merge away from repeating this.

Verification

1. The series exists. otelcol_exporter_sent_spans{exporter="otlp/tempo"} returns data on every replica. Check presence first and rate second - absence is what the incident was.
2. The ratio is right. Sent spans should climb at roughly the rate accepted spans climb. A ratio near one third means one replica of three took the change.
3. The debug exporter is gone. otelcol_exporter_sent_spans{exporter="debug"} must return no series. If it still returns one, the pipeline is fanning out to it and the misleading yes is still available to the next investigation.
4. The end-to-end probe passes. One span, a unique service name, ten seconds, and a Tempo query that finds it. Nothing measured inside the collector is a substitute for this.
5. The derived paths recovered. An exemplar link on the RED dashboard produced after the restart lands on a real trace, and the trace-to-logs pivot from it lands on real lines.
6. The new alerts would have caught it. Evaluate the absent() rule against the two days you have just lived through and confirm it would have fired on Tuesday at 14:23. An alert that cannot detect the incident that motivated it is decoration.
7. The gap is documented. First minute without traces and first minute with them, read from Tempo rather than from the collector, published where the people searching that window will find it.

Prevention

• Write absent() alerts for anything whose disappearance is possible. Every rate-and-threshold alert in the estate is blind to its own subject being removed, and removal is a normal consequence of ordinary config work.
• Probe end to end, per signal. Emit a known record on a schedule and query the backend for it. Component self-metrics told a completely consistent story here and every part of it was true; none of it was the answer.
• Diff the rendered configuration, not the values file. helm template, or the ConfigMap on the running pod, is what the process will execute. The values diff is what somebody intended, and merge semantics live in the gap.
• Internalise the merge rule, because it is not specific to this chart: maps merge deeply, lists replace wholesale. Any list in a generated config - pipeline members, processor chains, scrape job names, volume mounts - is a place where an additive-looking change can silently remove an entry.
• Keep debug out of production pipelines. It costs a little CPU and it buys a permanent false positive: with it present, “something is exporting” can be true while nothing useful is happening.
• Put an accepted-versus-sent ratio panel per signal on the platform dashboard. A receiver counter that climbs beside an exporter counter that does not exist is a wiring fault, and it reads that way in one glance.