Exporter Broken

A trace lookup returns trace not found. The on-call engineer follows links A through C; all are healthy. The SDK is running, the traceparent is on the wire, the collector is healthy and accepting spans. The next link is the exporter. The collector log shows otlp/tempo exporter: exporting failed; server is not processing: connection error: desc = "transport: Error while dialing: x509: certificate signed by unknown authority". The collector’s otelcol_exporter_sent_spans counter is flat at zero. otelcol_exporter_send_failed_spans rises by the thousands. The collector has been retrying since the TLS cert was rotated yesterday. Link D is broken; the exporter cannot deliver.

What it is

“Exporter broken” is the failure shape where the collector has spans in memory but cannot deliver them to Tempo. Three sub-shapes exist, distinguished by where in the delivery chain the failure sits:

1. Exporter cannot connect. The exporter’s gRPC stream cannot open to Tempo. The endpoint env var is wrong, the service is down, the network policy blocks egress.
2. Exporter connects but TLS fails. The gRPC connection opens, the TLS handshake fails. The certificate is expired, the client CA does not match, the cert was rotated without updating the collector’s trust bundle.
3. Exporter delivers but Tempo rejects. The gRPC stream is healthy, batches are sent, but Tempo returns errors. The auth_context extractor is missing a tenant header; the distributor’s rate limiter rejects the tenant.

The three sub-shapes have three different fixes. The wrong fix is to scale the collector when the exporter endpoint is wrong; that adds replicas all pointed at the wrong service.

Why a sysadmin cares

Link D is the layer with the most configuration surface. The exporter has TLS settings, header injection, retry behaviour, queue sizing, timeout tuning, and a target endpoint. A misconfiguration in any one of those breaks delivery without breaking collection. The collector looks healthy in dashboards (it is receiving spans); the Tempo ingestion rate is flat.

The diagnostic must distinguish the three sub-shapes because the fix differs:

• “Cannot connect” → fix DNS, network policy, or endpoint URL.
• “TLS fails” → fix cert rotation, client_ca_file, or SANs.
• “Tempo rejects” → fix headers, tenant, rate limit.

The collector’s exporter self-observability counters name the sub-shape in seconds.

How it works

The exporter is the last leg of the collector pipeline:

  +---------------+    +---------------+    +---------------+
  |   receivers   | -> |  processors   | -> |   exporters   |
  |   (OTLP gRPC) |    |  (batch,      |    |   (otlp/tempo) |
  |               |    |   attributes, |    |               |
  |               |    |   tail_sample)|    |               |
  +---------------+    +---------------+    +---------------+
                              |                     |
                              | batch queue         | gRPC write
                              |                     v
                              |               +---------------+
                              |               |    Tempo      |
                              |               |  distributor  |
                              |               +---------------+
                              |                     |
                              |    queue_size grows |
                              |    on exporter fail |
                              v                     |
                       (queue overflow -> drops) <----+

The exporter holds a queue of batches. On each scheduled_delay (default 5s) or on batch full, the exporter flushes the batch to the target. A failed flush keeps the batch in the queue; the queue grows; the queue overflows at max_queue_size (default 50,000 spans); further spans are dropped with otelcol_exporter_send_failed_spans rising.

Exporter self-observability

The exporter reports its state through several metrics:

  otelcol_exporter_sent_spans{exporter="otlp/tempo"}      - successfully sent
  otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} - failed sends
  otelcol_exporter_queue_size{exporter="otlp/tempo"}      - current queue depth
  otelcol_exporter_queue_capacity{exporter="otlp/tempo"}   - max queue size

The diff between sent and the count the receiver reports is the dropped count. The diff between the queue depth and queue capacity is the headroom. A flat sent with a rising failed is “exporter trying, failing”. A flat sent and flat failed with a rising receiver counter is “exporter backed up”.

How to configure it

OTLP exporter to Tempo (no TLS)

  # /etc/otelcol-contrib/config.yaml
  exporters:
    otlp/tempo:
      endpoint: tempo-distributor.observability.svc:4317
      tls:
        insecure: true
      sending_queue:
        enabled: true
        num_consumers: 10
        queue_size: 5000
        timeout: 5s
      retry_on_failure:
        enabled: true
        initial_interval: 1s
        max_interval: 30s
        max_elapsed_time: 300s

Severity: CONFIGURATION. Restart the collector.

OTLP exporter to Tempo (with TLS)

  exporters:
    otlp/tempo:
      endpoint: tempo-distributor.observability.svc:4317
      tls:
        ca_file: /etc/otelcol-contrib/tls/ca.crt
        cert_file: /etc/otelcol-contrib/tls/client.crt
        key_file: /etc/otelcol-contrib/tls/client.key
        # server_name override for SAN validation
        server_name_override: tempo.internal
      headers:
        X-Scope-OrgID: single-tenant
      sending_queue:
        enabled: true
        num_consumers: 10
        queue_size: 5000
        timeout: 5s
      retry_on_failure:
        enabled: true
        initial_interval: 1s
        max_interval: 30s
        max_elapsed_time: 300s

Severity: CONFIGURATION. Restart the collector.

Batch processor in front of the exporter

  processors:
    batch:
      send_batch_size: 8192
      send_batch_max_size: 10000
      timeout: 200ms
      send_batch_max_size: 10000

Severity: CONFIGURATION. Restart the collector.

How to validate it

Severity: READ-ONLY.

Confirm the exporter counter

  curl -s http://otel-collector.observability.svc:8889/metrics | \
    grep -E '^otelcol_exporter_(sent|send_failed|queue_size)_spans'
  # otelcol_exporter_sent_spans{exporter="otlp/tempo"} 128421
  # otelcol_exporter_send_failed_spans{exporter="otlp/tempo"} 0
  # otelcol_exporter_queue_size{exporter="otlp/tempo"} 0

A rising send_failed counter with a flat sent counter is link D in flight. The cause is in the exporter’s configuration or in the upstream target.

Confirm the exporter can reach Tempo

  kubectl exec deploy/otel-collector -n observability -- \
    nc -zv tempo-distributor.observability.svc 4317
  # tempo-distributor.observability.svc (10.20.4.20:4317) open

A connection refused is “exporter cannot connect”; a timeout is the network policy blocking egress.

Confirm the TLS handshake

  openssl s_client -connect tempo-distributor.observability.svc:4317 \
    -CAfile /etc/otelcol-contrib/tls/ca.crt \
    -cert /etc/otelcol-contrib/tls/client.crt \
    -key /etc/otelcol-contrib/tls/client.key \
    -servername tempo.internal < /dev/null 2>&1 | \
    grep -E 'subject=|issuer=|Verification|verify return code'
  # subject=CN = tempo.internal
  # issuer=CN = Internal CA G2
  # Verification: OK

A Verification: failed with a verify return code is the exact TLS sub-shape of link D.

Confirm the queue is not overflowing

  curl -s http://otel-collector.observability.svc:8889/metrics | \
    grep -E '^otelcol_exporter_(queue_size|queue_capacity)'
  # otelcol_exporter_queue_size{exporter="otlp/tempo"} 0
  # otelcol_exporter_queue_capacity{exporter="otlp/tempo"} 5000

A queue size near queue capacity is “exporter backed up”. The fix is rate limiting, scaling, or queue sizing.

Confirm Tempo is receiving

  curl -s http://tempo-distributor.observability.svc:3200/metrics | \
    grep -E '^tempo_distributor_spans_received_total'
  # tempo_distributor_spans_received_total 128421

A tempo counter matching the exporter’s sent counter confirms the round trip.

How it can fail

Six failure shapes, ordered by frequency in production fleets:

1. Cert rotation did not propagate to the collector. Tempo’s certificate is rotated by cert-manager. The collector’s tls.ca_file still points at the old CA. The TLS handshake fails with x509: certificate signed by unknown authority. Symptom: send_failed_spans rises; collector log shows x509 errors.

2. OTLP exporter endpoint points at the Tempo querier instead of the distributor. The exporter endpoint is tempo:4317 (the DNS name resolves to the querier service IP). The querier accepts gRPC and rejects OTLP writes. Symptom: send_failed_spans rises; collector log shows unknown service "opentelemetry.proto.collector.trace.v1.TraceService".

3. Missing or wrong tenant header. Tempo’s distributor expects X-Scope-OrgID for multi-tenant deployments. The collector’s exporter does not inject the header; the distributor returns 401 or 403. Symptom: collector log shows failed to authenticate request; Tempo’s tempo_distributor_dropped_spans_total rises.

4. Tempo distributor rate-limited the tenant. The tenant’s per-second span rate exceeded the configured limit. The distributor returns ResourceExhausted. Symptom: send_failed_spans rises for one tenant label; other tenants are unaffected.

5. Collector exporter queue overflow. The exporter cannot drain the queue fast enough. Spans accumulate; otelcol_exporter_queue_size reaches otelcol_exporter_queue_capacity; further spans are dropped with otelcol_exporter_dropped. Symptom: flat sent, flat failed, rising dropped; the queue is full.

6. Network policy blocks egress from the collector to Tempo. A new NetworkPolicy was applied to the observability namespace that allows ingress but not egress to the Tempo namespace. Symptom: nc -zv to Tempo times out; the collector log shows connection timeout; the receiver is still receiving.

How to troubleshoot it

The diagnostic order, link D first, cheapest signal first:

1. Read the exporter counters. otelcol_exporter_sent_spans and otelcol_exporter_send_failed_spans. A flat sent with a rising failed is “exporter trying, failing”.
2. Confirm network reachability. nc -zv tempo 4317 from the collector pod. A connection refused is the endpoint wrong; a timeout is the network policy.
3. Confirm the TLS handshake. openssl s_client from the collector host. A Verification: failed is a cert chain mismatch.
4. Read the collector log for the specific error. kubectl logs deploy/otel-collector | grep -E 'x509|ResourceExhausted|connection|tenant'. The error string is the diagnostic.
5. Check the Tempo distributor. Look at tempo_distributor_dropped_spans_total for the matching tenant label. A rising counter at Tempo confirms Tempo is rejecting.
6. Check the queue depth. otelcol_exporter_queue_size near capacity is the exporter is the bottleneck.

Security implications

The exporter is the security boundary to the storage layer.

• TLS. The OTLP exporter supports TLS natively. In production, the OTLP connection to Tempo must be TLS. Plain HTTP is acceptable only inside the cluster network with a strict NetworkPolicy.
• mTLS. The exporter can present a client certificate to Tempo. The Tempo distributor’s tls.client_ca_file validates the client. mTLS is appropriate for service meshes where client identity is enforced.
• Tenant isolation. The exporter must inject the X-Scope-OrgID header for multi-tenant Tempo. A misconfigured header can leak spans across tenants.
• Secrets. The exporter’s TLS key, the bucket access key, and any auth tokens must be sourced from a secret store (Vault, Kubernetes Secret, AWS Secrets Manager). They must not be in the YAML file in plaintext.

Performance implications

The exporter is CPU- and network-bound. The gRPC stream carries protobuf batches; the serialization is CPU; the transmission is network. The bottleneck is usually network at the storage layer.

The exporter queue is a memory cost. A 5,000-span queue at 1 KiB per span is 5 MiB; the memory cost is small. The hidden cost is the queue that grows unbounded because the exporter cannot drain. A queue that grows to capacity indicates the storage layer is too slow; the right fix is storage throughput, not collector scaling.

The exporter’s num_consumers setting governs parallelism. The default of 10 is appropriate for most deployments. A setting of 1 serialises batches and caps throughput; a setting of 50 saturates the gRPC stream and causes head-of-line blocking. Tune on evidence, not on theoretical throughput.

Production guidance

• Always enable TLS on the exporter in production. Plain HTTP inside a private network is acceptable with a strict NetworkPolicy; plain HTTP across a public network is not.
• Always set num_consumers: 10 as the default. Tune on observed queue depth.
• Always inject the X-Scope-OrgID header for multi-tenant Tempo. The header is the tenant boundary.
• Always scrape the collector’s exporter metrics. Alert on send_failed_spans rising for any exporter.
• Always pin the exporter’s CA bundle to the cluster’s cert-manager CA. Cert rotation that does not propagate is the dominant link-D failure.

Verification

You should now be able to answer:

• What are the three sub-shapes of “exporter broken”?
• Which metric pair distinguishes “exporter trying and failing” from “exporter backed up”?
• How do you confirm a TLS handshake failure from the command line?
• What header must the exporter inject for multi-tenant Tempo?
• What is the right diagnostic order when the exporter’s send_failed counter rises?

Quiz