Exporters

A platform team wires the gateway collector to Loki and Tempo. The logs arrive. The traces do not. The team investigates. The tempo exporter is configured with the right endpoint and the right tenant header; the agent log shows no errors; the self-metric otelcol_exporter_sent_spans is climbing. The failures are at Tempo. The exporter ships the spans; Tempo returns 4xx on every batch. The cause is a stale CA bundle.

This lesson is the exporter surface of the collector: how each exporter writes to its backend, what arguments are production-relevant, and how to choose the right exporter for the destination at hand.

What it is

An exporter is the outbound edge of a collector pipeline. It consumes pdata from the previous processor and writes to a backend or to another collector. An exporter never accepts input from the wire; a receiver never exports. The boundary is strict.

   processors ...
        |
        v
   exporter_1     --> Loki       (logs)
   exporter_2     --> Tempo      (traces)
   exporter_3     --> Mimir      (metrics, remote-write)
   exporter_4     --> file.jsonl (debugging)
   exporter_5     --> debug      (stdout/stderr)

The exporter decides the wire protocol and the transport; the rest of the pipeline sees the same pdata regardless of destination. That uniformity is the property that lets a single pipeline fan out to Loki, Tempo, and Mimir without any per-backend conditional logic.

Why a sysadmin cares

The exporter is the place where the pipeline meets the backend. Five failure shapes originate here.

1. The exporter without the tenant header. A loki exporter was added to a new pipeline without X-Scope-OrgID. Symptom: lines arrive in Loki but in the default tenant; production dashboards return empty.
2. The exporter with a stale CA bundle. The tls.ca_file pointed at a CA bundle that was rotated six months ago. Symptom: the exporter logs TLS handshake errors; the self-metric otelcol_exporter_send_failed_* climbs; no data reaches the backend.
3. The exporter without a sending_queue. A gateway exporter with no queue drops data on every backend blip. Symptom: gaps in the trace and metric series during a routine backend restart.
4. The exporter that sent to the wrong signal. A loki exporter was wired into the traces pipeline instead of the logs pipeline. Symptom: the collector refuses to start with a signal-mismatch error; the agent log shows the offending pipeline.
5. The exporter that exhausted the connection pool. A otlp exporter to a gateway was configured with num_consumers: 1 against a thousand concurrent streams. Symptom: the queue fills; the receivers back-pressure; the applications retry.

How it works

Wire exporters

Wire exporters push telemetry to a backend over the network.

• otlp exports OTLP over gRPC. The same protocol as the otlp receiver, but in the exporter role. The exporter holds a gRPC client connection per endpoint; the connection is multiplexed across streams. The exporter honours the sending_queue and retry_on_failure blocks.
• otlphttp exports OTLP over HTTP. Useful when gRPC is blocked by a firewall or terminated by a load balancer that does not support gRPC. The exporter serialises the pdata to protobuf or JSON and POSTs it to the endpoint.
• prometheusremotewrite exports metrics in the Prometheus remote-write protocol. The right exporter for Mimir, Thanos, and the Prometheus server itself when run in remote-write mode.
• loki exports logs to Loki over HTTP. The exporter converts each log record to a Loki stream entry; the labels on the stream come from resource attributes and from the configured label mappings.
• tempo exports traces to Tempo over the OTLP protocol. The exporter sends the pdata trace to the Tempo OTLP endpoint; Tempo ingests the spans into its own block store.

Local exporters

Local exporters write telemetry to a file or to the agent log.

• file writes pdata to a JSON or protobuf file. Useful for debugging and for replaying captured traffic against a new collector configuration.
• debug prints pdata to the agent log. Useful for verifying a configuration; never used in steady state.

How to configure it

Four exporters for the most common backends, annotated.

# /etc/otelcol/config.yaml

exporters:
  # 1. Loki for logs. The endpoint is the Loki distributor
  #    push path; the X-Scope-OrgID header is the tenant.
  #    default_labels_enabled maps resource attributes to
  #    stream labels; default is true.
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

  # 2. Tempo for traces. The exporter uses the OTLP protocol;
  #    the endpoint is the Tempo OTLP receiver.
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca-certificates.crt
    headers:
      X-Scope-OrgID: prod
    sending_queue:
      enabled: true
      num_consumers: 8
      queue_size: 10000

  # 3. Mimir for metrics via remote-write.
  prometheusremotewrite:
    endpoint: https://mimir.internal.example.com/api/v1/push
    auth:
      authenticator: bearertokenauth/mimir
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 10000

  # 4. Debug exporter for verifying a configuration. The
  #    verbosity level controls how much is printed.
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

Eight arguments that recur in production tuning.

• endpoint — the URL or host:port of the backend. For otlp, use host:port; for loki, use the full push path.
• headers — extra HTTP headers. The X-Scope-OrgID header is the Loki and Tempo tenant marker.
• tls.ca_file — the CA bundle used to verify the backend certificate. The default is the system trust store.
• tls.insecure — skip TLS verification. The default is false. Never set it to true in production.
• sending_queue.enabled — turn on the exporter queue. The default is true for wire exporters; turning it off means data is dropped on backend errors.
• sending_queue.queue_size — the bound on the queue. The default is 1000 entries; a gateway needs 5000 or more.
• sending_queue.num_consumers — the parallelism of the consumer goroutines. The default is 10; a slow backend benefits from fewer consumers to avoid pile-up.
• retry_on_failure — the retry policy. The default is enabled with exponential backoff. The right discipline is to bound the retry with a maximum elapsed time.

How to validate it

Validation is a parse-check plus a runtime check of the exporter counters.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm the exporter is shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 1872
otelcol_exporter_sent_metric_points{exporter="prometheusremotewrite"} 4218
otelcol_exporter_sent_spans{exporter="otlp/tempo"} 932

# READ-ONLY: confirm the exporter is not failing.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_send_failed

# (empty output = healthy; non-zero values mean the backend
#  is rejecting or unreachable)

The exporter-side diagnostic follows three steps.

1. The exporter is shipping. otelcol_exporter_sent_* climbs over time.
2. The exporter is not failing. otelcol_exporter_send_failed_* is zero or near zero; a non-zero value means the backend is rejecting the request.
3. The data arrives. Verify in the backend (Loki, Tempo, Mimir) that the expected data is present and that the tenant is correct.

If exporter_sent climbs but the data does not appear in the backend, the failure is downstream of the collector. Check the backend logs; check the network; check the tenant header.

How it can fail

Six failure modes specific to exporters.

1. The exporter without the tenant header. The loki exporter was wired into the pipeline without X-Scope-OrgID. Symptom: lines arrive in Loki but in the default tenant; production dashboards return empty.
2. The exporter with a stale CA bundle. The tls.ca_file points at a bundle that was rotated. Symptom: the exporter logs TLS handshake errors; otelcol_exporter_send_failed_* climbs; no data reaches the backend.
3. The exporter with no sending_queue. The exporter drops data on every backend blip. Symptom: gaps in the trace and metric series during a routine backend restart; otelcol_exporter_send_failed_* spikes and never recovers.
4. The exporter wired to the wrong signal. The loki exporter was wired into the traces pipeline. Symptom: the collector refuses to start with a signal-mismatch error.
5. The exporter with the wrong protocol. The otlp exporter was configured with an HTTP-only endpoint. Symptom: the exporter logs connection-refused; the gRPC client cannot negotiate an HTTP/2 stream.
6. The debug exporter that ran in production. The exporter was wired into the pipeline for verification and never removed. Symptom: the agent log fills with structured pdata; the disk fills; the host enters a degraded state.

How to troubleshoot it

When an exporter is not behaving, the diagnostic order matters.

1. Confirm the exporter is shipping. otelcol_exporter_sent_* should climb. A flat zero means the exporter is running but the chain is not feeding it; the failure is upstream.
2. Confirm the exporter is not failing. otelcol_exporter_send_failed_* should be near zero. A non-zero value means the backend is rejecting the request or is unreachable.
3. Confirm the backend received the data. Check the backend’s own metrics; check the tenant; check the labels. The exporter may ship successfully and the backend may still reject the data.
4. Check the TLS chain. A handshake error means the CA bundle does not trust the backend certificate. Update the bundle or set tls.insecure: false (the default).
5. Tail the agent log at the exporter. The exporter logs the response from the backend with status code and a truncated body. The first error is usually the only one.

Security implications

Exporters are the egress boundary of the collector. The defaults are encrypted; production deployments need to keep them that way.

• TLS to the backends. Every wire exporter supports tls.ca_file, tls.cert_file, tls.key_file, and tls.insecure. Never set tls.insecure: true in production. A stale CA bundle is the most common cause of silent shipping failure.
• Credentials. Use ${env:VAR} or ${file:/path} references for the Authorization header and the bearertokenauth token. A literal in a committed config is a credential leak.
• Tenant isolation. The X-Scope-OrgID header is the tenant marker for Loki and Tempo. A missing header means the exporter ships to the default tenant. The cost of a missing header is silent misrouting.
• Network exposure. The exporter opens a connection to the backend. The collector egress is the place to enforce NetworkPolicy; restrict the destination to the backend CIDR.

Performance implications

The exporter is the cheapest component per record, but the most expensive per call.

• Wire exporters. The cost is the cost of the network round-trip and the serialisation. A loki exporter with send_batch_size: 100 makes 10 calls per 1000 records; the same exporter with send_batch_size: 1000 makes 1 call. The larger batch trades latency for throughput.
• sending_queue. The queue is in-memory by default. A 5000-entry queue at 1 KiB per entry holds 5 MiB; a 50,000 entry queue holds 50 MiB. Plan the queue size against the worst-case backend outage.
• file_storage extension. The disk-backed queue holds data across restarts. The disk cost is real; the queue_size should be planned against the worst-case outage.
• num_consumers. The parallelism of the consumer goroutines. A high value parallelises the call but piles up on a slow backend; a low value serialises the call and back-pressures sooner. The right value is the value that matches the backend’s accept rate.

Production guidance

• Set tls.insecure: false on every wire exporter. The default is false; never override it.
• Reference the CA bundle explicitly. The default is the system trust store. A backend with a private CA needs tls.ca_file pointing at the explicit bundle.
• Use ${env:VAR} for credentials. A literal in a committed config is a credential leak.
• Enable sending_queue on every wire exporter. The queue bounds the data on a backend blip; without it, the chain drops on every error.
• Smoke test after every config change. Ship a known record with a unique UUID and confirm it arrives in the right backend with the expected labels within ten seconds.

Verification

You should now be able to answer:

• What is the difference between an exporter and a receiver, and what does each side of the pipeline consume?
• Which exporter should you use for Loki, for Tempo, and for Prometheus remote-write?
• What does tls.ca_file do, and why is a stale bundle the most common cause of silent shipping failure?
• What does a non-zero otelcol_exporter_send_failed_* counter tell you, and what is the first diagnostic step?
• Why is sending_queue important on a wire exporter?

Quiz