Pipelines

A platform team runs a single collector that fans out to Loki, Tempo, and Mimir. The logs arrive. The traces arrive. The metrics arrive. The metric-to-log pivot in Grafana is broken. The investigation shows that the logs pipeline stamps a trace_id attribute on every record; the traces pipeline never sees the attribute because the pipelines are separate. The fix is a connector that routes the trace_id attribute from the logs pipeline into the traces pipeline so that the two signals share correlation data.

This lesson is the wiring: how the service.pipelines block turns a list of components into a flowing pipeline, how the per-signal pipelines interact, and how connectors carry data between them.

What it is

The service.pipelines block is the wiring layer of the collector. Each entry in the block declares one pipeline by signal (traces, metrics, logs), names the receivers, processors, and exporters in that pipeline, and the runtime constructs the graph.

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [otlp/tempo]
    metrics:
      receivers:  [otlp, prometheus]
      processors: [memory_limiter, resource, batch]
      exporters:  [prometheusremotewrite]
    logs:
      receivers:  [otlp, filelog, journald]
      processors: [memory_limiter, filter, resource, batch]
      exporters:  [loki]

Three rules govern the wiring.

1. A pipeline is bound to one signal. A receiver that emits traces cannot be wired into a logs pipeline. The collector refuses to start with a signal-mismatch error.
2. A pipeline runs the processors in order. The first processor is the gate (memory_limiter); the last is the coalescer (batch); the rest is the transformation chain.
3. A pipeline fans out to multiple exporters. The same pipeline can ship to Loki and to a debug file in parallel; the runtime constructs a fan-out branch at the end of the chain.

A connector sits at the boundary between two pipelines. A connector is both an exporter (it ships out of one pipeline) and a receiver (it feeds into the next). Connectors are how data crosses the pipeline boundary without leaving the collector process.

Why a sysadmin cares

The pipeline is the operational shape of the collector. The same components produce different signals depending on the wiring.

1. The pipeline that lost its tenant. A loki exporter was added to a new pipeline without X-Scope-OrgID. The pipeline ran; the lines arrived in Loki but in the wrong tenant. The fix took an hour to diagnose because the symptom (lines arriving) looked like success.
2. The pipeline that OOMed. A batch processor was placed before memory_limiter. A downstream outage caused the batch to grow unbounded; the collector OOMed; the host lost its metrics. The fix was to reorder the chain so memory_limiter runs first.
3. The pipeline that never saw its traces. A traces pipeline was wired into the metrics receivers by mistake. The collector refused to start with a signal-mismatch error; the agent log showed the offending pipeline. The fix was to read the error and correct the signal.

How it works

The pipeline factory

The collector’s pipeline factory builds one pipeline per signal per entry in service.pipelines. Each pipeline is a chain of receivers feeding a chain of processors feeding a set of exporters in parallel. The collectors run as goroutines connected by Go channels.

service.pipelines.logs
  receiver (otlp)
      |
      v
  receiver (filelog)
      |
      v
  receiver (journald)
      |
      v
  +-----+-----+
  |     |     |
  v     v     v
 proc proc proc    <-- memory_limiter (first)
  |     |     |
  v     v     v
 proc proc proc    <-- filter, resource, attributes ...
  |     |     |
  v     v     v
 proc proc proc    <-- batch (last)
  |     |     |
  +-----+-----+
        |
        v
  exporter (loki)

Each receiver pushes pdata into the first channel; the processors consume from the previous channel and push to the next; the exporters consume from the final channel and send. The exporters run in parallel; each exporter holds its own bounded sending_queue.

The per-signal pipelines

Three signals; three pipelines.

• traces. Accepts OTLP, Jaeger, or Zipkin; applies sampling, tail-based decision, resource enrichment, and batching; ships to Tempo or to a tracing backend.
• metrics. Accepts OTLP, Prometheus pull, or host metrics; applies resource enrichment and batching; ships to Mimir, Prometheus remote-write, or to an OTLP endpoint.
• logs. Accepts OTLP, filelog, or journald; applies filtering, parsing, resource enrichment, and batching; ships to Loki or to a logging backend.

The signals are not interchangeable. A traces pipeline cannot carry logs; a logs pipeline cannot carry metrics. The collector refuses to start with a signal-mismatch error.

The order of processors

The chain order is the discipline. The right chain for a logs pipeline on a per-host agent.

1. memory_limiter first — the gate that prevents OOM.
2. filter next — drop noise before any other work.
3. resource — stamp static labels.
4. resourcedetection — add cloud/host metadata once.
5. attributes — mutate per-entry fields.
6. batch last — coalesce for exporter efficiency.

The principle is to drop early, enrich late, batch last. A batch placed before filter wastes capacity on entries that will be dropped. A memory_limiter placed after batch never sees the queue.

Connectors

A connector is a component kind that is declared in both the exporters block and the receivers block. The connector acts as an exporter in one pipeline (it ships out of the chain) and as a receiver in another (it feeds into the next).

connectors:
  forward:
    logs:
    traces:
    metrics:

service:
  pipelines:
    logs/in:
      receivers:  [otlp, filelog]
      processors: [memory_limiter, batch]
      exporters:  [loki, forward]
    traces/in:
      receivers:  [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters:  [otlp/tempo, forward]
    logs/out:
      receivers:  [forward]
      processors: [attributes]
      exporters:  [loki/prod]

The forward connector carries pdata between pipelines. The logs/in pipeline ships to Loki and forwards to logs/out; logs/out adds a tenant attribute and ships to the production Loki. The two-pipeline pattern lets the platform team apply different transformations to different backends without duplicating receivers.

How to configure it

A complete annotated otelcol.yaml for a gateway collector that fans out to Loki, Tempo, and Mimir.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    headers:
      X-Scope-OrgID: prod
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
  otlp/tempo:
    endpoint: tempo.internal.example.com:4317
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    headers:
      X-Scope-OrgID: prod
  prometheusremotewrite:
    endpoint: https://mimir.internal.example.com/api/v1/push
    auth:
      authenticator: bearertokenauth/mimir
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt

extensions:
  bearertokenauth/mimir:
    token: ${env:MIMIR_TOKEN}

service:
  extensions: [bearertokenauth/mimir]
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [otlp/tempo]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [prometheusremotewrite]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, resource, batch]
      exporters:  [loki]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info

Three rules to internalise before promoting the configuration.

1. memory_limiter first, batch last. The chain order is the discipline. A reorder is the most common fix.
2. One signal per pipeline. A traces pipeline cannot carry logs. The collector refuses to start with a signal-mismatch error.
3. Tenant header on every Loki and Tempo exporter. A missing X-Scope-OrgID means the exporter ships to the default tenant.

How to validate it

Validation is a parse-check plus a runtime check of the pipeline counters.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm the receivers are accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="otlp"} 421
otelcol_receiver_accepted_metric_points{receiver="otlp"} 89
otelcol_receiver_accepted_spans{receiver="otlp"} 932

# READ-ONLY: confirm the exporters are shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 421
otelcol_exporter_sent_metric_points{exporter="prometheusremotewrite"} 89
otelcol_exporter_sent_spans{exporter="otlp/tempo"} 932

# READ-ONLY: confirm the chain is not dropping.
curl -s http://localhost:8888/metrics | grep otelcol_processor_refused

# (empty output = healthy; a non-empty value means the
#  memory_limiter is refusing data)

If receiver_accepted climbs but exporter_sent does not, the failure is in the pipeline. If exporter_sent climbs but the data does not appear in the backend, the failure is downstream of the collector. If the per-signal counters are flat at zero, the pipeline is wired but the receiver is not bound to the right address.

How it can fail

Six failure modes specific to pipelines.

1. The pipeline that references an undeclared component. A service.pipelines.logs.receivers entry names filelog but the receivers block has no filelog entry. Symptom: the collector refuses to start with receiver "filelog" is not declared.
2. The pipeline with the wrong signal. A loki exporter was wired into the traces pipeline. Symptom: the collector refuses to start with a signal-mismatch error.
3. The chain with memory_limiter after batch. The batch grows unbounded; the limiter never sees the queue. Symptom: process_runtime_total_alloc_bytes climbs to 2-3x the configured limit; the kernel OOM-kills the process.
4. The pipeline without the tenant header. The loki exporter ships to the default tenant instead of the production tenant. Symptom: lines arrive in Loki but in the wrong tenant; production dashboards return empty.
5. The connector that never connected. The forward connector was declared but not wired into any pipeline. Symptom: the collector starts; no data flows between the pipelines; the agent log shows the unused component.
6. The pipeline that fed two exporters with one queue. Two exporters were wired into one pipeline; both share the same fan-out channel. Symptom: a slow exporter back-pressures the chain; the fast exporter stops accepting too.

How to troubleshoot it

When a pipeline is not behaving, the diagnostic order matters.

1. Confirm the pipeline exists. Read service.pipelines.* from top to bottom. Every receiver, processor, and exporter named in the pipeline must be declared in its block.
2. Confirm the signal matches. A traces pipeline cannot carry logs; a logs pipeline cannot carry metrics. The collector refuses to start with a signal-mismatch error.
3. Confirm the chain order. memory_limiter first; batch last. A reorder is the most common fix.
4. Confirm the receivers are bound. ss -tlnp shows the listeners. A receiver on localhost is invisible from outside the host.
5. Confirm the exporters are reaching the backend. Check the backend’s own metrics; check the tenant; check the TLS chain.
6. Compare per-pipeline counters. Each pipeline has its own set of receiver_accepted_*, processor_*_refused, and exporter_sent_* counters. A pipeline with a flat receiver_accepted has a bind or routing problem; a pipeline with a climbing receiver_accepted and a flat exporter_sent has a chain problem.

Security implications

The pipeline is the configuration; it does not hold the secrets. The wiring decisions, however, have security implications.

• Tenant isolation. Each pipeline that ships to Loki or Tempo must carry the X-Scope-OrgID header. A pipeline with a missing header ships to the default tenant; the cost is silent misrouting.
• Filesystem access. A filelog receiver in the logs pipeline reads whatever the collector process can read. Run the process as a dedicated user with read access to the intended paths and no more.
• TLS to the backends. Every exporter in every pipeline must set tls.insecure: false. The default is false; never override it.
• Credentials. Use ${env:VAR} or ${file:/path} references for the Authorization header and the bearertokenauth token. A literal in a committed config is a credential leak.

Performance implications

The pipeline is the place where the per-signal costs are shaped.

• Per-signal chain. Each signal has its own chain; the costs are independent. A traces pipeline with tail_sampling costs memory; a logs pipeline with filter costs CPU.
• Connector cost. The forward connector is a copy. A pipeline that feeds a connector and an exporter doubles the output cost. Use connectors deliberately.
• Fan-out cost. Multiple exporters in one pipeline run in parallel. A slow exporter back-pressures the chain; a fast exporter waits. Plan the exporter count against the per-exporter cost.
• Receiver count. A pipeline that lists three receivers runs three source loops. A pipeline that lists ten receivers runs ten source loops. The cost is the cost of the slowest receiver.

Production guidance

• memory_limiter first, batch last. The chain order is the discipline. A reorder is the most common fix.
• One signal per pipeline. A receiver that emits traces cannot be wired into a logs pipeline.
• Tenant header on every Loki and Tempo exporter. A missing X-Scope-OrgID means the exporter ships to the default tenant.
• Smoke test after every config change. Ship a known record with a unique UUID and confirm it arrives in the right backend with the expected labels within ten seconds.

Verification

You should now be able to answer:

• What does service.pipelines wire, and what is the relationship between a pipeline and a signal?
• Why is the chain order memory_limiter first and batch last?
• What is the difference between a fan-out to multiple exporters and a connector between pipelines?
• How do you confirm that a pipeline is flowing data end to end from the self-metrics?
• What does a signal-mismatch error mean, and where does it appear?

Quiz