Collector Anatomy

A 200-pod fleet speaks OTLP. Each pod pushes spans, metrics, and log records to a single collector process. The collector is the choke point: every signal the platform sees passes through it, and every loss the platform suffers is decided there. A single misnamed component refuses to start the whole process. A processor placed in the wrong order lets the queue grow until the kernel kills the binary. The cost of a vague mental model is paid at 03:00.

This lesson is the anatomy of the OpenTelemetry Collector: the five component kinds, the pipeline that wires them, the lifecycle each component runs, and the deployment topologies that change the operational shape of the whole system.

What it is

The OpenTelemetry Collector is a single Go binary that accepts telemetry from many sources, transforms it through a pipeline, and ships it to many backends. The configuration is a single YAML document organised into five top-level blocks. The runtime constructs the pipeline as a graph; data flows through the graph as pdata (the in-memory representation shared by all OpenTelemetry SDKs).

+----------+     +-----------+     +----------+
| receivers| --> |processors | --> |exporters |
+----------+     +-----------+     +----------+
       \             |               /
        +----- service.pipelines ----+

+----------+     +-----------+     +----------+
| extension|     |  receiver |     | exporter |
| health_  |     |   otlp    |     |  loki    |
| check    |     | filelog   |     | otlp     |
| pprof    |     | journald  |     | otlphttp |
| file_    |     | prometheus|     | debug    |
| storage  |     |           |     | file     |
+----------+     +-----------+     +----------+

The five blocks.

• Receivers accept telemetry from the wire or from a local source. The receiver is the inbound edge of the pipeline. It produces pdata; it never consumes from the wire and ships somewhere else.
• Processors transform, batch, filter, or enrich telemetry between a receiver and an exporter. A processor takes pdata in and emits pdata out. It accepts nothing from the wire and writes to no backend.
• Exporters ship telemetry to a backend or to another collector. An exporter consumes pdata and writes it to the outside world.
• Extensions provide capabilities that do not fit the receiver-processor-exporter shape: health endpoints, profiling endpoints, persistent storage for the sending queue, and authentication helpers.
• service.pipelines is the wiring block. Each entry names the signal, the receivers, the processors, and the exporters for one pipeline.

A sixth element, the connector, sits at the boundary between two pipelines. A connector is both an exporter (it ships out of one pipeline) and a receiver (it feeds into the next). Connectors are how a router pipeline and a backend pipeline share data inside a single collector process.

Why a sysadmin cares

The collector is the strategic collector when the fleet is OTel-instrumented. Three properties make it so.

1. Vendor neutrality. The collector accepts and emits OTLP, the protocol every OTel-instrumented application already speaks. The configuration is YAML; the schema is open; no single vendor owns it. The cost of a vendor change is the cost of swapping an exporter, not the cost of re-instrumenting the fleet.
2. Processor as a first-class concept. A processor is a named, declared, wired unit. memory_limiter, attributes, resource, transform, tail_sampling — each is a building block with a documented cost and a documented benefit. The alternative is per-component ad-hoc transformation code in every application.
3. Two clean deployment topologies. Agent mode runs the collector on every host; gateway mode runs a small number of collectors at the cluster edge. The two compose: agents forward to a gateway; the gateway fans out to the backends.

The cost is verbosity. An OTel Collector configuration for a modest pipeline is roughly twice the line count of the equivalent Alloy configuration. The benefit is that the verbosity lives in a single file per host, in version control, and is owned by one team.

How it works

The factory model

Every component is a Go interface in the collector codebase. The collector binary embeds the factories for the components it ships. The YAML configuration names a factory and supplies its arguments; the runtime builds the component and wires it into the pipeline.

The factory model produces three load-bearing properties.

• A missing factory is a startup failure. If the YAML names filelog but the binary does not ship the filelog factory, the collector refuses to start with component "filelog" not found in the binary. There is no plugin loader; the binary is what it is.
• A wrong argument is a startup failure. If the YAML passes start_at: beginnin (typo), the collector refuses to start with unknown value "beginnin" for field start_at.
• A pipeline that references an undeclared component is a startup failure. A receiver named in service.pipelines must exist in the receivers block; the same rule applies to processors, exporters, and extensions.

These three rules together mean a misconfigured collector fails fast, loudly, and on the first reload. The cost of debugging is the cost of reading the error.

Component lifecycle

Every component moves through four states.

  +------+     +-------+     +---------+     +----------+
  | New  | --> | Start | --> | Running | --> | Shutdown  |
  +------+     +-------+     +---------+     +----------+
      \                                  /
       \------ Start failure -----/

1. New — the factory built the component. The runtime has its configuration; the component has not begun work.
2. Start — the component opens its sockets, connects to its backends, and registers with the rest of the pipeline. A Start failure is fatal for the collector.
3. Running — the component accepts, transforms, or ships data. The Running state is observable via the collector self-metrics on localhost:8888.
4. Shutdown — the component drains its queues, closes its sockets, and releases its resources. Shutdown is invoked by SIGTERM (clean stop) or SIGHUP (reload).

A failure in Start is a hard failure: the collector exits with non-zero status and systemd (or the container runtime) restarts it. A failure in Running is a soft failure for that component: the receiver or exporter stops accepting data, the pipeline back-pressures, and the operator sees the gap on the dashboard.

Data flow

Each entry in service.pipelines declares a pipeline for one signal: traces, metrics, or logs. The runtime builds the pipeline as a chain of Go channels connecting goroutines.

  receiver  -->  processor  -->  processor  -->  exporter
     |              |                |              |
     v              v                v              v
  channel_1  --> channel_2  -->  channel_3  --> channel_4
                                                  |
                                                  v
                                            sending_queue
                                            (memory or disk)
                                                  |
                                                  v
                                                backend

The receivers push pdata into the first channel. Each processor consumes from its input channel and pushes to its output. The final processor feeds the exporters in parallel; each exporter holds its own bounded sending_queue. When the queue fills, the exporter applies backpressure to its input channel; the processors slow; the receivers eventually stop accepting data. The upstream source sees a back-off and reacts (the OTLP SDK retries with jitter).

Backpressure is the safety valve. Without it, a slow Loki would fill the collector memory until the kernel OOM-killed the process. The memory_limiter processor is the gate that refuses data before the queue fills. Without memory_limiter, the queue is the only brake.

Extensions

Extensions are components that do not fit the receiver-processor-exporter shape. They are wired via service.extensions, separately from pipelines. The most common ones.

• health_check — exposes /status on localhost:13133. The container or systemd unit polls this endpoint to decide whether the collector is ready.
• pprof — exposes /debug/pprof/ for profiling. Bind to localhost; never expose on the cluster network without authentication.
• zpages — exposes human-readable diagnostics under /debug/. Useful when the GUI dashboard is not available.
• file_storage — backs the sending_queue of an exporter on disk. Lets a gateway survive a downstream outage without losing buffered data.
• bearertokenauth / basicauth / oauth2client — client-side authentication helpers used by exporters.

An extension declared in the extensions block but not wired into service.extensions is loaded but unused. An extension wired into service.extensions but not declared is a startup failure.

How to configure it

A minimum viable collector configuration that accepts OTLP, applies a memory limiter and a batch, and exports to Loki. The configuration below is the smallest useful shape; the lessons that follow extend it.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}

extensions: []

service:
  extensions: []
  pipelines:
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [loki]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info

Three rules to internalise before the next lessons.

1. memory_limiter is first. The chain order is the discipline. A batch placed before memory_limiter grows unbounded under downstream pressure and the process OOMs.
2. The collector self-metrics bind to localhost. The default for the metrics endpoint is localhost:8888; the default for the health endpoint is localhost:13133. Bind to the cluster network only with authentication.
3. Headers carry the tenant. The loki exporter routes by X-Scope-OrgID. Without it, the distributor falls back to the default tenant and dashboards return nothing.

How to validate it

Validation is a parse-check against the schema plus a runtime check of the pipeline counters.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# READ-ONLY: list every component shipped by the binary.
otelcol components

receivers:
  - otlp
  - filelog
  - journald
  - prometheus
  - hostmetrics
processors:
  - batch
  - memory_limiter
  - attributes
  - resource
  - transform
  - filter
exporters:
  - otlp
  - otlphttp
  - loki
  - debug
extensions:
  - health_check
  - pprof
  - file_storage

# READ-ONLY: confirm the receiver is accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="otlp"} 1872

# READ-ONLY: confirm the exporter is shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 1872

If receiver_accepted climbs but exporter_sent does not, the failure is in the pipeline: a processor is dropping, a memory limiter is refusing, or the batch has not flushed. If exporter_sent climbs but lines do not arrive in Loki, the failure is downstream of the collector.

How it can fail

Six failure modes specific to the collector as a whole.

1. The receiver named in a pipeline but not declared. A service.pipelines.logs.receivers entry names filelog but the receivers block has no filelog entry. Symptom: the collector refuses to start with receiver "filelog" is not declared.
2. The component not shipped by the binary. The YAML names filelog; the binary is otelcol (core), which does not ship filelog. Symptom: the collector refuses to start with component "filelog" not found in the binary.
3. The memory_limiter placed after batch. The chain grows the batch unbounded; the limiter never sees the queue. Symptom: process_runtime_total_alloc_bytes climbs; the kernel OOM-kills the process; the host loses its metrics agent.
4. The exporter without the tenant header. The loki exporter ships to the default tenant instead of the production tenant. Symptom: lines arrive in Loki but in the wrong tenant; production dashboards return empty.
5. The SIGHUP that did not reload the pipelines. A collector process that ignores SIGHUP (some container distributions do). Symptom: the on-disk config is new; the running config is old; the agent log shows no reload entry.
6. The extension wired but not declared. A service.extensions entry names file_storage but the extensions block has no file_storage entry. Symptom: the collector refuses to start with extension "file_storage" is not declared.

How to troubleshoot it

When the collector is not behaving, the order of the diagnostic matters.

1. Read the error first. Both parse errors and factory errors are printed with a line number and the offending argument. The first error is usually the only one.
2. Confirm the binary matches the YAML. otelcol components lists every component shipped by the binary. Cross-reference against the YAML. A missing component is a startup failure.
3. Validate against the schema. otelcol validate runs the same parse the runtime runs at start. Run it on the file before sending SIGHUP.
4. Tail the agent log on first reload. The first batch after a reload fails loudly if anything is misconfigured. Watch for factory errors, parse errors, and rejected entries.
5. Compare pipeline counters. receiver_accepted minus receiver_refused should approximately equal exporter_sent plus exporter_failed. A persistent gap means the chain is dropping.
6. Confirm the export path is healthy. A configuration that parses cleanly but ships to a dead endpoint looks identical to a working one until the metrics are inspected.

Security implications

The collector exposes the same surfaces as any telemetry component. The defaults are local-only; production deployments often relax them in ways that need an audit.

• The debug and metrics endpoints. Defaults are localhost:8888 for /metrics, localhost:8889 for /debug, and localhost:13133 for the health check. All bind to localhost. Expose them on the cluster network only with authentication.
• The OTLP receiver. The otlp receiver accepts data on 0.0.0.0:4317 (gRPC) and 0.0.0.0:4318 (HTTP) by default. In agent mode, bind to localhost or to a private interface. In gateway mode, restrict the listener with a NetworkPolicy.
• TLS to the exporters. The loki, otlp, and otlphttp exporters accept tls blocks for CA bundles, client certificates, and insecure_skip_verify. A stale CA bundle is the most common cause of silent shipping failure.
• Filesystem access. The filelog and journald receivers read whatever the collector process can read. Run the process as a dedicated user with read access to the intended paths and no more.

Performance implications

The on-host cost is roughly 100-150 MiB RAM and 50-100 millicores CPU at modest line rates. The cost grows with batch size, queue depth, and per-record transform work.

• Batching. The batch processor coalesces entries to reduce per-call cost. A larger send_batch_size with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; five seconds is a more realistic starting point.
• Memory limiter. The memory_limiter processor refuses data when the process approaches a memory limit. The limit_percentage and spike_limit_percentage are the trade-off knobs. The limit must be the first processor in the chain.
• Sending queue. The sending_queue on exporters is in-memory by default. The file_storage extension backs it on disk for durability across restarts; the disk cost is real, and the queue_size should be planned against the worst-case outage.

Production guidance

• Use otelcol-contrib unless the deployment only needs core. The component set in contrib matches the production needs of almost every fleet. The binary is larger; the cost of a missing component at 03:00 is more.
• Place memory_limiter first in every pipeline. After the limiter, place batch, then the rest. The chain order is the discipline.
• Pin the collector version. The collector releases monthly. Read the release notes. Breaking changes to component schemas do happen; the migration is usually a one-line edit but the breaking change is the signal to upgrade.
• Smoke test after every config change. Ship a known line with a unique UUID and confirm it arrives in the right backend within ten seconds.

Verification

You should now be able to answer:

• What are the five blocks in otelcol.yaml, and which block wires the others into a pipeline?
• What is the difference between a Start failure and a Running failure, and which one is recoverable?
• Why is the memory_limiter processor placed first in the pipeline, and what happens if it is not?
• What do otelcol_receiver_accepted_* and otelcol_exporter_sent_* tell you about the pipeline?
• What is the difference between agent mode and gateway mode, and when does each apply?

Quiz