OpenTelemetry Collector Overview

A team runs OTel-instrumented services across 200 pods. The applications export OTLP to a collector. The collector was deployed as a single instance with no queue, no retry, and no backpressure. A 30-second Loki outage overwhelmed the collector’s in-memory queue; the collector started rejecting batches with resource_exhausted; the applications saw retries and back-off cascades. The fix was a gateway topology with a disk queue, a memory limiter, and a per-pipeline batch processor.

This lesson is the architecture of the OpenTelemetry Collector: the components, the pipelines, the distributions, and the deployment modes. The trade-offs are real; the wrong topology for the failure shape is the wrong answer.

What it is

The OpenTelemetry Collector is the vendor-neutral telemetry collector published by the CNCF OpenTelemetry project. It is a single binary that runs as a process, accepts telemetry from many sources, transforms and batches it, and exports it to many backends. The configuration language is YAML. The architecture is organised into four component kinds, declared in their own blocks and wired into pipelines in a fifth block.

+-----------+     +-------------+     +----------+
| receivers | --> | processors  | --> | exporters|
+-----------+     +-------------+     +----------+
       \              |               /
        \             |              /
         +----- service.pipelines ---+

• Receivers accept telemetry from the wire or from a local source. otlp, filelog, journald, prometheus, zipkin, k8s_cluster, hostmetrics. Receivers produce telemetry; they do not export it.
• Processors transform, batch, filter, or enrich telemetry between a receiver and an exporter. batch, memory_limiter, attributes, resource, filter, transform. Processors do not accept input from the wire or write to a backend.
• Exporters ship telemetry to a backend or to another collector. otlp, otlphttp, loki, debug, file, prometheusremotewrite. Exporters consume telemetry; they do not produce it for the local pipeline.
• Extensions provide capabilities that are not strictly receivers, processors, or exporters. health_check, pprof, zpages, bearertokenauth. Extensions are wired via service.extensions.
• service.pipelines is the wiring. Each entry in service.pipelines declares a pipeline by signal (traces, metrics, logs), names the receivers, processors, and exporters in the pipeline, and the runtime constructs the graph.

Why a sysadmin cares

The Collector is the strategic collector when the fleet is OTel-instrumented. The reasons are operational, not ideological.

1. The vendor-neutral receiver surface. A team that has already instrumented its services with the OTel SDK has already chosen a vendor-neutral protocol (OTLP). The collector is the consistent next step: same protocol, same data model, same metadata conventions. Alloy can receive OTLP, but the collector is the protocol-native answer.
2. The processor as a first-class concept. Processors are declared in their own block, wired into pipelines by name, and shared across signal types. A memory_limiter placed before the batch processor protects every pipeline; the same protection in Alloy requires per-block configuration.
3. The distribution that matches the deployment. The collector ships as core (a small set of receivers, processors, and exporters) and contrib (everything from core plus the community-contributed components). The right distribution is the one whose component set matches the deployment; running contrib everywhere is wasteful, running core everywhere is under-equipped.

The wrong answer is to treat the collector as a Grafana-stack replacement. The collector does not ship to Loki with the same ergonomics as Alloy; the protocol-native receiver for Loki is the loki exporter, and the configuration is verbose.

How it works

The factory model

Every component kind is a Go interface in the OpenTelemetry Collector codebase. The collector binary embeds the factories for the components it ships; the YAML configuration names the factories and supplies their arguments. A receiver named otlp in the YAML maps to the otlp factory; the factory builds the receiver; the pipeline wires it.

The factory model means:

• A missing component is a startup failure. If the YAML names a receiver that the binary does not ship, the collector refuses to start. There is no plugin loader; the binary is what it is.
• A misconfigured component is a startup failure. If the YAML passes an argument the factory does not accept, the collector refuses to start. The error names the argument.
• The pipeline is the contract. A receiver that is not in any pipeline is loaded but never receives data; an exporter that is not in any pipeline is loaded but never receives data. Both add memory cost for no benefit.

Distributions

The collector is published as two main distributions.

• otelcol (core) - the basic set of receivers, processors, exporters, and extensions maintained by the OpenTelemetry project. No third-party components. The core binary is small.
• otelcol-contrib (contrib) - everything in core plus the community-contributed components. The contrib binary is large (hundreds of components) and is the de facto standard for production deployments.

The choice is governed by the component set you need. If the deployment only needs otlp, batch, and otlphttp, core is sufficient and the binary is smaller. If the deployment needs filelog, journald, loki, or any other contrib component, contrib is required.

Deployment modes

The collector ships two operational modes.

• Agent mode - one collector process per host or per pod. The agent receives telemetry from local sources, batches it, and forwards it to a backend (or to a gateway). Agent mode is the on-host pattern.
• Gateway mode - one or more collector processes per cluster, receiving from many agents. The gateway is the central pipeline; it batches, filters, and ships to the backends. Gateway mode is the multi-tenant pattern.

The two modes can compose. An agent fans in to a gateway; the gateway fans out to the backends. The gateway is a single point of failure unless it is deployed as a StatefulSet with at least two replicas behind a load balancer.

How to configure it

A minimum viable collector configuration that receives OTLP, applies a memory limiter and a batch, and exports to Loki.

# /etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}

extensions: []

service:
  extensions: []
  pipelines:
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [loki]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info

The service.telemetry block configures the collector’s own metrics and logs. The default metrics endpoint is localhost:8888. The default health-check endpoint is localhost:13133. Both bind to localhost by default.

How to validate it

Validation is a parse-check against the schema.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# SERVICE-IMPACT: reload via SIGHUP. The collector diffs the
# new config against the running one and only restarts the
# changed components.
kill -HUP $(pidof otelcol)

# READ-ONLY: confirm the receiver is accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="otlp"} 1872

# READ-ONLY: confirm the exporter is shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 1872

If receiver_accepted climbs but exporter_sent does not, the failure is in the pipeline (a processor is dropping, a memory limiter is refusing, or the batch has not flushed yet). If exporter_sent climbs but lines do not appear in Loki, the failure is downstream of the collector.

How it can fail

Five failure modes specific to the OpenTelemetry Collector.

1. The missing component in the binary. The YAML names a receiver that the running binary does not ship. Symptom: the collector refuses to start with component "filelog" not found in the binary. The fix is to switch from otelcol to otelcol-contrib, or to remove the missing component.
2. The pipeline that references an undeclared component. A service.pipelines.logs.receivers entry names filelog but the receivers block has no filelog entry. Symptom: the collector refuses to start with receiver "filelog" is not declared.
3. The exporter without headers for the tenant. The loki exporter ships to the default tenant instead of the production tenant. Symptom: lines appear in Loki but in the wrong tenant. Dashboards return nothing for the production tenant.
4. The memory_limiter placed after the batch. A collector configured with processors: [batch, memory_limiter] lets the batch grow unbounded; the limiter never sees the queue. Symptom: the collector OOMs under a downstream outage; the metric process_runtime_total_alloc_bytes climbs until the kernel kills the process.
5. The SIGHUP that did not reload the pipelines. A collector process that ignores SIGHUP (some container distributions do). Symptom: kill -HUP $(pidof otelcol) succeeds; the agent log shows no reload entry; the running config is unchanged. The fix is a systemctl restart or a container restart.

How to troubleshoot it

When the collector refuses to start, the order matters.

1. Read the error. Both the parse error and the configuration error are printed with a line number and the offending argument. The first error is usually the only one.
2. Check the binary. otelcol components lists every component in the binary. Cross-reference against the YAML. A missing component is a startup failure.
3. Validate against the schema. otelcol validate runs the same parse the runtime runs at start. Run it on the file before SIGHUP.
4. Tail the agent log on first reload. The first batch after a reload will fail loudly if anything is misconfigured. Watch for parse errors, factory errors, and rejected entries.
5. Confirm the export path is healthy. A configuration that parses cleanly but ships to a dead endpoint looks identical to a working one until the metrics are inspected.

Security implications

The collector exposes the same surfaces as Alloy; the names differ.

• The debug and metrics endpoints. Defaults are localhost:8888 for /metrics, localhost:8889 for /debug, and localhost:13133 for the health check. All bind to localhost. Expose them on the cluster network only with authentication.
• The OTLP receiver. The otlp receiver accepts data on 0.0.0.0:4317 (gRPC) and 0.0.0.0:4318 (HTTP) by default. In agent mode, bind to localhost or to a private interface. In gateway mode, restrict the listener with NetworkPolicy.
• TLS to the exporters. The loki, otlp, and otlphttp exporters accept tls configuration blocks for CA bundles, client certificates, and insecure-skip-verify (the last only for development). A stale CA bundle is the most common cause of silent shipping failure.

Performance implications

The on-host cost is similar to Alloy - roughly 100-150 MiB RAM and 50-100 millicores CPU at modest line rates. The differences appear at scale and at the gateway.

• Batching. The batch processor coalesces entries to reduce per-call cost. A larger send_batch_size with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; five seconds is a more realistic starting point.
• Memory limiter. The memory_limiter processor refuses data when the process approaches a memory limit. It must be the first processor in the chain; otherwise the batch grows unbounded and the limiter never sees the queue.
• Disk queue. The file_storage extension backs the sending_queue on exporters. A disk queue lets the gateway survive a downstream outage without rejecting batches.

Production guidance

• Use otelcol-contrib unless the deployment only needs core. The component set in contrib matches the production needs of almost every fleet. The binary is larger; the cost of a missing component at 03:00 is more.
• Place memory_limiter first in the pipeline. After the limiter, place batch, then the rest. The chain order is the discipline.
• Pin the collector version. The collector releases monthly. Read the release notes. Breaking changes to component schemas do happen; the migration is usually a one-line edit but the breaking change is the signal to upgrade, not the work itself.
• Smoke test after every config change. Ship a known line with a unique UUID and confirm it arrives in the right backend within ten seconds.

Verification

You should now be able to answer:

• What are the four component kinds, and which block wires them into pipelines?
• Why is otelcol-contrib the de facto standard rather than otelcol?
• What is the difference between agent mode and gateway mode, and when does each apply?
• Why must memory_limiter be the first processor in the pipeline?

Quiz