OTel Collector Configuration

A platform team needs to ship three log sources to Loki: OTLP from the application services, syslog from the host, and journald from the systemd services. They have a single collector per host. The config has to handle three receivers, a shared processing chain, and one exporter with the right tenant header. The validation has to confirm that the receivers are accepting, the processors are not dropping, and the exporter is sending.

This lesson is the configuration grammar of otelcol.yaml: receivers, processors, exporters, and the wiring that turns them into a pipeline.

What it is

The OpenTelemetry Collector configuration is a single YAML document with five top-level sections.

receivers:
  <name>: {args }

processors:
  <name>: {args }

exporters:
  <name>: {args }

extensions:
  <name>: {args }

service:
  extensions: [<list>]
  pipelines:
    <signal>:
      receivers:  [<list>]
      processors: [<list>]
      exporters:  [<list>]
  telemetry:
    metrics: {args }
    logs:    {args }

• receivers - a map of named receiver instances. The name is the operator’s choice; the kind is the factory.
• processors - a map of named processor instances. Same naming rules as receivers.
• exporters - a map of named exporter instances.
• extensions - a map of named extension instances. Extensions are wired separately, via service.extensions.
• service.pipelines - the wiring. Each pipeline declares a signal (traces, metrics, logs), a list of receivers, a list of processors, and a list of exporters. A receiver named in a pipeline must be declared in the receivers block; the same applies to processors and exporters.

Why a sysadmin cares

The configuration is the contract between the platform team and the host fleet. Three failure shapes appear when the configuration is treated as an implementation detail.

1. The pipeline that lost its tenant. A loki exporter was added to a new pipeline without the X-Scope-OrgID header. The pipeline ran; the lines arrived in Loki but in the default tenant. The dashboards returned nothing. The fix took an hour to diagnose because the symptom (lines arriving) looked like success.
2. The chain that OOMed. A batch processor was placed before memory_limiter. A downstream outage caused the batch to grow unbounded; the collector OOMed; the host lost its metrics. The fix was to reorder the chain so memory_limiter runs first.
3. The reload that was a no-op. A SIGHUP was sent to a collector process that ignored it (some container distributions do). The on-disk config was new; the running config was old. The fix was a systemctl restart, not a SIGHUP.

How it works

Receivers

A receiver accepts telemetry from a source. The most common receivers for a logs pipeline are listed below.

• otlp - accepts OTLP over gRPC (port 4317) or HTTP (port 4318). The protocol-native receiver for OTel-instrumented applications.
• filelog - tails files from the host filesystem. The closest equivalent to Promtail’s scrape_configs. Supports include / exclude patterns, multiline, encoding, and a chain of operators for parsing.
• journald - tails journald entries from the systemd journal. Equivalent to journalctl -f but as a streaming receiver.
• hostmetrics - scrapes host-level metrics (CPU, memory, disk, network). Not a log receiver but a common companion.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  filelog:
    include:
      - /var/log/app/*.log
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'
      - type: move
        from: body
        to:   attributes

  journald:
    directory: /var/log/journal
    units:
      - sshd
      - nginx

Processors

A processor transforms, batches, filters, or enriches telemetry between a receiver and an exporter.

• memory_limiter - refuses data when the process approaches a memory limit. Must be the first processor in the chain.
• batch - coalesces entries to reduce per-call cost. A larger send_batch_size with a longer timeout trades latency for throughput.
• resource - mutates resource attributes on every entry. Useful for stamping a job or env label.
• attributes - mutates the attributes on an entry. The scope is per-entry, not per-resource.
• filter - drops entries that do not match an expression. Use to drop noise before batching.
• transform - applies a small DSL (the OpenTelemetry Transformation Language) to entries. The most flexible processor; the most expensive.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    timeout: 5s
    send_batch_size: 8192
  resource:
    attributes:
      - key: job
        value: checkout
        action: upsert
  filter:
    logs:
      exclude:
        match_type: strict
        bodies:
          - "DEBUG"

Exporters

An exporter ships telemetry to a backend or to another collector.

• loki - pushes logs to Loki over HTTP. Accepts default_labels_enabled, headers (for X-Scope-OrgID), and tls for CA configuration.
• otlp - exports OTLP over gRPC. The same protocol as the otlp receiver, but in the exporter role.
• otlphttp - exports OTLP over HTTP. Useful when gRPC is blocked by a firewall.
• debug - prints the telemetry to the agent log. Useful for debugging; never used in steady state.

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt

The wiring

The service.pipelines block is the wiring. Each pipeline declares a signal, a list of receivers, a list of processors, and a list of exporters. The runtime constructs the pipeline as a graph; the receivers feed the processors in order; the processors fan out to the exporters.

service:
  pipelines:
    logs:
      receivers:  [otlp, filelog, journald]
      processors: [memory_limiter, filter, resource, batch]
      exporters:  [loki]

The chain order matters. memory_limiter must be first or the batch grows unbounded under a downstream outage. filter should run before batch so dropped entries do not consume batch capacity. resource should run before batch so the labels are present when the batch is exported.

How to configure it

A complete annotated otelcol.yaml for a host that tails application logs, scrapes journald, and exports to Loki.

# /etc/otelcol/config.yaml

receivers:
  # 1. OTLP from OTel-instrumented applications.
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

  # 2. Tail /var/log/app/*.log as log records.
  filelog:
    include:
      - /var/log/app/*.log
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'
      - type: move
        from: body
        to:   attributes
      - type: add
        field: attributes.level
        value: attributes["level"]

  # 3. Tail systemd journal.
  journald:
    directory: /var/log/journal
    units:
      - sshd
      - nginx

processors:
  # memory_limiter first; it protects the rest of the chain.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

  # Drop debug entries before batching.
  filter:
    logs:
      exclude:
        match_type: strict
        bodies:
          - "DEBUG"

  # Stamp a job label.
  resource:
    attributes:
      - key: job
        value: checkout
        action: upsert
      - key: host
        value: ${env:HOSTNAME}
        action: upsert

  # Batch last (in this chain); larger sends, longer timeout.
  batch:
    timeout: 5s
    send_batch_size: 8192

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

extensions: []

service:
  extensions: []
  pipelines:
    logs:
      receivers:  [otlp, filelog, journald]
      processors: [memory_limiter, filter, resource, batch]
      exporters:  [loki]
  telemetry:
    metrics:
      address: localhost:8888
    logs:
      level: info

The sending_queue block on the loki exporter gives the exporter its own bounded queue. Under a downstream outage, the exporter queues up to queue_size entries; once the queue is full, the receivers see it and start back-pressuring.

How to validate it

Validation is a parse-check against the schema, plus a runtime-check of the pipeline metrics.

# CONFIGURATION: parse-check.
otelcol validate --config=/etc/otelcol/config.yaml

# (no output on success; non-zero exit on error)

# READ-ONLY: list the components in the binary.
otelcol components

# (long output; cross-reference against the YAML to confirm
#  every named component is shipped by the binary)

# SERVICE-IMPACT: reload via SIGHUP. Some container images
# ignore SIGHUP; verify the agent log shows a reload entry.
kill -HUP $(pidof otelcol)

# READ-ONLY: confirm the receivers are accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="filelog"} 1872
otelcol_receiver_accepted_log_records{receiver="otlp"} 421

# READ-ONLY: confirm the exporter is shipping.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 2293

If receiver_accepted climbs but exporter_sent does not, the failure is in the pipeline. If exporter_sent climbs but lines do not appear in Loki, the failure is downstream of the collector.

How it can fail

Five failure modes specific to otelcol.yaml.

1. The receiver named in the pipeline but not declared. A service.pipelines.logs.receivers entry names filelog but the receivers block has no filelog entry. Symptom: the collector refuses to start with receiver "filelog" is not declared.
2. The processor out of order. A batch processor placed before memory_limiter. Symptom: under a downstream outage, the batch grows unbounded; the process is OOM-killed; the metric process_runtime_total_alloc_bytes climbs until the kernel acts.
3. The exporter without headers for the tenant. The loki exporter ships to the default tenant instead of the production tenant. Symptom: lines appear in Loki but in the wrong tenant; dashboards return nothing.
4. The filelog receiver that never recovered its position. A filelog receiver was configured with start_at: beginning and a stale storage extension. Symptom: after restart, the receiver re-reads every file from byte zero; Loki receives duplicates.
5. The TLS that trusted nothing. The loki exporter was configured with tls.insecure_skip_verify: true because the CA bundle was missing. Symptom: connections succeed; lines ship; nobody notices until the certificate changes and the collector refuses to verify the new chain.

How to troubleshoot it

When the collector refuses to start, the order matters.

1. Read the error. Both the parse error and the factory error are printed with a line number. The first error is usually the only one.
2. Check the binary. otelcol components lists every component in the binary. Cross-reference against the YAML. A missing component is a startup failure.
3. Validate against the schema. otelcol validate runs the same parse the runtime runs at start. Run it on the file before SIGHUP.
4. Tail the agent log on first reload. The first batch after a reload will fail loudly if anything is misconfigured. Watch for factory errors, parse errors, and rejected entries.
5. Compare the pipeline counters. receiver_accepted minus receiver_refused should approximately equal exporter_sent plus exporter_failed. A persistent gap means the chain is dropping.

Security implications

The configuration holds the wiring; it does not hold the secrets.

• Secrets in the config. The collector supports ${env:VAR} and ${file:/path} references for sensitive values. Use them for passwords and tokens. A literal in a committed config is a credential leak.
• TLS to the exporters. The loki, otlp, and otlphttp exporters accept tls blocks for CA bundles, client certificates, and insecure_skip_verify (the last only for development). A stale CA bundle is the most common cause of silent shipping failure.
• Filesystem access. The filelog and journald receivers read whatever the collector process can read. The process runs as the otelcol user by default; ensure the user has read access to the intended paths and no more.

Performance implications

• Batching. The batch processor coalesces entries to reduce per-call cost. A larger send_batch_size with a longer timeout trades latency for throughput. The default timeout: 200ms is too aggressive for a gateway; five seconds is a more realistic starting point.
• Memory limiter. The memory_limiter processor refuses data when the process approaches a memory limit. It must be the first processor in the chain. The limit_percentage and spike_limit_percentage are the trade-off knobs.
• Sending queue. The sending_queue on exporters backs the exporter with a bounded queue. The queue is in-memory by default; the file_storage extension backs it on disk for durability across restarts.

Production guidance

• Use otelcol-contrib unless the deployment only needs core. The component set in contrib matches the production needs of almost every fleet.
• Place memory_limiter first in the pipeline. The chain is memory_limiter, then filter, then resource, then batch, then the rest. The chain order is the discipline.
• Pin the collector version. The collector releases monthly. Read the release notes. Breaking changes to component schemas do happen.
• Smoke test after every config change. Ship a known line with a unique UUID and confirm it arrives in the right backend with the expected labels within ten seconds.

Verification

You should now be able to answer:

• What is the difference between receivers, processors, and exporters in otelcol.yaml?
• Why must memory_limiter be the first processor in the pipeline?
• How does the loki exporter distinguish the production tenant from the default tenant, and where is it configured?
• What does otelcol components tell you that otelcol validate does not?

Quiz