Receivers

An application team ships OTel-instrumented services. The traces arrive. The metrics arrive. The logs do not, because the services log to /var/log/app/*.log on the host, not over OTLP. The team spends two hours debating whether to add a log forwarder. The answer was in the receiver list all along: filelog was already available; the configuration simply omitted it.

This lesson is the receiver surface of the collector: how each receiver gets its data, what arguments are production-relevant, and how to choose the right receiver for the source at hand.

What it is

A receiver is the inbound edge of a collector pipeline. It accepts telemetry from the wire (push protocols) or scrapes it from a local source (pull protocols), converts the source format into pdata, and emits it onto the pipeline. A receiver never exports; an exporter never receives. The boundary is strict.

                Wire (push)                 Local (pull)
              +-------------+            +---------------+
              |  otlp :4317 |            |  filelog      |
              |  otlp :4318 |            |  journald     |
              |  jaeger    |            |  prometheus   |
              |  zipkin    |            |  hostmetrics  |
              |  kafka     |            |  k8sobjects   |
              |  otlphttp  |            |  k8s_cluster  |
              +-------------+            +---------------+
                     |                          |
                     v                          v
                +------+                   +------+
                | pdata|                   | pdata|
                +------+                   +------+
                     |                          |
                     +-----------+--------------+
                                 |
                                 v
                          Processors ...

The receiver decides the wire format and the parsing work; the rest of the pipeline sees the same pdata regardless of source. That uniformity is the property that lets a single loki exporter ship logs from otlp, filelog, and journald without any per-source conditional logic.

Why a sysadmin cares

The receiver is the place where data enters the platform. Three failure shapes originate here.

1. The receiver that was never declared. The platform team configured otlp in the pipeline but the application does not speak OTLP; the team forgot to add filelog for the file-based logs. Symptom: the file-based logs never arrive.
2. The receiver that is declared but bound to the wrong interface. The otlp receiver listens on 0.0.0.0 but the host firewall blocks the port; the applications connect to a different collector. Symptom: traces from this host do not arrive; the collector self-metrics show zero accepted.
3. The receiver that scrapes a target it should not. The prometheus receiver accidentally scrapes a Kubernetes service that has 50,000 metrics endpoints. Symptom: the collector heap grows; process_runtime_total_alloc_bytes climbs until the kernel OOM-kills the process.

The receiver choice is the cheapest decision to get right and the most expensive to get wrong. The wrong receiver costs a missed signal; the right receiver costs a line of YAML.

How it works

Wire receivers

Wire receivers accept telemetry from a network endpoint. The collection mode is push; the source initiates the connection.

• otlp is the protocol-native receiver for the collector. It accepts OTLP over gRPC (0.0.0.0:4317) and HTTP (0.0.0.0:4318). The OTel SDK in the application initialises a gRPC or HTTP client; the client connects to the receiver; the receiver parses the protobuf or JSON payload into pdata.
• otlphttp is a separate receiver that accepts OTLP over HTTP/JSON only. Useful when a load balancer in front of the collector terminates TLS and re-encrypts as HTTP.
• jaeger accepts the Jaeger Thrift and gRPC protocols on the legacy Jaeger ports (14250, 14268, 9411). Useful when migrating a fleet off the Jaeger agent. New fleets should export OTLP, not Jaeger.
• zipkin accepts the Zipkin v1 JSON and v2 thrift/protobuf formats. Same role as jaeger for Zipkin-instrumented fleets.
• kafka consumes telemetry from Kafka topics. Used when the fleet already publishes to Kafka and the collector is downstream of the bus.

Local receivers

Local receivers scrape or tail a source on the host. The collection mode is pull; the receiver initiates the read.

• filelog tails files from the host filesystem. The closest equivalent to Promtail’s scrape_configs. The receiver tracks file position in a checkpoint store backed by the file_storage extension; the receiver resumes from the checkpoint after a restart.
• journald tails journald entries from the systemd journal. Equivalent to journalctl -f but as a streaming receiver. The receiver uses the systemd journal API; it requires the host to be running systemd.
• hostmetrics scrapes host-level metrics from /proc, /sys, and the cgroup filesystem. CPU, memory, disk, network, load, paging. The receiver is the collector-native replacement for node_exporter when the fleet already runs the collector.
• prometheus scrapes Prometheus-formatted metrics endpoints. The receiver walks a list of targets declared in the configuration or discovered through a service or pod association in Kubernetes.
• k8sobjects watches the Kubernetes API for object changes and emits each change as a log record. Useful for capturing pod lifecycle events into Loki.

How to configure it

Three receivers for the most common sources, annotated.

# /etc/otelcol/config.yaml

receivers:
  # 1. OTLP from OTel-instrumented applications.
  #    Binds to localhost in agent mode, to a private
  #    interface in gateway mode. The two protocols (gRPC
  #    and HTTP) can share the same config block.
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: localhost:4318

  # 2. Tail application log files.
  #    The include list is a glob; the operators chain parses
  #    each line into structured fields. Storage is required
  #    for the checkpoint; the file_storage extension must be
  #    declared and wired in service.extensions.
  filelog:
    include:
      - /var/log/app/*.log
    start_at: end
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<level>\S+) (?P<msg>.*)$'
      - type: move
        from: body
        to:   attributes

  # 3. Host metrics. The collector scrapes /proc and /sys
  #    itself; no separate exporter is needed.
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory: {}
      disk: {}
      filesystem: {}
      network: {}
      load: {}
      paging: {}

  # 4. Prometheus pull. The receiver walks the static list
  #    of targets; in Kubernetes the service/pod associations
  #    generate the target list dynamically.
  prometheus:
    config:
      scrape_configs:
        - job_name: app
          scrape_interval: 30s
          static_configs:
            - targets: [localhost:9100]

Five arguments that recur in production tuning.

• start_at on filelog — beginning re-reads every file from byte zero on restart; end starts at the current end of file. The production value is end unless the use case is one-shot log replay.
• collection_interval on hostmetrics — the scrape interval. The default is 10 seconds; 30 seconds is enough for host-level metrics and halves the storage cost.
• scrape_interval on prometheus — the scrape interval. Match it to the receiver’s collection_interval for consistency.
• max_recv_msg_size_mib on otlp/grpc — the largest protobuf message the receiver accepts. The default is 4 MiB; traces with large span batches need 16 MiB or more.
• encoding on filelog — the file encoding (utf-8, utf-16le, gbk, etc.). The default is utf-8; UTF-16 log files are the silent failure shape.

How to validate it

Validation is a parse-check plus a runtime check of the receiver counters.

# CONFIGURATION: parse-check against the schema.
otelcol validate --config=/etc/otelcol/config.yaml

# READ-ONLY: confirm every receiver is accepting data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="filelog"} 1872
otelcol_receiver_accepted_log_records{receiver="otlp"} 421
otelcol_receiver_accepted_metric_points{receiver="hostmetrics"} 89

# READ-ONLY: confirm no receiver is refusing data.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_refused

# (empty output = healthy; a non-empty value means the
#  memory_limiter is refusing data)

The receiver-side diagnostic follows three steps.

1. The process is running. pidof otelcol returns a PID; the agent log shows no Start failures.
2. The receiver is accepting. otelcol_receiver_accepted_* climbs over time.
3. The receiver is not refusing. otelcol_receiver_refused_* is zero or near zero; a non-zero value means the memory_limiter is rejecting the receiver’s batches.

If the receiver is running but receiver_accepted is flat at zero, the source is not reaching it. Check the bind address, the firewall, and the source-side configuration in that order.

How it can fail

Six failure modes specific to receivers.

1. The filelog receiver that never recovered its position. The filelog receiver was configured with start_at: beginning and no file_storage extension. Symptom: after restart, the receiver re-reads every file from byte zero; Loki receives duplicates; the metric otelcol_receiver_accepted_log_records shows a step on every restart.
2. The OTLP receiver that never saw a connection. The receiver binds to localhost:4317 in agent mode; the application is configured with OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317. The application and the collector cannot resolve each other. Symptom: otelcol_receiver_accepted_* is flat at zero; the application log shows connection-refused.
3. The prometheus receiver that scraped the world. The receiver was given a target list with 50,000 endpoints. The collector heap grows; the memory_limiter begins refusing. Symptom: otelcol_receiver_refused_metric_points climbs; process_runtime_total_alloc_bytes climbs; the kernel OOM-kills the process.
4. The hostmetrics receiver that ran as root. The receiver needs read access to /proc and /sys. A container image that drops privileges cannot read those paths. Symptom: otelcol_receiver_accepted_metric_points is zero; the agent log shows permission-denied.
5. The journald receiver that read no journal. The receiver was configured with units: [sshd] but the host runs sysvinit, not systemd. Symptom: otelcol_receiver_accepted_log_records is flat at zero; the agent log shows journal-not-found.
6. The k8sobjects receiver that watched too much. The receiver was configured with no objects filter; the collector emits every pod, service, and configmap change as a log record. Symptom: log volume spikes; the Loki cost climbs; the platform team gets a budget alert.

How to troubleshoot it

When a receiver is not behaving, the diagnostic order matters.

1. Read the receiver-specific counter. Find the receiver name in otelcol_receiver_accepted_*. A flat zero is the signature of “receiver is running, source is not reaching it”.
2. Confirm the source is reachable. For wire receivers, ss -tlnp shows whether the listener is bound. For local receivers, the file path or cgroup path must exist and be readable by the collector user.
3. Check the firewall. A wire receiver on 0.0.0.0 is still invisible if the host firewall blocks the port or the network policy in Kubernetes drops the traffic.
4. Tail the agent log at Start. A receiver that fails to Start logs the reason with a line number and the offending argument. The first error is usually the only one.
5. Test with a known input. Send a known OTLP request with otelcli or curl; check that receiver_accepted_* ticks. If the counter ticks, the receiver is fine; the issue is upstream.

Security implications

Receivers are the most permissive component in the collector. The defaults are local-only; production deployments often relax them in ways that need an audit.

• Bind to localhost in agent mode. The OTLP, Zipkin, and Jaeger receivers default to 0.0.0.0. In agent mode, change the bind to localhost. In gateway mode, restrict the listener with a NetworkPolicy.
• Authentication on the wire receivers. The bearertokenauth and basicauth extensions provide client-side authentication helpers; the OTLP receiver supports TLS but does not enforce authentication on its own. Add a reverse proxy if mutual TLS is required.
• Filesystem access. The filelog, journald, and hostmetrics receivers read whatever the collector process can read. Run the process as a dedicated user with read access to the intended paths and no more.
• Log content. The filelog receiver reads file content verbatim. Sensitive data in the log path (credentials, PII) is captured into pdata and shipped downstream. The redaction processor (or attributes with action: delete) strips it before the exporter.

Performance implications

The receiver is the cheapest part of the pipeline. The cost appears at scale.

• OTLP receiver. The gRPC and HTTP servers are goroutine-based; the cost is roughly proportional to the number of concurrent streams. A receiver that accepts thousands of streams holds thousands of goroutines.
• filelog receiver. The receiver tails files; the cost is the cost of reading and parsing. A regex operator with a backtracking pattern can dominate CPU. Use anchored regex.
• hostmetrics receiver. The receiver scrapes /proc and /sys on every collection interval. A 10-second interval is cheap; a 1-second interval can dominate CPU on large hosts.
• prometheus receiver. The cost is the cost of scraping each target. A target list with thousands of endpoints is expensive; the cardinality cost is paid downstream.
• k8sobjects receiver. The receiver watches the API server; the cost is the cost of the watch and the per-change emission. Use objects filters aggressively.

Production guidance

• Bind OTLP receivers to localhost in agent mode. In gateway mode, restrict with NetworkPolicy. The default 0.0.0.0 is too permissive for a production host.
• Set start_at: end on filelog receivers. A receiver that starts at the beginning on restart duplicates every line. The right value is end for steady state; beginning is for one-shot replay.
• Declare the file_storage extension for filelog and journald. Without it, the checkpoint is in-memory and every restart re-reads the file from byte zero.
• Filter k8sobjects aggressively. Watch only the objects you intend to log. Every watched object becomes a log line on every change.
• Match scrape intervals to the metric resolution you need. A 10-second interval on host metrics is overkill; 30 seconds halves the storage cost for almost no loss in dashboard usefulness.

Verification

You should now be able to answer:

• What is the difference between a wire receiver and a local receiver, and which one is push?
• Which receiver should you use for an OTel-instrumented application that exports OTLP?
• Which receiver should you use for a service that writes logs to /var/log/app/*.log?
• Why does the filelog receiver need a file_storage extension in steady state?
• What does a flat-zero otelcol_receiver_accepted_* counter tell you?

Quiz