The Collector Layer · Observability

The on-call engineer notices the in-flight metric for checkout is climbing. They open Prometheus and the metric is there. They open Tempo and the trace is not there. The collector is receiving the metric on the Prometheus scrape target but the OTLP receiver is rejecting the trace. The on-call engineer has a stack that is producing partial telemetry, and the partiality is not the workload’s fault.

The collector layer is where the contracts become enforcement. The platform team owns this layer. The application team contracts into it. When the collector is misconfigured, every workload downstream is dark.

What it is

The collector layer is the set of processes that receive telemetry from the workload, transform it, and forward it to the backend. In this course the two canonical options are:

Grafana Alloy — Grafana’s distribution of the OTel Collector with a flow-based configuration language. The same components, the same OTLP pipeline, the same exporters as upstream. Alloy adds a friendlier filesystem layout and a reload-on-SIGHUP pattern.
OpenTelemetry Collector — the upstream distribution. The otelcol binary plus a YAML configuration file. The reference for every other collector.

The collector is not a storage component. It does not retain data; it forwards it. The collector is the only component in the stack that sees every workload, and that property is what makes it the right place to enforce cardinality caps and attribution labels.

The third option, node_exporter, is technically a collector of host metrics, but it is treated as part of the workload layer in this course. The exporter is host-local; the collector is the platform-wide aggregation point.

Why a sysadmin cares

The collector is the platform team’s interface to the workload. The contract is:

The workload emits telemetry.
The collector accepts it, transforms it, and forwards it.
The backend stores it.

The contract is enforced in the collector. The platform team that owns the collector owns the cardinality budget, the attribution policy, the retry policy, and the buffering policy. The application team that owns the workload writes to the contract and trusts the collector to do the rest.

The five failure shapes that appear in a stack with a collector that is treated as a black box:

Receiver is configured but no exporter is connected. The collector accepts traffic but drops it on the floor. The workload sees “telemetry sent”; the backend sees nothing.
High-cardinality label flows through. The collector is the right place to drop a user_id label. With no transform processor, the label reaches the backend and the TSDB OOMs.
Buffer fills and drops batches. The collector is not configured for the actual workload volume. The exporter queue is full; the receiver drops batches. The metric otelcol_exporter_queue_size is at the limit.
Wrong pipeline written for the wrong signal. A metrics pipeline attached to a trace exporter. The backend stores nothing because the data type is wrong.
Collector is a SPOF. A single collector in agent topology is fine — it is per-host. A single collector in gateway topology that processes every workload in a region is not fine. Its failure takes the region dark.

The collector is the right place to design these out before production.

How it works

The mental model is a pipeline: receiver → processor → exporter. A single collector can run many pipelines, one per signal type, and the pipelines are independent.

                       Collector process
                       +-------------------+
  workloads --------> |  receivers         |
                       |    otlp            |
                       |    prometheus      |
                       |    loki (push)     |
                       |    host_metrics    |
                       +---------+---------+
                                 |
                                 v
                       +---------+---------+
                       |  processors       |
                       |    batch          |
                       |    memory_limiter |
                       |    attributes     |
                       |    filter         |
                       |    relabel        |
                       +---------+---------+
                                 |
                                 v
                       +---------+---------+
                       |  exporters        |
                       |    otlp/tempo      |
                       |    prometheus_rw   |
                       |    loki           |
                       +---------+---------+
                                 |
                                 v
                              backends

Each component has a single responsibility:

Receiver. Accepts a wire format. OTLP gRPC, OTLP HTTP, Prometheus scrape, Jaeger, Zipkin, Kafka, syslog, journald, and dozens of others.
Processor. Transforms a payload in place. The most useful in production are batch, memory_limiter, attributes (add/remove keys), filter (drop on predicate), resource (validate semantic conventions), transform (expression-based edits), and the OTel redaction processor for PII.
Exporter. Forwards to a backend. Prometheus remote_write, OTLP, Loki push, Kafka, file, debug.

The pipeline is data-type-scoped. A metrics pipeline has metric receivers, metric processors, and metric exporters. A trace pipeline has trace receivers, trace processors, and trace exporters. Mixing data types within a pipeline is a configuration error.

Under the hood

The collector is a Go binary. The binary loads a YAML configuration file and wires receivers, processors, and exporters into a graph. The graph is package-private; the operator only sees the configuration.

The dispatcher is the in-process queue. Each pipeline has a send_batch_size, a send_batch_max_size, and a timeout. The collector buffers batches in memory and ships them when the batch is full or the timeout fires. The memory_limiter processor is the bound — when in-memory queue exceeds spike_limit_mib, the receiver rejects new data and the workload sees backpressure.

The collector’s own self-telemetry is exposed on the configured service.telemetry.metrics.level endpoint. The metrics follow the OTel collector’s own semantic conventions:

otelcol_receiver_accepted_metric_points
otelcol_exporter_sent_metric_points
otelcol_exporter_queue_size

The difference between receiver_accepted and exporter_sent is the datapoints the collector still owes the backend. The platform team monitors this difference.

How to configure it

The OpenTelemetry Collector configuration is YAML. A real production config that handles all three signals with a memory limiter and a batch processor:

# /etc/otelcol/config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318

  prometheus:
    config:
      scrape_configs:
        - job_name: node
          scrape_interval: 15s
          static_configs:
            - targets: [localhost:9100]
              labels:
                cluster: prod-eu-west-1
        - job_name: app
          scrape_interval: 15s
          static_configs:
            - targets: [localhost:9100]
              labels:
                cluster: prod-eu-west-1

  filelog:
    include:
      - /var/log/containers/*.log
    operators:
      - type: json_parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

  batch:
    send_batch_size: 8192
    send_batch_max_size: 10000
    timeout: 200ms

  attributes/remove_user_id:
    actions:
      - key: user_id
        action: delete

  resource:
    attributes:
      - key: deployment.environment
        from_attribute: cluster
        action: insert

  filter/drop_debug:
    traces:
      span:
        - 'attributes["debug"] == true'

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

  otlp/tempo:
    endpoint: tempo.monitoring.svc:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000

  loki:
    endpoint: http://loki.monitoring.svc:3100/loki/api/v1/push
    labels:
      attributes:
        cluster: ""
        service_name: ""
    headers:
      X-Scope-OrgID: tenant-a

service:
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, attributes/remove_user_id, resource, batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/remove_user_id, resource, filter/drop_debug, batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [filelog]
      processors: [memory_limiter, attributes/remove_user_id, resource, batch]
      exporters: [loki]

The same topology in Grafana Alloy is expressed as a flow file. The components are identical; the configuration is linear instead of nested:

// /etc/alloy/config.alloy
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    metrics = [otelcol.processor.memory_limiter.default.input]
    traces  = [otelcol.processor.memory_limiter.default.input]
    logs    = [otelcol.processor.memory_limiter.default.input]
  }
}

otelcol.processor.memory_limiter "default" {
  check_interval = "1s"
  limit_mib      = 1024
  spike_limit_mib = 256

  output {
    metrics = [otelcol.processor.attributes.strip.input]
    traces  = [otelcol.processor.attributes.strip.input]
    logs    = [otelcol.processor.attributes.strip.input]
  }
}

otelcol.processor.attributes "strip" {
  action {
    key = "user_id"
    action = "delete"
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  send_batch_size = 8192
  timeout         = "200ms"

  output {
    metrics = [otelcol.exporter.prometheusremotewrite.default.input]
    traces  = [otelcol.exporter.otlp.tempo.input]
    logs    = [otelcol.exporter.loki.default.input]
  }
}

otelcol.exporter.prometheusremotewrite "default" {
  endpoint { url = "http://prometheus.monitoring.svc:9090/api/v1/write" }
}

otelcol.exporter.otlp "tempo" {
  client { endpoint = "tempo.monitoring.svc:4317" }
}

otelcol.exporter.loki "default" {
  endpoint {
    url = "http://loki.monitoring.svc:3100/loki/api/v1/push"
  }
}

The choice between OTel Collector and Alloy is a tooling choice, not a topology choice. Both run the same pipeline.

How to validate it

CONFIGURATION — validate the config before reload.

otelcol validate --config=/etc/otelcol/config.yaml
# (OpenTelemetry Collector Contrib 0.110.x)
# Validation successful. Component graph is well-formed.

alloy fmt /etc/alloy/config.alloy > /tmp/alloy.restyled
diff -u /etc/alloy/config.alloy /tmp/alloy.restyled
# alloy fmt normalises block ordering and formatting; review the diff before applying

SERVICE-IMPACT — reload the collector with a new config.

systemctl reload otelcol
# (otelcol-contrib service unit with ExecReload=/bin/kill -HUP $MAINPID)

alloy fmt /etc/alloy/config.alloy > /etc/alloy/config.alloy.new
mv /etc/alloy/config.alloy.new /etc/alloy/config.alloy
kill -HUP $(pidof alloy)

READ-ONLY — confirm the collector’s self-telemetry.

curl -sf http://localhost:8888/metrics | grep '^otelcol_receiver_accepted_metric_points'
# otelcol_receiver_accepted_metric_points{receiver="otlp",transport="grpc"} 12823

curl -sf http://localhost:8888/metrics | grep '^otelcol_exporter_sent_metric_points'
# otelcol_exporter_sent_metric_points{exporter="prometheusremotewrite"} 12823

READ-ONLY — confirm the receiver is accepting traffic.

nc -vz localhost 4317
# Ncat: Connected to 127.0.0.1:4317.

curl -sf http://localhost:4318/v1/traces -X POST -d '{}' | jq .
# A 400 response is the right outcome; the receiver is listening.

READ-ONLY — confirm an end-to-end trace is flowing.

curl -sf "http://tempo.monitoring.svc:3200/api/search?tags=service.name%3Dcheckout&limit=1" | jq '.traces | length'
# 1

READ-ONLY — confirm the cardinality cap is enforced.

curl -sf 'http://prometheus.monitoring.svc:9090/api/v1/query?query=count({__name__=~".+"}+on()grouping(user_id))' | jq .
# { "status": "success", "data": { "resultType": "vector", "result": [] } }
# Empty result confirms the user_id label was stripped at the collector.

How it can fail

Pipeline defined but no exporter attached. The receiver accepts datapoints. The processor transforms them. The pipeline ends with no destination. The otelcol_exporter_sent_* metric is zero. The on-call engineer sees no data in the backend.
Memory limiter under-sized. The collector’s limit_mib is set to 256. The workload’s burst fills the queue. The receiver rejects new data. The otelcol_refused_metric_points counter climbs.
Wrong data type in a pipeline. A traces pipeline receives a metrics payload. The collector logs the rejection but the operator does not see the log. The otelcol_receiver_failed_parse metric is the signal.
Exporter endpoint wrong. The remote_write URL points to the staging cluster. Production metrics silently disappear into the staging TSDB.
Filter processor drops everything. A filter rule with a predicate that is always true. The collector accepts data, the processor drops it, the exporter receives nothing. otelcol_processor_filter_dropped is at the same rate as otelcol_receiver_accepted_*.
Singleton gateway collector. A region-wide gateway collector crashes. Every workload in the region goes silent. The fix is multiple replicas or per-host agents.

How to troubleshoot it

The diagnostic order for “the collector is up but telemetry is missing”:

Is the collector process up? systemctl status otelcol or alloy and the self-telemetry endpoint http://localhost:8888/metrics.
Is the receiver accepting? The otelcol_receiver_accepted_* metric is the source of truth. If it is zero, the workload is not sending.
Is the processor dropping? The otelcol_processor_*_dropped metric family. The processor is the most likely culprit for a “data stops here” symptom.
Is the exporter sending? The otelcol_exporter_sent_* metric. If it is zero while the receiver is accepting, the processor is the suspect.
Is the queue filling? The otelcol_exporter_queue_size metric. If it is at the limit, the backend is the suspect.
End-to-end probe. Trigger a single synthetic trace from the workload and search for it in Tempo. The journey through the pipeline is observable.

Security implications

The collector is the platform’s most exposed telemetry endpoint. The receivers listen on the network. The exporters authenticate to the backend.

OTLP receivers. mTLS is the right choice in production. The plain-text 0.0.0.0:4317 listener is the development default. The production default is a network policy that limits the listener to the workload subnet.
Prometheus receiver. The scrape config is the auth boundary. The collector scrapes a known endpoint; the endpoint is on loopback or on a network policy.
Exporter credentials. The remote_write URL holds a credential. The credential is a secret. The collector reads it from the environment or a secret file with strict permissions.
Log file reads. The filelog receiver reads /var/log/containers/*.log. The collector runs as a service account with read access to that directory. The account does not have write access.
Self-telemetry. The service.telemetry.metrics.address: 0.0.0.0:8888 exposes the collector’s internal metrics. Bind to loopback in production.

Performance implications

The collector is in the request path. The four knobs:

Batch size. A larger batch uses more memory but reduces the per-call overhead. The default send_batch_size: 8192 is a reasonable starting point; raise it for high-volume workloads.
Memory limit. The memory_limiter is the upper bound on the collector’s RSS. The right setting is the lesser of 25% of the host’s memory or the host’s budget per service.
Queue size. The exporter’s sending_queue.queue_size is the upper bound on in-flight batches. At the limit, the receiver rejects new data.
Number of consumers. The sending_queue.num_consumers controls the per-exporter concurrency. The right setting is min(4, CPU/2).

The collector’s own CPU is bounded by the processors. The attributes and transform processors are CPU-bound. The batch and memory_limiter processors are memory-bound. The filter processor is fast.

Production guidance

Agent topology for most workloads. One collector per host. Failure domain is local. Routing config is distributed.
Gateway topology for central routing. When the routing config changes often, or when the workload cannot run a collector. Run multiple replicas. The gateway is a SPOF unless it is replicated.
Validate before reload. otelcol validate and alloy fmt are fast. The cost of a validation is small; the cost of a misconfigured pipeline is large.
Monitor self-telemetry. The otelcol_exporter_queue_size and otelcol_receiver_accepted_* metrics are alerting inputs.
Cardinality caps at the collector. Drop labels with unbounded values at the collector. The workload is the source; the collector is the gate.

Verification

You should now be able to answer:

What is the difference between agent and gateway collector topology, and which one is the right default?
What is the difference between a receiver, a processor, and an exporter?
Why is the collector the right place to enforce a cardinality cap?
What is the failure mode when a pipeline is missing an exporter?
What metric tells you the queue is full?