OTel collectors and agents - the deployment model · Linux

OTel has two deployment modes: agents (per host) and gateways (central). This lesson covers both, and the common agent/gateway pattern.

Agent / Gateway pattern

Hosts                                       Gateway
+--------+                                +----------+
| App    |---OTLP (gRPC or HTTP)--------->| OTel      |---Prometheus
| OTel   |                                | Gateway   |---Loki (OTLP)
| agent  |                                |          |---Tempo (OTLP)
+--------+                                +----------+
+--------+
| App    |---OTLP
| OTel   |
| agent  |
+--------+

Agent (per host): receives OTLP from applications, batches, ships to gateway. Reduces network calls and adds local buffering.
Gateway (central): receives from many agents, processes, exports to backends. Centralised configuration and routing.

OTel agent

The agent runs on every host:

# /etc/otel-collector/agent.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

# `storage: file_storage` below refers to this extension by name. Declare it
# AND list it under service.extensions, or the collector refuses to start with
# "extension \"file_storage\" is not configured".
extensions:
  file_storage:
    directory: /var/lib/otelcol/sending_queue

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

exporters:
  otlp:
    endpoint: gateway:4317
    tls:
      insecure: false
      # ca_file is what verifies the GATEWAY's certificate. Without it the
      # collector falls back to the system trust store, which does not
      # contain your internal CA - so every export fails with
      # "x509: certificate signed by unknown authority", and mTLS with a
      # public CA in the store would verify the wrong peer.
      ca_file: /etc/ssl/internal-ca.pem
      cert_file: /etc/ssl/agent.pem      # this agent's client certificate
      key_file: /etc/ssl/agent.key
    # Telemetry is not worth losing during the incident you need it for.
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0                # 0 = retry indefinitely
    sending_queue:
      enabled: true
      storage: file_storage              # survives a collector restart

service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

The agent:

Receives OTLP from local applications.
Batches for efficiency.
Ships to the central gateway.

OTel gateway

The gateway runs centrally:

# /etc/otel-collector/gateway.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  otlphttp/loki:
    logs_endpoint: http://loki:3100/otlp/v1/logs
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]

The gateway:

Receives from agents.
Limits memory.
Routes to backends per pillar.

The jaeger and loki exporters no longer exist

Older tutorials - and older versions of this page - end the trace pipeline with a jaeger exporter and the log pipeline with a loki exporter. Both components have been removed from otelcol-contrib. A config that names them does not degrade gracefully; the collector exits at start-up with an unknown-type error and the whole gateway is down.

Both backends now speak OTLP natively, so the generic exporters are the replacement:

Removed	Use instead	Why
`jaeger`	`otlp/tempo` (or `otlp/jaeger`)	Jaeger accepts OTLP on 4317; the proprietary 14250 protocol was retired
`loki`	`otlphttp/loki` with `logs_endpoint`	Loki exposes an OTLP endpoint at `/otlp/v1/logs`

Note the logs_endpoint key. otlphttp normally derives /v1/logs from endpoint, but Loki serves its OTLP handler under an /otlp prefix, so the full path must be given explicitly or every log export returns 404.

The type/name form (otlp/tempo) is how you define two instances of the same component type. The part before the slash is the type; the part after is a label of your choosing. Pipelines must reference the full name, slash included.

Validate before you deploy - the failure is a start-up failure, not a runtime warning:

otelcol validate --config /etc/otel-collector/gateway.yaml

Host receiver

The host receiver scrapes host metrics (like node_exporter):

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      disk:
      network:
      filesystem:
      load:
      processes:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      exporters: [prometheus]

The receiver key is hostmetrics: and its only settings are collection_interval and scrapers. There is no type: key inside a receiver - the map key is the type. Adding one is an unknown-field error and the collector will not start.

This gives the same host metrics as node_exporter, in OTLP format, so it can replace node_exporter on a fleet already running the agent.

Validate first, overlap second, remove node_exporter last

Removing node_exporter as step one is the mistake. Do it in this order:

Validate the config with the binary that will consume it: otelcol validate --config /etc/otel-collector/agent.yaml. A generic YAML linter is not enough - the collector uses Go yaml.v3 and rejects things a lenient loader accepts.
Deploy on one host and confirm the unit is active (running), not restarting.
Point Prometheus at the collector endpoint and update the dashboards to the OTel metric names.
Run node_exporter and the collector in parallel until the new dashboards and alerts have fired at least once on real data.
Only then remove node_exporter.

If you remove node_exporter first and the collector fails to start, you lose metrics, logs and traces at the same instant, because the agent is the single telemetry path in this architecture. That outage is self-concealing: there is no telemetry left to alert on the telemetry being down, and flat dashboards look exactly like a quiet night. Keep an alert that fires on the absence of host metrics, not only on bad values.

Bridging what you already run

Most estates do not start from zero. They already forward syslog and already scrape node_exporter. The Collector is built to terminate both, so adoption is additive rather than a migration.

receivers:
  # Terminate the rsyslog forwarding you already have.
  # Point rsyslog omfwd at this port; applications change nothing.
  syslog:
    tcp:
      listen_address: 0.0.0.0:5514
    protocol: rfc5424

  # Scrape the node_exporter fleet you already have.
  # This takes Prometheus scrape_config verbatim.
  prometheus:
    config:
      scrape_configs:
        - job_name: node
          scrape_interval: 30s
          static_configs:
            - targets: ['10.0.0.11:9100', '10.0.0.12:9100']

service:
  pipelines:
    logs:
      receivers: [otlp, syslog]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp, hostmetrics, prometheus]
      processors: [batch]
      exporters: [otlp]

prometheus is both a receiver and an exporter

The same component name appears on both sides of a pipeline and means opposite things:

Position	Direction	Use
`receivers: [prometheus]`	Collector pulls	Scrapes existing Prometheus targets such as node_exporter
`exporters: [prometheus]`	Collector is scraped	Exposes `/metrics` for your Prometheus server to pull

The gateway example above uses the exporter form. Seeing it there and concluding that prometheus is “an exporter” is the easy mistake — it is both, and the receiver form is the one you need to ingest a fleet you have not re-instrumented yet.

syslog lives in the contrib distribution (otelcol-contrib), not in the core binary. If your build has no syslog receiver, the Collector fails at startup with an unknown-type error — that is a distribution problem, not a configuration error.

Deployment patterns

Sidecar (Kubernetes pod): one OTel agent per pod.
DaemonSet (Kubernetes): one agent per node.
Host agent (VM/bare-metal): one agent per host.
Gateway (central): one or more per region/datacenter.

For most production, daemon-set on K8s or host agent on VMs, with one or more gateways per region.