Choosing the Right Collector

A team runs 800 hosts across two data centres. The observability fleet is 800 Alloy DaemonSets and a single Loki cluster. The loki write path is healthy. The team considers a redesign: a central gateway in front of Loki to absorb batched writes and protect the backends from per-host spikes. The trade-off is non-trivial. The gateway is a new failure domain; the per-host agent is more reliable per host but generates more network traffic. The wrong answer is the one chosen without the trade-off in mind.

This lesson is the topology decision: where the collector runs, what each topology costs, and what each topology buys.

What it is

A shipping strategy is the topology that connects the sources of telemetry (applications, hosts, kernels) to the backends (Loki, Mimir, Tempo). Three topologies are common.

• Per-host agent - one collector process per host or node. In Kubernetes, the pattern is a DaemonSet: one pod per node. The agent tails the host’s logs, scrapes the host’s metrics, and forwards everything to the backends.
• Sidecar - one collector process per application pod. The sidecar shares the pod’s network namespace; the application exports OTLP to localhost:4317. The sidecar forwards to the backends.
• Central gateway - one or more collector processes per cluster, receiving from many agents. The gateway is the central pipeline; it batches, filters, and ships to the backends. The agents fan in; the gateway fans out.

   Per-host agent                 Sidecar                Central gateway
   ---------------                ---------              ----------------

   +-----------+                  +-----------+           +-----------+
   |  host     |                  |  app pod  |           |  host     |
   | +-------+ |                  | +-------+ |           | +-------+ |
   | | alloy | | ---> Loki        | | otel  | |           | | alloy | | ---> Gateway ---> Loki
   | +-------+ |                  | | sidecar| | ---> Loki| +-------+ |
   +-----------+                  | +-------+ |           +-----------+
                                  +-----------+

The three topologies are not mutually exclusive. A common production pattern is a per-host agent that tails system logs and scrapes host metrics, plus a sidecar that tails application logs and receives OTLP, plus a central gateway that batches and ships.

Why a sysadmin cares

The topology decision is one of the most expensive decisions in the observability stack. Three failure shapes appear when the topology is chosen by default rather than by analysis.

1. The sidecar fleet that doubled the node count. A team added a sidecar to every application pod for “clean separation”. The cluster grew by 50%; the scheduler started evicting pods for fit; the on-call rotation spent a week tuning requests and limits. The sidecar was the right answer for OTLP-instrumented apps but the wrong answer for log-tailing; the agent DaemonSet should have tailed the host’s logs while the sidecar handled OTLP.
2. The central gateway that became a single point of failure. A team moved every agent behind a single collector gateway. The gateway was a single replica. A rollout broke the gateway; the entire fleet stopped shipping. The fix was a StatefulSet with three replicas behind a load balancer; the lesson was that the gateway is a new failure domain and must be sized accordingly.
3. The per-host agent that overwhelmed the backends. A team ran 800 agents, each shipping to Loki directly. The ingest path spiked on every node restart; Loki’s distributor throttled; the agents saw 429 Too Many Requests; the applications saw back-pressure. The fix was a central gateway that batches and rate-limits.

How it works

Per-host agent

The agent runs as a DaemonSet (Kubernetes) or a host-level service (bare metal, VMs). It tails the local log files, scrapes the local metrics endpoints, and accepts OTLP from the local applications. It ships to the backends directly or to a central gateway.

# Kubernetes DaemonSet (excerpt)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: alloy
spec:
  selector:
    matchLabels:
      name: alloy
  template:
    metadata:
      labels:
        name: alloy
    spec:
      hostNetwork: true
      containers:
        - name: alloy
          image: grafana/alloy:latest
          args:
            - run
            - /etc/alloy/config.alloy
          volumeMounts:
            - name: logs
              mountPath: /var/log
              readOnly: true
            - name: config
              mountPath: /etc/alloy
      volumes:
        - name: logs
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: alloy-config

The hostNetwork: true setting is critical for the agent to reach localhost:9100 (node_exporter) and the local log files. Without it, the pod sees its own network namespace and cannot reach the host services.

Sidecar

The sidecar runs as a second container in the application pod. The application exports OTLP to localhost:4317; the sidecar forwards to the backends or to a central gateway.

# Kubernetes pod with OTel Collector sidecar (excerpt)
apiVersion: v1
kind: Pod
metadata:
  name: checkout
spec:
  containers:
    - name: app
      image: checkout:latest
      env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://localhost:4317
    - name: otelcol
      image: otel/opentelemetry-collector-contrib:latest
      args:
        - --config=/etc/otelcol/config.yaml
      volumeMounts:
        - name: config
          mountPath: /etc/otelcol
  volumes:
    - name: config
      configMap:
        name: otelcol-sidecar-config

The sidecar shares the pod’s network namespace, so the application’s localhost:4317 reaches the sidecar directly. The sidecar’s resource cost is per-pod; the on-call cost is per-pod.

Central gateway

The gateway runs as a Deployment or StatefulSet in the cluster. It receives from many agents, batches, and forwards to the backends.

# Kubernetes Deployment (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otelcol-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otelcol-gateway
  template:
    metadata:
      labels:
        app: otelcol-gateway
    spec:
      containers:
        - name: otelcol
          image: otel/opentelemetry-collector-contrib:latest
          args:
            - --config=/etc/otelcol/gateway.yaml
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 2
              memory: 2Gi

The gateway is a StatefulSet when the file_storage extension backs the sending_queue; the StatefuleSet gives each replica a stable identity for the storage.

How to configure it

The three topologies differ in the receiver surface of the collector.

Agent-side: receive OTLP, tail logs

// /etc/alloy/config.alloy (agent)

loki.source.file "system" {
  targets = [{ __path__ = "/var/log/syslog", job = "system" }]
  forward_to = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint {
    url       = "https://gateway.observability.svc.cluster.local/loki/api/v1/push"
    tenant_id = "prod"
  }
}

Gateway-side: receive OTLP, batch, export

# /etc/otelcol/config.yaml (gateway)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
  batch:
    timeout: 5s
    send_batch_size: 16384

exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 10000

service:
  pipelines:
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [loki]

The gateway is the choke point; the queue is bounded; the backends see a smoothed ingest rate.

How to validate it

The validation shape is the same; the receiver surface differs.

# READ-ONLY: confirm the agent is shipping.
curl -s http://localhost:12345/metrics | grep loki_write_sent_entries_total

loki_write_sent_entries_total{component="loki.write.central"} 1872

# READ-ONLY: confirm the gateway is accepting from the fleet.
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted

otelcol_receiver_accepted_log_records{receiver="otlp",transport="grpc"} 421337

# READ-ONLY: confirm the gateway is exporting to the backend.
curl -s http://localhost:8888/metrics | grep otelcol_exporter_sent

otelcol_exporter_sent_log_records{exporter="loki"} 421337

The receiver count and the exporter count should match (modulo batching). A persistent gap means the gateway is the failure domain.

How it can fail

Five failure modes specific to topology choices.

1. The per-host agent with hostNetwork: false. A Kubernetes DaemonSet without hostNetwork: true cannot reach localhost:9100 (node_exporter) and the local log files. Symptom: the agent’s prometheus_scrape_targets counter is empty; loki_source_file_targets is empty; the agent ships nothing. The fix is to add hostNetwork: true.
2. The sidecar that shared the pod’s resources. A sidecar configured with no resources.limits consumed the entire pod’s memory under a downstream outage. Symptom: the application was OOM-killed; the on-call engineer concluded the application had a memory leak; the actual leak was in the sidecar’s sending_queue.
3. The gateway that became a single point of failure. A single-replica gateway was rolled; the rollout broke the receiver; the entire fleet stopped shipping. Symptom: otelcol_receiver_accepted is flat across all agents; otelcol_exporter_sent is flat across the gateway; Loki returns nothing.
4. The agent that overwhelmed the backends. 800 agents shipping to Loki directly caused the distributor to throttle. Symptom: loki_write_sent_entries_total is flat; loki_write_dropped_entries_total climbs; the agents see 429 Too Many Requests. The fix is a central gateway to batch and rate-limit.
5. The DaemonSet that left the sidecar behind. A team moved from sidecars to a DaemonSet, but some applications still exported OTLP to a sidecar that no longer existed. Symptom: the application’s logs vanished; the sidecar’s absence was silent because no collector was configured to log “OTLP received but no exporter”.

How to troubleshoot it

When telemetry is missing, the topology narrows the search.

1. Identify the closest collector. The application pod has a sidecar; the host has an agent; the cluster has a gateway. The closest collector is the one that should be receiving the data first.
2. Confirm the network path. kubectl exec into the application pod; curl the sidecar’s localhost:13133. If the health check passes, the sidecar is up. If it fails, the sidecar is the failure.
3. **Confirm the agent sees the host. kubectl exec into the DaemonSet pod on the same node; curl the agent’s localhost:12345/-/ready. If the readiness fails, the agent is the failure.
4. Confirm the gateway is exporting. curl the gateway’s localhost:8888/metrics. If otelcol_receiver_accepted climbs but otelcol_exporter_sent does not, the gateway is the failure.
5. Confirm the backends are accepting. curl Loki’s /ready. If Loki is not ready, the backends are the failure.

Security implications

The topology has a security shape.

• The agent. Exposes the debug and metrics endpoints on the host network. Bind to localhost in production; expose on the cluster network only with authentication.
• The sidecar. Shares the pod’s network namespace. The application’s OTLP traffic and the sidecar’s metrics endpoints share a localhost. The blast radius is the pod.
• The gateway. Exposes a wide receiver surface (OTLP on 4317 and 4318 by default). Restrict the listener with NetworkPolicy. The gateway is the choke point; an unauthenticated gateway is an open ingest surface.

Performance implications

The cost shape of each topology.

• Per-host agent. Roughly 150 MiB RAM and 50-100 millicores CPU per host at modest line rates. At 800 hosts, the fleet cost is 120 GiB RAM and 40-80 cores.
• Sidecar. Roughly 150 MiB RAM per pod. At 5,000 pods, the fleet cost is 750 GiB RAM. The cost scales with pod count, not host count.
• Central gateway. Roughly 2 GiB RAM per replica. At three replicas, the fleet cost is 6 GiB RAM. The cost is fixed regardless of pod count.

The cost trade-off is per-host vs per-pod vs per-cluster. The blast-radius trade-off is per-host vs per-pod vs per-cluster. The right answer is the topology whose blast radius matches the failure budget.

Production guidance

• Use a per-host agent for system logs and host metrics. The DaemonSet pattern is the right answer for tails of /var/log and scrapes of /metrics. The blast radius is one host.
• Use a sidecar only for OTLP-instrumented applications that need pod-level isolation. The blast radius is one pod. The cost is one process per pod.
• Use a central gateway to protect the backends. The gateway batches and rate-limits; the backends see a smoothed ingest rate. The blast radius is the gateway; size it accordingly (three replicas minimum).
• Document the topology. A topology that is not documented is a topology that is not understood. A topology that is not understood is the one that gets rolled out without a plan.

Verification

You should now be able to answer:

• What are the three common shipping topologies, and what blast radius does each one carry?
• Why does a per-host agent need hostNetwork: true in a Kubernetes DaemonSet?
• What is the on-host resource cost of Alloy at modest line rates, and how does it scale at the fleet level?
• Why must a central gateway be deployed as a StatefulSet with three replicas?

Quiz