Agents vs Sidecars

A 4,000-pod fleet ships logs to Loki. The on-call engineer looks at the distributor and sees 4,000 separate clients opening 4,000 separate connections. Connection churn is visible in the distributor logs. The fix is not a Loki tuning knob; it is a topology change. Every pod runs its own collector, the topology decides the cost.

This lesson compares the two patterns and the cost they impose at fleet scale. The Kubernetes terminology is in the lesson because the patterns are most often chosen in a Kubernetes context, but the trade-offs apply to any container orchestrator or any host fleet with mixed-density workloads.

What it is

An on-host agent runs once per host (or once per Kubernetes node, in a DaemonSet). It tails files on the local filesystem or accepts push-style writes from co-located processes, then ships the assembled batches to Loki.

A sidecar runs once per workload, alongside the application container in the same pod. It reads from a shared emptyDir or a local socket, transforms, and ships. Each application gets its own collector instance.

The two patterns are not mutually exclusive. A common production shape is a DaemonSet agent for node-level logs (kubelet, container runtime, audit) plus sidecars only for the workloads that need per-pod transforms.

Why a sysadmin cares

The decision is irreversible at fleet scale and the cost shows up on the bill at the end of the month. Three failure shapes appear when the pattern is chosen by default rather than deliberately:

1. The sidecar fan-out. Every pod running its own collector multiplies the connection count to Loki by the pod count. A 4,000-pod fleet means 4,000 keepalive connections to the distributor. The distributor’s per-tenant connection cap becomes the limit, not the storage capacity.
2. The agent that misses per-pod context. A DaemonSet agent tails /var/log/containers/<pod>.log and ships it. It works. But the moment a workload needs a custom parse, a redaction rule, or a label that lives only in the application’s view, the agent cannot see it without per-pod config drift.
3. The sidecar that cannot survive eviction. A sidecar in a pod is part of the pod. When the kubelet evicts the pod for node pressure, the sidecar’s buffer goes with it. Anything in memory is lost; anything on the sidecar’s emptyDir survives only as long as the pod does. The on-host agent stays.

The trade-off is operational simplicity (agent) against per-pod fidelity (sidecar). The cost difference is roughly an order of magnitude in RAM and connection count.

How it works

The two patterns differ in three places: the lifecycle boundary, the configuration boundary, and the connection boundary.

On-host agent (DaemonSet)
-------------------------
   +-------------------+
   | node-1            |
   |  +-------------+  |     shared connection
   |  | alloy       |--+---------------------+
   |  +-------------+  |                     |
   |  +-------------+  |                     v
   |  | pod A       |  |                +---------+
   |  | app + side  |--+---ignored--+   |  Loki   |
   |  +-------------+  |            |   +---------+
   |  +-------------+  |            |
   |  | pod B       |  |            |
   |  | app + side  |--+---ignored--+
   |  +-------------+  |
   +-------------------+

Sidecar per pod
---------------
   +-------------------+
   | node-1            |
   |  +-------------+  |   per-pod connection
   |  | pod A       |  |
   |  | app         |  |
   |  | alloy       |--+------------------+
   |  +-------------+  |                  |
   |  +-------------+  |                  v
   |  | pod B       |  |             +---------+
   |  | app         |  |             |  Loki   |
   |  | alloy       |--+-------------+
   |  +-------------+  |
   +-------------------+

In the on-host pattern, the agent tails the kubelet’s container log files (or /var/log/containers/*.log) for every pod on the node. It does not need to know what is inside the pod. In the sidecar pattern, the sidecar is a co-located container that shares an emptyDir volume with the application and reads the logs directly.

Resource accounting

Sidecars multiply by the pod count. Agents multiply by the node count. At fleet scale the comparison is:

The connection cap on Loki 3.x defaults to ingester_limits plus distributor-side flow control. A 4,000-pod sidecar deployment that opens 4,000 connections will hit the cap before it hits the storage limit.

How to configure it

Two minimal real configs. The on-host agent tails the kubelet’s container logs; the sidecar reads from a shared emptyDir.

On-host agent (DaemonSet)

The DaemonSet runs one Alloy pod per node. Each pod mounts the host’s /var/log and /var/lib/alloy and tails the container log directory:

// Per-node Alloy config; deployed as a DaemonSet.
// /etc/alloy/config.alloy

loki.source.kubernetes "cluster" {
  // The cluster label is constant per deployment; the
  // namespace, pod, container labels come from the kubelet
  // JSON envelope.
  cluster_name = "prod-eu-west-1"

  // Discover every namespace except kube-system by default.
  // Adjust to match what you actually want shipped.
  namespaces     = ["app", "infra", "data"]
  selector       = "{namespace!~\"kube-system|kube-public\"}"

  forward_to = [loki.process.cluster.receiver]
}

loki.process "cluster" {
  // Promote pod metadata to labels so LogQL can filter on them.
  stage.labels {
    values = {
      namespace = "namespace",
      pod       = "pod",
      container = "container",
    }
  }

  // Keep one tenant per environment.
  stage.static_labels {
    values = {
      tenant = "prod",
    }
  }

  forward_to = [loki.write.loki.receiver]
}

loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
  external_labels = { collector = "alloy-agent" }
}

The Kubernetes manifest for the DaemonSet is out of scope here (it lives in the Kubernetes module), but the config side is the same loki.source.kubernetes block the module returns to.

Sidecar per pod

The sidecar shares an emptyDir with the application and tails the application’s stdout stream. The application’s Deployment or StatefulSet defines both containers; the sidecar’s config is rendered from a ConfigMap:

// Per-pod Alloy config; deployed as a sidecar container.
// /etc/alloy/config.alloy

// Discover the application's log path from the shared volume.
local.file "app_logs" {
  filename = "/var/log/app/app.log"
}

loki.source.file "app" {
  targets    = local.file.app_logs.targets
  job        = "checkout"
  host       = constants.hostname

  forward_to = [loki.process.app.receiver]
}

loki.process "app" {
  // Promote application-specific structured fields.
  stage.json {
    expressions = {
      level   = "level",
      request = "request_id",
      user    = "user_id",
    }
  }

  stage.labels {
    values = {
      level = "level",
    }
  }

  // Redact the request_id from labels to keep cardinality bounded.
  stage.template {
    source   = "request_id"
    template = "{{ .request }}"
  }

  forward_to = [loki.write.loki.receiver]
}

loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
  }
  external_labels = { collector = "alloy-sidecar" }
}

The two configs look similar. The differences are the source component (kubernetes versus file), the labelset, and the blast radius: the sidecar’s lifecycle is the pod’s lifecycle; the agent’s lifecycle is the node’s.

How to validate it

For the on-host agent:

# CONFIGURATION: validate the config before applying.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T12:01:11Z level=info msg="config valid"

# READ-ONLY: confirm the DaemonSet rolled out to every node.
kubectl -n observability get ds alloy

NAME    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
alloy   24        24        24      24           24

# READ-ONLY: pick one node and check the agent metrics.
kubectl -n observability exec -it ds/alloy -- \
  curl -s http://localhost:12345/metrics | grep loki_source_kubernetes_pods

loki_source_kubernetes_pods{namespace="app"} 312
loki_source_kubernetes_pods{namespace="infra"} 84

# READ-ONLY: confirm lines are arriving in Loki with the kubelet
# labels stamped on them.
logcli query --since=2m \
  '{collector="alloy-agent", namespace="app"} |~ "checkout-failed"' \
  --addr=https://loki.internal.example.com

For the sidecar:

# READ-ONLY: pick a pod with the sidecar and check its metrics.
kubectl -n app exec -it deploy/checkout -c alloy-sidecar -- \
  curl -s http://localhost:12345/metrics | grep loki_write

loki_write_sent_entries_total 18753
loki_write_dropped_entries_total 0
loki_write_remote_write_errors_total 0

# READ-ONLY: confirm the connection count to Loki grew by the
# pod count, not by the node count. From the Loki side:
curl -s -u admin:$(cat /etc/loki/admin) \
  https://loki.internal.example.com/metrics | grep loki_distributor_ingester_clients

If loki_distributor_ingester_clients is roughly equal to the node count, the agent pattern is in effect. If it is roughly equal to the pod count, the sidecar pattern is in effect.

How it can fail

Five failure modes that are distinct to the two patterns.

1. The DaemonSet that did not roll. A node is cordoned but the DaemonSet controller has not rescheduled. The agent on that node is still running but the workload is gone. Symptom: kubectl get ds shows DESIRED 23 and CURRENT 24. The extra pod is on the cordoned node.
2. The sidecar with no shared volume. The Deployment declares two containers but the volumeMounts are missing on one. The sidecar reads from /var/log/app/app.log which does not exist inside the sidecar’s container. Symptom: loki_source_files_failed_total climbing on the sidecar; loki_source_file_target_last_parsed_timestamp_seconds frozen.
3. The sidecar evicted with the pod. Node pressure triggers a pod eviction. The kubelet destroys the pod; the sidecar’s memory buffer goes with it. Symptom: a window of logs missing in Loki; kube_pod_container_status_terminated_reason shows Evicted.
4. The agent without host mount. The DaemonSet config mounts only /var/lib/alloy, not /var/log. The agent starts and reports zero targets. Symptom: loki_source_kubernetes_* counters all zero; the /metrics endpoint is otherwise healthy.
5. The connection-cap blowup. A sidecar fleet exceeds the Loki distributor’s per-tenant connection cap. Symptom: loki_distributor_ingester_clients capped at the limit, loki_write_remote_write_errors_total rising, individual sidecars retrying in lockstep.

How to troubleshoot it

When logs are missing in Loki for a specific workload, the diagnostic order depends on which pattern is in use.

For the on-host agent pattern:

1. Is the agent running on the workload’s node? kubectl -n observability get pods -o wide | grep <node>.
2. Is the kubelet writing the container’s log file? kubectl exec -it node-debug -- ls /var/log/containers/.
3. Is the agent parsing it? On the agent’s host, curl -s http://localhost:12345/metrics | grep loki_source_kubernetes.
4. Is the agent shipping it? curl -s http://localhost:12345/metrics | grep loki_write.

For the sidecar pattern:

1. Is the sidecar running in the pod? kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}'.
2. Can the sidecar read the application logs? kubectl exec <pod> -c alloy-sidecar -- ls /var/log/app/.
3. Is the sidecar parsing? kubectl exec <pod> -c alloy-sidecar -- curl -s http://localhost:12345/metrics | grep loki_source_file.
4. Is the sidecar shipping? kubectl exec <pod> -c alloy-sidecar -- curl -s http://localhost:12345/metrics | grep loki_write.

The two diagnostic trees diverge at step 1 and 2. From step 3 onwards they are identical because the source component abstracts the difference.

Security implications

The patterns have different attack surfaces.

• On-host agent. A single process with read access to every container log on the node. A compromise exposes every workload’s stdout. The blast radius is the node. Run as a low-privilege service account; restrict hostPath mounts to the minimum (/var/log, /var/lib/alloy); drop all Linux capabilities.
• Sidecar. Each workload gets its own collector with its own credentials and its own network policy. A compromise exposes one workload’s stdout. The blast radius is the pod. But the surface is multiplied by the pod count: every sidecar has its own copy of the config, its own CA bundle, its own secrets mount, and its own failure modes.

In Kubernetes, the sidecar pattern also amplifies the surface of the secrets in the cluster. The password_file mount appears in every pod’s spec; rotating it requires a rollout of every workload. The agent pattern rotates one secret on the DaemonSet’s ConfigMap.

Performance implications

• On-host agent. One process per node, scaling with the node count. RAM ~120 MiB, CPU ~50 millicores per node at modest line rates. Connections to Loki: one per node.
• Sidecar per pod. One process per workload, scaling with the pod count. RAM ~120 MiB per pod, CPU ~50 millicores per pod. Connections to Loki: one per pod.

A 4,000-pod fleet running 1,000 nodes: 1,000 collector instances in the agent pattern; 4,000 in the sidecar pattern. The CPU and RAM cost scales linearly with the pod count for the sidecar pattern. The connection cost scales linearly too, and that is usually what trips first.

For workloads that need per-pod transforms (custom redaction, proprietary format), the sidecar pattern is justified. For everywhere else, the agent pattern is cheaper to run and cheaper to operate.

Production guidance

• Default to on-host. A DaemonSet agent handles ~90% of workloads. Reach for a sidecar only when the workload has a custom parse or redaction rule that the agent cannot apply.
• Right-size the sidecar’s requests and limits. RAM ~120 MiB, CPU ~100 millicores is a reasonable starting point for a modest-volume workload. Use VPA to learn the actual consumption; tune from there.
• Co-locate credentials. The sidecar’s password_file should be a projected token or a CSI-backed secret, not a copy of the same password in every pod’s Secret manifest. The rotation story is the same regardless of pattern, but the sidecar pattern multiplies the rotation work.
• Document the choice. The deployment manifest should declare in a comment why this workload is a sidecar and not an agent. The decision is not obvious to the next operator who looks at it.

Verification

You should now be able to answer:

• What is the resource cost difference between an on-host agent and a per-pod sidecar at fleet scale?
• When does the sidecar pattern earn its keep, and when is the on-host pattern the right answer?
• How does the symptom of a stuck sidecar differ from the symptom of a misrouted on-host agent?
• Why does the sidecar pattern amplify the secret-rotation work relative to the agent pattern?

Quiz