DaemonSet use cases — log collectors, CNI agents, monitoring, and more

The DaemonSet pattern is the answer to a specific question: “what workload must exist on every node for the cluster to function?” The canonical answers are well-defined: log shippers, CNI agents, monitoring exporters, service-mesh data planes, and storage fabrics. This lesson walks each one and the configuration that makes it work in production.

Log collectors

The log collector reads container logs from /var/log/containers (CRI-O, containerd) or /var/lib/docker/containers (Docker) on every node and ships them to a central store.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: Exists
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.2
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - name: var-log
          mountPath: /var/log
          readOnly: true
        - name: var-lib-containers
          mountPath: /var/lib/containers
          readOnly: true
        - name: var-log-pods
          mountPath: /var/log/pods
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        securityContext:
          runAsNonRoot: true
          runAsUser: 10000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
      volumes:
      - name: var-log
        hostPath:
          path: /var/log
          type: Directory
      - name: var-lib-containers
        hostPath:
          path: /var/lib/containers
          type: DirectoryOrCreate
      - name: var-log-pods
        hostPath:
          path: /var/log/pods
          type: DirectoryOrCreate
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      terminationGracePeriodSeconds: 30

The configuration choices that matter in production:

• hostPath for /var/log and /var/lib/containers — these are the source directories; the DaemonSet must read them.
• readOnly: true — the log collector never writes to host paths; it only reads.
• runAsNonRoot and readOnlyRootFilesystem: true — minimum privileges. The log collector does not need root.
• tolerations: operator: Exists — runs on every node, including control-plane.
• Modest resource requests — 100m CPU, 128Mi memory. The cluster-wide cost is node_count × 100m CPU.

flowchart LR
    N1[node-01] --> P1[fluent-bit]
    N2[node-02] --> P2[fluent-bit]
    P1 -->|logs| LB["Load Balancer / Forward"]
    P2 -->|logs| LB
    LB --> S["Central log store<br/>Loki / Elasticsearch"]

CNI agents

The CNI agent configures pod networking on each node. Cilium, Calico, and other CNIs run a per-node agent that communicates with the control-plane component and the kernel networking stack.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cilium
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cilium
  template:
    metadata:
      labels:
        app: cilium
    spec:
      hostNetwork: true
      serviceAccountName: cilium
      tolerations:
      - operator: Exists
      priorityClassName: system-node-critical
      containers:
      - name: cilium-agent
        image: quay.io/cilium/cilium:v1.16
        securityContext:
          privileged: true   # needs BPF / kernel access
        volumeMounts:
        - name: bpf-maps
          mountPath: /sys/fs/bpf
          mountPropagation: Bidirectional
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
      volumes:
      - name: bpf-maps
        hostPath:
          path: /sys/fs/bpf
          type: DirectoryOrCreate
      - name: lib-modules
        hostPath:
          path: /lib/modules

The CNI agent requires:

• privileged: true for BPF / kernel module access. This is one of the few legitimate uses of privileged mode.
• hostNetwork: true to manipulate node-level routing.
• priorityClassName: system-node-critical so the Pod is not pre-empted.
• /sys/fs/bpf mount for BPF maps (Cilium) or /proc access (Calico).

Monitoring exporters

node-exporter is the canonical monitoring exporter. It exposes Prometheus metrics about the host (CPU, memory, disk, network, kernel stats).

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      tolerations:
      - operator: Exists
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.8.0
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          mountPropagation: HostToContainer
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

The configuration choices:

• hostPID: true to read process names from /proc.
• hostNetwork: true and hostPort: 9100 so the exporter is reachable on the node’s IP.
• Read-only mounts for /proc, /sys, /. node-exporter reads only.

Service-mesh data planes

Istio runs an Envoy proxy as a sidecar (per-Pod) by default, but alternative patterns deploy the Envoy as a per-node DaemonSet (the “ambient mesh” model or simpler sidecar replacement):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: istio-proxy
spec:
  selector:
    matchLabels:
      app: istio-proxy
  template:
    spec:
      hostNetwork: true
      containers:
      - name: istio-proxy
        image: gcr.io/istio-release/proxyv2:1.22
        # configures iptables for transparent interception

The proxy requires NET_ADMIN capability for transparent traffic interception. Production meshes typically use a sidecar pattern (per-Pod) for stronger isolation, but the DaemonSet pattern is valid for simpler deployments.

Storage fabrics

Ceph, GlusterFS, and Rook-Ceph run a per-node storage agent that provides the local cache and client connection to the distributed storage:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rook-ceph
spec:
  selector:
    matchLabels:
      app: rook-ceph-agent
  template:
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: rook-ceph-agent
        image: rook/ceph:v1.14
        securityContext:
          privileged: true
        volumeMounts:
        - name: dev
          mountPath: /dev
        - name: sys-bus
          mountPath: /sys/bus

The storage agent requires direct block-device access (/dev), privileged mode, and bus access for hot-plug.

The cluster-wide cost

A 100-node cluster with all four DaemonSets reserves ~45 CPU and ~96Gi memory cluster-wide. This is invisible on a 10-node dev cluster; on a production 1000-node cluster, it is a major line item in capacity planning.

Quiz

Production discipline

• Pin image tags. A latest tag on a DaemonSet is a cluster-wide surprise release.
• Set tight resource requests. A DaemonSet’s request is reserved on every node, even when the workload is idle.
• Use priorityClassName: system-node-critical. A DaemonSet that fails admission under pressure is the cluster’s CNI / monitoring / logging down.
• Verify the count. A dashboard alert that fires on kube_daemonset_status_desired_number_scheduled != kube_daemonset_status_current_number_scheduled catches drift quickly.
• Test the privilege boundary. A privileged DaemonSet must be reviewed like any other privileged workload. Verify the image source, the tag, the SBOM, and the vulnerability posture.

The DaemonSet pattern is well-defined and well-tested in production. The failure modes are silent (a node without a DaemonSet is observably the same as a node with a broken DaemonSet). Operators who run DaemonSets verify the count, not just the readiness.