DaemonSets — one Pod per node for node-local agents

A DaemonSet is a controller that ensures one Pod runs on every node that matches its node selector. As nodes join the cluster, a Pod is created on each; as nodes leave, the Pod is deleted. This is the controller for cluster-wide node-local workloads — log collectors, CNI agents, monitoring agents, kube-proxy replacements — where the workload is meaningful only if it exists on every node.

What DaemonSets do

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
      - name: fluentbit
        image: fluent/fluent-bit:2.2
        volumeMounts:
        - name: var-log
          mountPath: /var/log
          readOnly: true
        - name: var-lib-containers
          mountPath: /var/lib/containers
          readOnly: true
      volumes:
      - name: var-log
        hostPath:
          path: /var/log
          type: Directory
      - name: var-lib-containers
        hostPath:
          path: /var/lib/containers
          type: Directory
      tolerations:
      - operator: Exists

The DaemonSet controller watches the node list and the DaemonSet’s selector. For every node that matches, the controller ensures exactly one Pod is scheduled onto it. If the node is later cordoned or removed, the Pod is deleted.

flowchart TB
    DS["DaemonSet: log-collector"]
    N1[node-01] --> P1[Pod on node-01]
    N2[node-02] --> P2[Pod on node-02]
    N3[node-03] --> P3[Pod on node-03]
    N4["node-04 - cordoned<br/>no Pod"]
    DS -.->|matches| N1
    DS -.->|matches| N2
    DS -.->|matches| N3
    DS -.->|does not match| N4

Compared to Deployments

A Deployment is a count of Pods. A DaemonSet is a count of nodes. The difference is fundamental: a Deployment’s replicas: 6 says “I want 6 Pods wherever they fit.” A DaemonSet says “I want 1 Pod on every matching node.”

Compared to StatefulSets

StatefulSets give stable identity and ordered lifecycle. DaemonSets give node locality. The two are orthogonal — a DaemonSet Pod has a random name suffix, not a stable ordinal. A StatefulSet with replicas: N does not ensure one per node; with replicas: 100 and 50 nodes, two Pods land on each node. A DaemonSet with the same selector produces 50 Pods, one per node.

What DaemonSets are correct for

The DaemonSet pattern is correct when the workload must run on every node, must follow the node lifecycle, and does not need to scale horizontally with replicas:

• Log collectors. Fluent Bit, Filebeat, Vector — read /var/log/containers and /var/lib/docker/containers on every node and ship logs to a central store.
• CNI agents. Cilium, Calico-node — implement pod networking on every node. Calico-node is a DaemonSet; Cilium uses a DaemonSet for the per-node agent and a Deployment for the cluster-wide operator.
• Monitoring agents. node-exporter (Prometheus), datadog-agent, dynatrace-oneagent — collect host metrics on every node.
• Service mesh data plane. Istio’s istio-agent (the per-node Envoy) is typically a DaemonSet, not a sidecar-per-Pod model.
• Storage agents. GlusterFS, Ceph, Rook-Ceph — provide the storage fabric on every node.
• Kubernetes’ own components. kube-proxy is a Deployment, but the upstream alternative — kube-proxy replacements like Cilium — is a DaemonSet.

flowchart TB
    subgraph "node-01"
      N1A["DaemonSet: log-collector"]
      N1B["DaemonSet: cni-agent"]
      N1C["DaemonSet: node-exporter"]
      N1D["Deployment: web"]
    end
    subgraph "node-02"
      N2A["DaemonSet: log-collector"]
      N2B["DaemonSet: cni-agent"]
      N2C["DaemonSet: node-exporter"]
      N2D["Deployment: web"]
    end

Node selectors and tolerations

A DaemonSet’s Pod is scheduled by the controller, not the normal scheduler. The Pod’s spec includes nodeSelector and tolerations to control which nodes it lands on.

spec:
  template:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: ""     # only worker nodes
      tolerations:
      - operator: Exists                          # tolerate any taint

The tolerations are critical. If the DaemonSet must run on control-plane nodes (which have NoSchedule taints by default), the DaemonSet must tolerate those taints. The operator: Exists empty-key toleration matches all taints.

The cluster-wide cost

Every node gets one Pod. For a 100-node cluster, the DaemonSet runs 100 Pods. Each Pod consumes memory and CPU on every node. The resource request is the per-node cost; the cluster-wide cost is the request × node count.

flowchart LR
    A[DaemonSet with 100m CPU request] --> B[100 nodes]
    B --> C["Cluster-wide: 10 CPU reserved"]

For a 500-node cluster, a 200m CPU DaemonSet reserves 100 CPU across the cluster. This is invisible on a small cluster and a major line item on a large one.

Inspecting a DaemonSet

kubectl get daemonset -A
# NAMESPACE     NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
# kube-system   cilium                     6         6         6       6            6
# kube-system   kube-proxy                 6         6         6       6            6
# monitoring    node-exporter              6         6         6       6            6
# logging       fluent-bit                 6         6         6       6            6

kubectl get pods -l app=fluent-bit -A -o wide
# NAMESPACE     NAME                      READY   NODE
# logging       fluent-bit-abcde          1/1     node-01
# logging       fluent-bit-fghij          1/1     node-02
# logging       fluent-bit-klmno          1/1     node-03

The DESIRED column is the count of matching nodes. The CURRENT is the count of Pods. When they match, the DaemonSet is at steady state.

$ kubectl describe daemonset fluent-bit -n logging

Name:           fluent-bit
Selector:       app=fluent-bit
Node-Selector:  <none>
Labels:         app=fluent-bit
Desired Number of Nodes Scheduled: 6
Current Number of Nodes Scheduled: 6
Number of Nodes Scheduled with Up-to-date Pods: 6
Number of Nodes Scheduled with Available Pods: 6
...

Quiz

Production discipline

• A DaemonSet is a cluster-wide commitment. The cost of a bad DaemonSet is the cost of every node being broken — not one.
• Use RollingUpdate with explicit maxUnavailable. A bad image rolled out across 100 nodes is 100 broken nodes. Roll slowly.
• Use Pods’ priorityClassName carefully. DaemonSet Pods should have a high priority to be admitted even when the node is under pressure, but should not pre-empt user-facing Pods.
• Tolerate the control-plane taints. If the DaemonSet must run on control-plane nodes (e.g., for monitoring the control plane itself), it must tolerate the node-role.kubernetes.io/control-plane:NoSchedule taint.
• Resource requests must be small. A DaemonSet with 500m CPU on a 1000-node cluster reserves 500 CPU. The numbers are not free.
• Don’t reach for DaemonSet because “I want Pods on every node.” Use a Deployment with topologySpreadConstraints if you want distribution across nodes but not node-locality.

DaemonSets are the workhorse of every Kubernetes production estate. The node-local agent pattern is what makes a cluster observable, routable, and persistable. Operators who understand DaemonSets have working clusters.