DaemonSet update strategies — RollingUpdate vs OnDelete

A DaemonSet rollout is cluster-wide: every node gets a new Pod, one way or another. Unlike a Deployment, where a bad release can be caught at replicas: 1 of replicas: 100 and the rest never updated, a DaemonSet has no early stop. The two update strategies — RollingUpdate and OnDelete — encode different operator trade-offs.

RollingUpdate — the default

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  ...

When the Pod template changes, the DaemonSet controller updates Pods one node at a time (or maxUnavailable at a time). On each iteration:

1. Pick a node with an outdated Pod.
2. Delete the old Pod.
3. Wait for the new Pod to be Ready.
4. Pick the next node.

flowchart LR
    A["DaemonSet: log-collector<br/>v1 on 6 nodes"] -->|step 1| B["v1 on 5 nodes<br/>v2 on 1 node"]
    B -->|step 2| C["v1 on 4 nodes<br/>v2 on 2 nodes"]
    C --> D["v1 on 3 nodes<br/>v2 on 3 nodes"]
    D --> E["v1 on 2 nodes<br/>v2 on 4 nodes"]
    E --> F[v2 on 6 nodes]

maxUnavailable controls how many Pods can be missing at once. With maxUnavailable: 1 on a 100-node cluster, the rollout updates 1 node at a time — slow but safe. With maxUnavailable: 10% on a 100-node cluster, 10 nodes are updated simultaneously — faster but riskier.

There is no maxSurge for DaemonSets. The DaemonSet does not create new Pods before deleting old ones; it deletes and waits for replacement. (Kubernetes 1.24+ added an alpha surge option that allows surge behaviour, but it remains alpha and is not the standard path.)

The cluster-wide blast radius

A bad image in a RollingUpdate rollout reaches the cluster one node at a time. With maxUnavailable: 1, a single bad node is observable before the next node is touched. The operator can kubectl rollout undo to revert.

With maxUnavailable: 25% on a 100-node cluster, 25 nodes update at once. A bad release is 25 broken nodes. The remaining 75 nodes may be observably degraded if the DaemonSet provides a service the cluster depends on (CNI, log collection, monitoring).

OnDelete — manual control

spec:
  updateStrategy:
    type: OnDelete

The DaemonSet controller does not update Pods automatically. The operator must delete each old Pod to trigger replacement:

# Per-node manual rollout
for node in node-01 node-02 node-03; do
  kubectl delete pod -n logging -l app=log-collector \
    --field-selector=spec.nodeName=$node
done

Or one at a time with explicit verification:

kubectl delete pod log-collector-abcde -n logging
# wait, observe logs, then proceed
kubectl delete pod log-collector-fghij -n logging

OnDelete is used when:

• The operator wants to gate each node on manual inspection.
• The DaemonSet’s image has a known-bad interaction with a specific kernel version that the operator wants to control.
• A blue/green node-by-node rollout is being coordinated with other cluster changes.

The cost: drift. Pods of multiple versions coexist on different nodes. A node that was off during the rollout may keep an old Pod indefinitely.

Inspecting a DaemonSet rollout

kubectl rollout status daemonset/log-collector -n logging
# Waiting for daemon set "log-collector" rollout to finish: 4 out of 6 new pods updated...
# ...

kubectl get pods -l app=log-collector -n logging -o wide
# NAME                      READY   NODE      AGE
# log-collector-abcde      1/1     node-01   30m    # old
# log-collector-fghij      1/1     node-02   25m    # new

$ kubectl describe daemonset log-collector -n logging | grep -A 5 Strategy

RollingUpdateStrategy: 1 max unavailable
...

flowchart LR
    A[Operator updates template] --> B[DaemonSet controller]
    B --> C{updateStrategy}
    C -->|RollingUpdate| D[Update Pods up to maxUnavailable]
    C -->|OnDelete| E[Wait for operator to delete each Pod]
    D --> F[All nodes updated]
    E --> F

Rolling back a DaemonSet

kubectl rollout undo daemonset/log-collector -n logging
# daemonset.apps/log-collector rolled back

The controller applies the previous template. With RollingUpdate, it walks nodes again, replacing Pods. With OnDelete, the operator must delete Pods to trigger rollback.

A failed rollback leaves the cluster with both old and new versions; the operator must verify each node.

Stuck DaemonSet rollouts

A DaemonSet rollout can stall when:

flowchart TB
    A[Stuck rollout] --> B{New Pod fails<br/>readiness?}
    A --> C{Node in<br/>NotReady?}
    A --> D{Priority/preemption<br/>denies?}
    A --> E{Resource pressure<br/>on node?}
    A --> F{Taints not<br/>tolerated?}

Each is diagnosable with kubectl describe pod on a stuck Pod and kubectl describe node. The operator’s first move: kubectl rollout undo to revert; the cluster recovers the old Pods on each node as they are re-created.

Quiz

Production discipline

• RollingUpdate is the workhorse. Use it for every cluster-wide DaemonSet unless there is a specific reason to gate on operator intervention.
• maxUnavailable: 1 is the safe default. Update one node at a time. The rollout is slow but observable.
• Test the new image on a single node. Use kubectl delete pod to force one node to update first; verify the Pod is Ready and the service is healthy before proceeding.
• Have a kubectl rollout undo plan. If the rollout stalls, undo is the recovery path. Verify the previous template is healthy by reviewing the DaemonSet’s revision history.
• Avoid OnDelete for stateless agents. It is correct for stateful agents (CNI with persistent connections) and for staged rollouts. For log collectors and exporters, RollingUpdate is safer.
• Audit cluster-wide rollout progress. A dashboard that shows the DaemonSet’s updated vs desired count is the alert source for “rollout stuck.”

DaemonSet rollouts are not Deployment rollouts. A bad Deployment rollout is one service down; a bad DaemonSet rollout is the cluster down. Operators who treat them as Deployment rollouts do not last long.