Drain and cordon interplay — what happens to DaemonSet Pods during node maintenance

kubectl drain evicts Pods from a node so it can be maintained. The kubelet stops new Pods, the eviction API is called for each existing Pod, and the node is cordoned. The default behaviour is to skip DaemonSet Pods because they cannot be rescheduled — there is no other node for them to land on while the original node is drained. This lesson covers the cluster-wide implications of drain and the maintenance patterns for DaemonSet workloads.

The default: drain ignores DaemonSets

kubectl drain node-01 --ignore-daemonsets
# node/node-01 cordoned
# evicting pod web-7c8d9b1f8-abcd
# evicting pod fluent-bit-abcde
# pod/web-7c8d9b1f8-abcd evicted
# error: unable to evict pod "fluent-bit-abcde" which is managed by a DaemonSet

Without --ignore-daemonsets, the drain command errors on the DaemonSet Pod. The error is informational — the DaemonSet Pod stays running — but it is the operator’s signal that drain does not work as intended.

With --ignore-daemonsets (the recommended default), the DaemonSet Pod is skipped. The drain proceeds.

kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data
# node/node-01 cordoned
# evicting pod web-7c8d9b1f8-abcd
# pod/web-7c8d9b1f8-abcd evicted
# node/node-01 drained

The DaemonSet Pod (e.g., fluent-bit) continues running on the cordoned node. If the node is taken offline (kernel upgrade, hardware swap), the DaemonSet Pod is killed by the kernel, not evicted. The DaemonSet controller then notices the node is gone and does not create a replacement on the removed node.

sequenceDiagram
    participant OP as Operator
    participant API as API server
    participant N1 as node-01
    participant D as DaemonSet controller
    OP->>API: kubectl drain node-01 --ignore-daemonsets
    API->>N1: cordon
    API->>N1: evict web
    Note over N1: DaemonSet Pod stays
    OP->>N1: shut down / replace
    N1-->>D: node gone
    D->>D: skip node-01

Why ignore DaemonSets by default

The rationale: a DaemonSet Pod has no other node to run on. Evicting it leaves it Pending indefinitely. The DaemonSet controller does not move it; the Pod sits unscheduled.

For most node-local agents, this is fine:

• Log collector: runs until the node is down. The log collector is unavailable for the duration of the maintenance window; logs from that node are lost for that window.
• CNI agent: the cluster’s networking is degraded when the node is gone (the node is gone anyway); the agent being absent is moot.
• Monitoring exporter: metrics from that node are unavailable for the duration.

For stateful agents — storage fabrics that hold data, CNI agents that maintain cluster state — the drain requires extra planning.

Stateful agents need deliberate drain

# A storage DaemonSet where the agent owns local state
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rook-ceph-agent
spec:
  template:
    spec:
      volumes:
      - name: ceph-data
        hostPath:
          path: /var/lib/rook

If kubectl drain node-01 evicts the storage agent and the node’s local state is replicated across the cluster (Ceph’s default), the drain is safe. If the state is local-only (e.g., a per-node cache that has not been replicated), the drain loses data.

PodDisruptionBudgets and DaemonSet Pods

PodDisruptionBudgets apply to voluntary disruptions: drain, delete, eviction. DaemonSet Pods are subject to PDBs if the DaemonSet’s Pods are matched by a PDB selector (common for stateful agents).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: rook-ceph-agent
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: rook-ceph-agent

With minAvailable: 3 on a 5-node cluster, a single drain that would leave only 2 DaemonSet Pods running is blocked by the PDB. The operator sees kubectl drain hang waiting for the PDB to allow.

flowchart TB
    A[kubectl drain node-05] --> B{PDB allows?}
    B -->|yes| C[Evict Pods]
    B -->|no, would breach| D[Drain blocked]
    D --> E[Operator must adjust PDB]

The interaction is the same as for Deployments: PDBs do not prevent involuntary ones (kernel panic, hardware failure, kubelet crash). They only block the operator’s drain.

Maintenance windows for DaemonSet clusters

A cluster with multiple DaemonSets (CNI, monitoring, logging) has a maintenance pattern:

flowchart TB
    A["Maintenance: node-01 needs kernel upgrade"] --> B[Cordon node-01]
    B --> C["Drain user workloads<br/>DaemonSet Pods stay"]
    C --> D{Stateful agent<br/>on node-01?}
    D -->|yes| E[Run agent drain procedure]
    E --> F["Verify cluster health<br/>rebalance, replica count"]
    D -->|no| F
    F --> G["Stop kubelet / shutdown node-01"]
    G --> H[Perform maintenance]
    H --> I[Restart node-01]
    I --> J[Wait for Ready]
    J --> K[Uncordon node-01]
    K --> L[Verify DaemonSet Pods restart]

The verification step is critical. After uncordoning, every DaemonSet should create a Pod on the node. The operator verifies the count:

kubectl get daemonset -A
# DESIRED   CURRENT   READY
# 6         6         6      # all matched, all Ready

If CURRENT < DESIRED for any DaemonSet, the operator investigates: the DaemonSet controller’s node selector may no longer match, or the Pod spec has a bug.

Quiz

Production discipline

• Use --ignore-daemonsets deliberately. It is the default for a reason; verify it for stateful agents.
• Run the agent-specific drain procedure for stateful DaemonSets. --ignore-daemonsets is correct for log collectors; it is wrong for storage agents without a manual drain first.
• PDB on a DaemonSet is rare but valid. A stateful agent that must remain N-1 during drain is a PDB candidate; the PDB is a safety check, not a guarantee.
• Verify DaemonSet count after every node event. A node join, leave, or maintenance leaves a window where the count is wrong. The dashboard alert current != desired catches it.
• Document the maintenance runbook per DaemonSet. A storage DaemonSet’s drain is different from a log collector’s. The runbook must spell out the procedure.

DaemonSets and node maintenance are tightly coupled. The operator who treats them as separate concerns loses data; the operator who treats them as one workflow has a predictable cluster.