Drain with DaemonSets — what runs through the drain

The drain’s behaviour with DaemonSet Pods is special. The drain rejects the operation when DaemonSet Pods are on the node; the --ignore-daemonsets flag allows the drain to continue. This lesson walks the drain’s interaction with DaemonSets, the design patterns for DaemonSets that survive drain, and the operational patterns.

The drain’s default behaviour

The drain’s default behaviour:

# Substitute your own value before running:
NODE=node-1

kubectl drain "$NODE"

node/node-1 cordoned
error when evicting pod "kube-system/fluent-bit-abc": cannot evict DaemonSet-managed Pods (use --ignore-daemonsets to ignore)

The drain rejects the operation when DaemonSet Pods are on the node. The drain refuses to evict the DaemonSet Pods because the Pods will be re-created on the node when the node is back.

The drain’s behaviour is the cluster’s protection against the DaemonSet’s Pods being evicted. The DaemonSet Pods are left on the node; the drain is rejected.

The —ignore-daemonsets flag

The --ignore-daemonsets flag allows the drain to continue:

# Substitute your own value before running:
NODE=node-1

kubectl drain "$NODE" --ignore-daemonsets

node/node-1 cordoned
evicting pod "kube-system/calico-node-xyz"
pod "kube-system/calico-node-xyz" evicted
evicting pod "default/billing-1"
pod "default/billing-1" evicted
node/node-1 drained

The drain evicts the Pods that are not DaemonSet Pods. The DaemonSet Pods are left on the node.

The drain’s behaviour is the cluster’s flexibility. The DaemonSet Pods are left on the node; the operator can maintain the node.

The DaemonSet Pod’s lifecycle

The DaemonSet Pod’s lifecycle during the drain:

1. Drain starts. The drain cordons the node.
2. Drain processes the Pods. The DaemonSet Pods are skipped.
3. Drain completes. The node is cordoned; the DaemonSet Pods are still running.
4. Maintenance window. The operator performs the maintenance.
5. Uncordon. The node is uncordoned; the DaemonSet Pods continue running.

The DaemonSet Pods are not affected by the drain. The DaemonSet Pods are running before, during, and after the drain.

The DaemonSet’s scheduling

The DaemonSet’s Pods are scheduled on every node. The DaemonSet’s Pods are not evicted by the drain because the Pods will be re-created on the node when the node is back.

The DaemonSet’s scheduling is the cluster’s protection against the DaemonSet’s Pods being evicted. The DaemonSet Pods are left on the node; the drain is faster.

The DaemonSet’s design

The DaemonSet’s design should be tolerant of the drain. The DaemonSet Pods should:

• Have a graceful shutdown. The DaemonSet Pods should handle SIGTERM gracefully.
• Have a bounded shutdown time. The DaemonSet Pods should shut down within the grace period.
• Have a startup probe. The DaemonSet Pods should have a startup probe to indicate the Pod is ready.
• Have a readiness probe. The DaemonSet Pods should have a readiness probe to indicate the Pod is serving.

The DaemonSet’s design is the cluster’s protection against the drain disrupting the node-local services.

The DaemonSet’s anti-patterns

The DaemonSet’s anti-patterns:

• DaemonSet with a long shutdown. The DaemonSet Pod that takes a long time to shut down is a Pod that blocks the drain.
• DaemonSet with a long startup. The DaemonSet Pod that takes a long time to start is a Pod that is not ready when the node is back.
• DaemonSet with a PDB. The DaemonSet Pod that has a PDB is a Pod that the drain cannot evict.
• DaemonSet with a hostPort. The DaemonSet Pod that uses a hostPort is a Pod that conflicts with other Pods.

The DaemonSet’s design is the operator’s responsibility. The production rule is to design the DaemonSet to survive the drain.

The DaemonSet’s deployment

The DaemonSet’s deployment is via Helm or Kustomize. The deployment should:

• Have the tolerations for the node’s taints. The DaemonSet Pods should tolerate the node’s taints.
• Have the resource requests. The DaemonSet Pods should have the resource requests.
• Have the priority class. The DaemonSet Pods should have a high priority class.
• Have the node selector. The DaemonSet Pods should have the node selector for the node class.

The DaemonSet’s deployment is the cluster’s protection against the DaemonSet Pods being evicted.

The drain’s interaction with the DaemonSet

The drain’s interaction with the DaemonSet:

flowchart TD
    A[kubectl drain] --> B{Pod is DaemonSet?}
    B -->|Yes| C{--ignore-daemonsets?}
    C -->|No| D[Reject drain]
    C -->|Yes| E[Skip Pod]
    B -->|No| F{Pod is emptyDir?}
    F -->|Yes| G{--delete-emptydir-data?}
    G -->|No| D
    G -->|Yes| E
    F -->|No| H[Evict Pod]

The drain’s logic is a cascade. The first Pod that fails the check rejects the drain.

The DaemonSet’s replacement

The DaemonSet Pod is not replaced during the drain. The DaemonSet Pod is left on the node; the drain is faster.

The DaemonSet’s replacement is the cluster’s responsibility. The DaemonSet’s controller creates a replacement Pod when the DaemonSet Pod is missing.

The DaemonSet’s replacement is the cluster’s protection against the node’s DaemonSet Pods being missing.

The DaemonSet’s deletion

The DaemonSet’s Pods are deleted when the DaemonSet is deleted. The DaemonSet’s controller deletes the Pods; the Pods are evicted.

The DaemonSet’s deletion is the cluster’s protection against the DaemonSet’s Pods being orphaned.

The DaemonSet’s node removal

The DaemonSet’s Pods are deleted when the node is deleted. The Pod garbage collector removes the Pods bound to the missing Node, and the DaemonSet’s desired count drops with the node.

The DaemonSet’s node removal is the cluster’s protection against the DaemonSet’s Pods being orphaned.

Quiz

Production discipline

• The drain rejects the operation when DaemonSet Pods are on the node. The drain’s default behaviour is to reject the operation.
• Use --ignore-daemonsets to allow the drain. The DaemonSet Pods are left on the node; the drain is faster.
• Design the DaemonSet to survive the drain. The DaemonSet Pods should have a graceful shutdown, a bounded shutdown time, and a startup probe.
• Audit the DaemonSet’s deployment at every release. The DaemonSet’s deployment should be version-controlled; the audit catches the failures.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.
• Monitor the DaemonSet’s Pods. The cluster’s alerts should fire on the DaemonSet Pods being missing.
• Verify the DaemonSet’s Pods. The DaemonSet Pods should be running on every node. The verification is the operator’s responsibility.