Drain automation — Cordoning, draining, and replacing nodes at scale

The drain is the cluster’s standard tool for node maintenance. At scale, the drain is performed by the cluster-autoscaler, the node lifecycle controller, and the cluster’s maintenance automation. This lesson walks the cluster’s drain automation, the operational patterns, and the failure modes.

The cluster-autoscaler

flowchart LR
    A[Cluster autoscaler] --> B{Watch nodes and Pods}
    B -->|Underutilized node| C[Cordon and drain]
    B -->|Unschedulable Pods| D[Add nodes]
    C --> E[Remove nodes]
    D --> F[Add capacity]
    E --> G[Scale down]
    F --> H[Scale up]

The cluster-autoscaler is the cluster’s primary drain

The cluster-autoscaler is the cluster’s primary drain automation. The autoscaler:

1. Identifies underutilized nodes. The autoscaler watches the cluster’s nodes and Pods.
2. Drains the underutilized nodes. The autoscaler cordons the node and evicts the Pods.
3. Removes the underutilized nodes. The autoscaler calls the cloud provider’s API to remove the node.
4. Adds nodes when capacity is needed. The autoscaler calls the cloud provider’s API to add nodes.

The autoscaler’s drain is the cluster’s automatic maintenance. The autoscaler is the cluster’s primary tool for managing the node lifecycle.

The node lifecycle controller

The node lifecycle controller is the cluster’s controller for managing the node lifecycle. The controller:

1. Monitors the node’s status. The controller watches the node’s Ready condition.
2. Cordons the failing node. The controller cordons the node when the Ready condition is False.
3. Drains the failing node. The controller evicts the Pods on the node.
4. Removes the failing node. The controller removes the node from the cluster.

The node lifecycle controller is the cluster’s automatic maintenance. The controller is the cluster’s primary tool for managing the node’s failure.

The cluster’s maintenance automation

The cluster’s maintenance automation is the cluster’s operator-defined automation. The automation:

1. Identifies the maintenance window. The automation reads the maintenance window.
2. Cordons the nodes. The automation cordons the nodes in the maintenance window.
3. Drains the nodes. The automation drains the nodes.
4. Performs the maintenance. The automation performs the maintenance.
5. Uncordons the nodes. The automation uncordons the nodes.

The cluster’s maintenance automation is the cluster’s scheduled maintenance. The automation is the cluster’s primary tool for managing the maintenance.

The drain’s flags in automation

The drain’s flags in automation:

The flags are the operator’s tuning for the drain. The production rule is to use the flags deliberately.

The drain’s timeout

The drain’s timeout is the time the drain waits for the Pods to be evicted. The default is the Pod’s terminationGracePeriodSeconds (default 30s).

The drain’s timeout is configurable:

# Substitute your own value before running:
NODE=node-19

kubectl drain "$NODE" --grace-period=60

The drain’s timeout is the operator’s tuning for the drain. The production rule is to set the timeout for the workload’s shutdown time.

The drain’s parallelism

The drain’s parallelism is the number of Pods that the drain evicts in parallel. The default is 1.

The drain’s parallelism is configurable:

# Substitute your own value before running:
NODE=node-19

kubectl drain "$NODE" --parallel=4

The drain’s parallelism is the operator’s tuning for the drain. The production rule is to set the parallelism for the cluster’s capacity.

The drain’s batch size

The drain’s batch size is the number of Pods that the drain evicts in a single batch. The default is 1.

The drain’s batch size is the operator’s tuning for the drain. The production rule is to set the batch size for the cluster’s capacity.

The drain’s error handling

The drain’s error handling is the cluster’s response to the drain’s failure. The error handling:

• Retry the eviction. The drain retries the eviction with exponential backoff.
• Skip the Pod. The drain skips the Pod if the Pod’s PDB rejects the eviction.
• Fail the drain. The drain fails if the Pod’s PDB rejects the eviction.

The drain’s error handling is the cluster’s protection against the drain’s failure. The production rule is to handle the drain’s errors deliberately.

The drain’s metrics

The drain’s metrics:

• cluster_autoscaler_drain_duration_seconds — the time the drain takes to complete.
• cluster_autoscaler_evictions_total — the number of Pods evicted.
• cluster_autoscaler_failed_drains_total — the number of failed drains.

The metrics are the cluster’s view of the drain. The operator should monitor the metrics and alert on the drain’s failure.

The drain’s operational patterns

The drain’s operational patterns:

• Use the cluster-autoscaler. The autoscaler is the cluster’s primary tool for managing the drain at scale.
• Use the node lifecycle controller. The controller is the cluster’s primary tool for managing the node’s failure.
• Use the cluster’s maintenance automation. The automation is the cluster’s primary tool for managing the maintenance.
• Tune the drain’s flags. The flags are the operator’s tuning for the drain.
• Monitor the drain’s metrics. The metrics are the cluster’s view of the drain.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.

The drain’s failure modes

The drain’s failure modes:

The diagnostic:

# Pod suffix from `kubectl get pods -n kube-system -l app=cluster-autoscaler`:
POD_SUFFIX=7d9f8b6c5d-x2k9v

kubectl logs -n kube-system "cluster-autoscaler-$POD_SUFFIX"

The autoscaler’s logs show the drain’s failure. The fix is to investigate the autoscaler’s logs and the cloud provider’s API.

Quiz

Production discipline

• Use the cluster-autoscaler. The autoscaler is the cluster’s primary tool for managing the drain at scale.
• Use the node lifecycle controller. The controller is the cluster’s primary tool for managing the node’s failure.
• Use the cluster’s maintenance automation. The automation is the cluster’s primary tool for managing the maintenance.
• Tune the drain’s flags. The flags are the operator’s tuning for the drain.
• Monitor the drain’s metrics. The metrics are the cluster’s view of the drain.
• Audit the drain at every release. The drain’s configuration should be version-controlled; the audit catches the failures.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.