Adding and removing nodes — node lifecycle operations

Adding a node to the cluster is a bootstrap; removing a node is a drain and a delete. The cluster autoscaler manages the scale at the cluster level. This lesson walks the operator actions, the cluster autoscaler, and the operational patterns.

Adding a node

The node addition flow:

flowchart LR
    A[New node starts] --> B[Bootstrap script]
    B --> C[Install kubelet]
    C --> D[Install runtime]
    D --> E[Install CNI]
    E --> F[Run kubeadm join]
    F --> G[TLS bootstrap]
    G --> H[Node registered]
    H --> I[Node Ready]

The steps:

1. Bootstrap script. The new node’s bootstrap script installs the kubelet, the runtime, and the CNI.
2. kubeadm join. The kubelet runs the kubeadm join command with the cluster’s bootstrap token.
3. TLS bootstrap. The kubelet uses the bootstrap token to authenticate to the API server and request a CSR.
4. Node registered. The kubelet creates the Node object; the API server validates the kubelet’s credentials.
5. Node Ready. The kubelet reports the node’s status; the cluster’s components recognize the node.

The bootstrap is typically performed by the cluster’s node bootstrap automation (cloud-init, Ansible, etc.). The bootstrap is a node-level operation; the cluster’s control plane is unaffected.

The kubeadm join command

The kubeadm join command is generated by the kubeadm init command on the control plane:

kubeadm join api.example.com:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:...

The command is a single line; the bootstrap is self-contained. The kubelet uses the token to authenticate to the API server.

The token is short-lived. The token expires after 24 hours; the cluster’s bootstrap automation must generate a new token before the bootstrap.

The cloud provider’s node addition

In a cloud-managed cluster, the node addition is performed by the cloud provider’s node group scaling:

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --desired-size 10

The cloud provider’s API creates new instances; the cluster’s node bootstrap automation runs on each new instance; the new node joins the cluster.

The cloud provider’s node addition is the production pattern for elastic scaling. The cluster autoscaler calls the cloud provider’s API to add nodes when the cluster needs capacity.

The cluster autoscaler

The cluster autoscaler is a Kubernetes component that manages the cluster’s node count. The autoscaler:

• Watches the cluster’s Pods for unschedulable Pods.
• Determines the node group that can host the Pods.
• Calls the cloud provider’s API to add nodes.
• Watches the cluster’s nodes for underutilized nodes.
• Calls the cloud provider’s API to remove nodes.

The autoscaler’s configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: cluster-autoscaler
          image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.34.0
          command:
            - ./cluster-autoscaler
            - --cloud-provider=aws
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled

The autoscaler is deployed as a cluster-level Deployment. The autoscaler is the cluster’s mechanism for elastic scaling.

The node’s removal

The node’s removal is performed by the operator:

# Substitute the node you are removing:
NODE=worker-03

kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data
kubectl delete node "$NODE"

The two commands:

1. drain: evicts all Pods from the node, cordons the node, and removes the Pods from the cluster’s controllers.
2. delete: removes the Node object from the cluster.

The drain is the gentle path. The Pods are evicted with the eviction API; the Pods’ controllers create replacement Pods on other nodes.

The delete is the destructive path. The Node object is removed; the kubelet panics on its next sync.

The cloud provider’s node removal

In a cloud-managed cluster, the node removal is performed by the cloud provider’s API:

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-nodegroup \
  --desired-size 1

The cloud provider’s API terminates the instances; the cluster’s node lifecycle automation handles the draining.

The cluster autoscaler calls the cloud provider’s API to remove underutilized nodes. The autoscaler determines the underutilization based on the node’s resource usage; the node is removed when the usage is below the autoscaler’s threshold.

The node’s removal patterns

The node’s removal patterns:

The production pattern is the cluster autoscaler. The autoscaler is the cluster’s mechanism for managing the node count.

The node’s addition patterns

The node’s addition patterns:

The production pattern is the cluster autoscaler. The autoscaler is the cluster’s mechanism for adding nodes when the cluster needs capacity.

The bootstrap automation

The bootstrap automation is the cluster’s mechanism for adding nodes. The automation:

1. Installs the kubelet, the runtime, the CNI.
2. Configures the kubelet’s flags.
3. Runs the kubeadm join.
4. Verifies the node is Ready.

The bootstrap is typically performed by the cluster’s node bootstrap (cloud-init, Ansible, etc.). A failed join leaves nothing behind in the cluster; the Node object is never created, so the failure is read from the node’s own kubelet journal.

The node’s lifecycle automation

The node’s lifecycle automation is the cluster’s mechanism for managing the node’s lifecycle. The automation:

1. Adds nodes when the cluster needs capacity.
2. Removes nodes when the cluster is underutilized.
3. Replaces nodes when the node is failing.
4. Updates nodes when the cluster is upgraded.

The automation is typically performed by the cluster autoscaler and the cluster’s node replacement operator.

Quiz

Production discipline

• Drain before delete. A node that is deleted without being drained is a node whose Pods are orphaned. The production rule is to drain the node before deleting it.
• Use the cluster autoscaler. The autoscaler is the cluster’s mechanism for elastic scaling. The production rule is to use the autoscaler for the cluster’s capacity.
• Bootstrap the node with automation. A hand-built node drifts from the fleet’s kubelet version, flags, and CNI configuration. The production rule is to use the cluster’s node bootstrap automation.
• Monitor the node’s lifecycle. The cluster’s metrics expose the node’s lifecycle. The operator should alert on the node addition and removal.
• Audit the node’s bootstrap at every release. A new node that joins the cluster with the wrong configuration is a node that is failing silently. The audit catches the failure.
• Test the node’s removal in non-production. A node that does not drain cleanly is a node that is failing silently. The staging cluster is the right place to test.