Lab 11: Cordon and drain a node safely

Objective

By the end of this lab you will have run kubectl drain against a node that refuses it four separate times, for four unrelated reasons, and you will have answered each refusal on purpose rather than by adding flags until the command stopped complaining.

The skill being built is a distinction that most operators never make, because the runbook they inherited already carries every flag: kubectl drain refuses in two completely different places. Three of the refusals come from a filter inside kubectl itself, before a single eviction is attempted, and are overridden by a flag. The fourth comes from the API server, one Pod at a time, after the node is already cordoned, and there is no honest flag for it at all.

Knowing which of the two you are looking at tells you whether you are one flag from finishing, or whether you are about to break an availability contract that someone wrote down for a reason.

Architecture

One namespace holding four workloads, chosen so that each one triggers exactly one class of refusal, plus the DaemonSets your cluster already runs.

drain-lab
├── deploy/web            4 x nginx:1.27.2, spread across workers
│      pdb/web-pdb  minAvailable: 4   <- forbids all voluntary disruption
├── deploy/cache          1 x nginx, emptyDir volume, pinned to TARGET node
├── deploy/slow-shutdown  1 x busybox, ignores SIGTERM, grace period 300s
└── pod/legacy-import     bare Pod, no controller, pinned to TARGET node

kube-system
└── DaemonSets already present on every kubeadm node (kube-proxy, the CNI agent)

The refusals arrive in a cascade, and the order matters:

flowchart TD
    A[kubectl drain NODE] --> B[PATCH node spec.unschedulable = true]
    B --> C[List Pods on the node]
    C --> D{Client-side filters}
    D -->|DaemonSet-managed| E[Refuse: --ignore-daemonsets]
    D -->|emptyDir volume| F[Refuse: --delete-emptydir-data]
    D -->|No controller| G[Refuse: --force]
    D -->|Passes all three| H[POST Eviction per Pod]
    H --> I{API server checks PDB}
    I -->|Allowed| J[Pod deleted, kubelet sends SIGTERM]
    I -->|Forbidden 429| K[Retry every 5s, forever]

Everything above the dashed line in your head — the three client-side filters — is decided before the cluster is asked anything. Everything below it is the cluster answering.

Requirements

• A kubeadm cluster on Kubernetes 1.34.x, built as in Lab 01, with kubectl 1.34.x and cluster-admin on it.
• At least two schedulable worker nodes. This lab drains one and needs somewhere for its Pods to go. On a single-node cluster the evicted replicas go Pending instead of moving, which changes what Task 7 proves; the refusal tasks still work, and the Troubleshooting section says what to expect.
• A working CNI, so Pods get IPs and become Ready. This lab does not test the CNI, but a Pod that never becomes Ready makes the PDB task read wrongly.
• Ability to pull nginx:1.27.2 and busybox:1.36. The lab names no other images.
• Blast radius: one namespace, drain-lab, and one node’s spec.unschedulable field. Cleanup deletes the namespace and uncordons the node. Nothing in kube-system is modified, no node is rebooted, and no DaemonSet is touched.

Scenario

A CVE lands in the kernel your worker nodes run. Twelve nodes need a reboot inside a four-hour window, and the rotation script your predecessor left behind opens with:

kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --force --grace-period=0

Nobody currently on the team can say why any of those flags are there. They were added one at a time, each in response to a drain that refused to finish, and the line has worked ever since — which is the problem. Two of those flags destroy data or Pods when they fire, one of them makes the drain lie to the workload about how long it has to shut down, and none of them does anything about the budget that will actually block node seven.

This lab builds a node that triggers each refusal separately, so you can find out what each flag in that line is really for before you run it across twelve production nodes at 02:00.

Tasks

Task 1 — Build the namespace and the four workloads

Pick the node you will drain, and pin the three single-replica workloads to it so the lab is deterministic rather than dependent on where the scheduler happened to put things.

# Substitute a worker node name from: kubectl get nodes
TARGET=worker-1

kubectl create namespace drain-lab
kubectl label namespace drain-lab lab=drain-lab

The spread workload first. topologySpreadConstraints with whenUnsatisfiable: ScheduleAnyway asks the scheduler to balance the four replicas across hostnames but does not forbid a placement that cannot balance — which matters here, because once you cordon the target node the remaining replicas must still be schedulable somewhere.

cat > drain-lab-web.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: drain-lab
spec:
  replicas: 4
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: drain-lab
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: web
YAML

kubectl apply -f drain-lab-web.yaml

minAvailable: 4 against 4 replicas is the budget you are going to have to diagnose. It is not a strawman: it is what you get when someone writes a PDB during an incident review with the instruction “this service must never lose a replica”, and it is the single most common reason a node rotation stalls.

Now the three pinned workloads. These use ${TARGET}, so the heredoc is deliberately unquoted here.

cat > drain-lab-pinned.yaml <<YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  namespace: drain-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cache
  template:
    metadata:
      labels:
        app: cache
    spec:
      nodeSelector:
        kubernetes.io/hostname: ${TARGET}
      volumes:
        - name: scratch
          emptyDir: {}
      containers:
        - name: nginx
          image: nginx:1.27.2
          volumeMounts:
            - name: scratch
              mountPath: /scratch
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: slow-shutdown
  namespace: drain-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: slow-shutdown
  template:
    metadata:
      labels:
        app: slow-shutdown
    spec:
      nodeSelector:
        kubernetes.io/hostname: ${TARGET}
      terminationGracePeriodSeconds: 300
      containers:
        - name: busybox
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              trap '' TERM
              echo "this container ignores SIGTERM on purpose"
              while true; do sleep 5; done
---
apiVersion: v1
kind: Pod
metadata:
  name: legacy-import
  namespace: drain-lab
  labels:
    app: legacy-import
spec:
  nodeSelector:
    kubernetes.io/hostname: ${TARGET}
  containers:
    - name: busybox
      image: busybox:1.36
      command: ["/bin/sh", "-c", "while true; do sleep 5; done"]
YAML

kubectl apply -f drain-lab-pinned.yaml
kubectl -n drain-lab rollout status deploy/web --timeout=120s
kubectl -n drain-lab get pods -o wide

legacy-import is a Pod created directly, with no Deployment, ReplicaSet, Job or StatefulSet above it. That is not a contrived object — it is what a one-off migration job looks like six months after the person who ran it left.

Task 2 — Capture what is on the node before you touch it

This is the step the inherited runbook does not have, and it is the only one that is free.

TARGET=worker-1

kubectl get pods -A -o wide --field-selector "spec.nodeName=${TARGET}"

--field-selector spec.nodeName is the cheapest possible question: one API call, no exec, no node access, and it answers “what am I about to disturb?” precisely. Record the list. Now enrich it with the three properties that decide whether each Pod can be evicted at all:

TARGET=worker-1

kubectl get pods -A --field-selector "spec.nodeName=${TARGET}" \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,GRACE:.spec.terminationGracePeriodSeconds,VOLS:.spec.volumes[*].name'

Read the OWNER column carefully. DaemonSet means the Pod is pinned to this node by design and will be recreated here the moment the node comes back — evicting it accomplishes nothing. An empty OWNER means nothing will recreate the Pod anywhere. ReplicaSet means a controller is watching and will place a replacement.

Then ask the second question the runbook skips — where would these Pods go?

kubectl get nodes -o wide
kubectl describe node worker-2 | sed -n '/Allocated resources/,/^Events/p'

Task 3 — Run the drain with no flags, and read the first wave

TARGET=worker-1

kubectl drain "${TARGET}"

It fails. Before you read the error, run this:

kubectl get nodes

The target node now reads Ready,SchedulingDisabled. The drain cordoned the node and then failed, and it did not undo the cordon. That is by design — the drain is resumable, and un-cordoning on failure would let new Pods land on a node you are trying to empty — but it means a failed drain leaves a node quietly out of the scheduling pool. On a twelve-node rotation with a script that does not check exit codes, this is how a cluster ends up with four cordoned nodes and no capacity.

The error itself names three separate refusals in one message:

$ kubectl drain worker-1

node/worker-1 cordoned
error: unable to drain node "worker-1" due to error: [cannot delete Pods that declare no controller (use --force to override): drain-lab/legacy-import, cannot delete Pods with local storage (use --delete-emptydir-data to override): drain-lab/cache-7d4f9c6b8-mq2xv, cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-hk9wd], continuing command...
There are pending nodes to be drained:
worker-1

The exact wording of these three strings has changed between kubectl releases, so compare against what your cluster actually prints rather than against the text above — that comparison is part of the deliverable. What has not changed is the structure, and the structure is the lesson: all three arrived together, before any eviction was attempted. They are a client-side filter. kubectl listed the Pods on the node, checked each one against three rules, collected every failure, and refused as a batch.

That is why the fix is a flag. You are not arguing with the cluster; you are telling your own client that you already know.

Task 4 — Answer each refusal on purpose

Do not add all three flags at once. Add them one at a time and watch the error shrink, because each one is a different admission.

--ignore-daemonsets costs nothing. Read the flag’s name literally: it does not evict DaemonSet Pods, it tells the drain to stop refusing on their account and carry on. They keep running on the node throughout, which is the entire point. A DaemonSet Pod exists to serve the node — the CNI agent, kube-proxy, the log shipper — and the DaemonSet controller ignores the unschedulable marking, so any eviction would be undone on the same node within seconds. Evicting the CNI agent on a node you are about to work on would also take the node’s networking with it, which is why kubectl drain will not delete a DaemonSet Pod even when you ask it to.

TARGET=worker-1

kubectl drain "${TARGET}" --ignore-daemonsets

The DaemonSet line is gone; two refusals remain. This is why every drain command in every runbook you will ever read carries this flag: on a kubeadm cluster, kube-proxy alone guarantees the refusal.

--delete-emptydir-data destroys data. An emptyDir lives and dies with the Pod. Evicting the cache Pod deletes /scratch and everything in it, and no controller anywhere can bring it back. The flag is you signing for that. Before you sign, find out what is in there:

POD=$(kubectl -n drain-lab get pod -l app=cache -o jsonpath='{.items[0].metadata.name}')

kubectl -n drain-lab exec "$POD" -- ls -la /scratch

For a cache this is trivially fine. For a Pod using emptyDir as a staging area mid-upload, it is not, and the only way to tell the two apart is to look.

--force deletes a Pod permanently. legacy-import has no controller. The flag does not “force harder”; it tells kubectl to delete the Pod outright and accept that nothing will ever recreate it. Find out what it is first:

kubectl -n drain-lab describe pod legacy-import | sed -n '1,25p'
kubectl -n drain-lab logs legacy-import --tail=20

Now run it with all three, and note carefully that you have written down what each one cost.

TARGET=worker-1

kubectl drain "${TARGET}" --ignore-daemonsets --delete-emptydir-data --force

Task 5 — The second wave: a budget that forbids everything

The command no longer fails immediately. Instead it starts evicting and then stops making progress, repeating a message every five seconds:

evicting pod drain-lab/web-6c8f7d9b74-4kx2p
error when evicting pods/"web-6c8f7d9b74-4kx2p" -n "drain-lab" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

This refusal is a different animal, and everything about how you handle it differs:

Stop the drain with Ctrl-C and ask the cluster why, rather than reading the retry message again:

kubectl get pdb -n drain-lab

$ kubectl get pdb -n drain-lab

NAME      MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
web-pdb   4               N/A               0                     14m

ALLOWED DISRUPTIONS 0 is the whole diagnosis, and the two numbers beside it say which kind of zero this is. EXPECTED PODS equal to the replica count with CURRENT HEALTHY equal to it too means the workload is entirely healthy and the budget still permits nothing — so this is not a workload that needs time to recover, it is a spec that forbids voluntary disruption outright.

kubectl describe pdb web-pdb -n drain-lab

Compare currentHealthy against desiredHealthy. Four healthy, four desired, zero disruptions allowed. Waiting will never change this. minAvailable equal to the replica count is a permanent block, not a transient one — which is precisely why reading ALLOWED DISRUPTIONS beats waiting to see whether the retry loop eventually gets through.

The repair is to change the contract, and the choice is a real one:

kubectl -n drain-lab patch pdb web-pdb --type=merge \
  -p '{"spec":{"minAvailable":null,"maxUnavailable":1}}'

kubectl get pdb -n drain-lab

maxUnavailable: 1 says “at most one replica may be voluntarily disrupted at a time” and, unlike minAvailable: 3, it keeps meaning that if the Deployment is later scaled or driven by an autoscaler. ALLOWED DISRUPTIONS should now read 1.

Task 6 — The drain that never returns

Re-run the drain. It gets further and then hangs on one Pod with no error at all:

TARGET=worker-1

kubectl drain "${TARGET}" --ignore-daemonsets --delete-emptydir-data --force

slow-shutdown ignores SIGTERM and asked for a 300-second grace period. The kubelet is honouring exactly what the Pod requested: send SIGTERM, wait 300 seconds, then SIGKILL. The drain is not stuck and nothing is broken — it is being polite, silently, for five minutes.

This is the failure mode that gets a drain killed with Ctrl-C and diagnosed as a cluster fault. Ask the Pod what it asked for:

kubectl -n drain-lab get pod -l app=slow-shutdown \
  -o jsonpath='{.items[0].spec.terminationGracePeriodSeconds}{"\n"}'

Three hundred. Now you know the wait is a contract, not a hang. Two different answers, and they are not equivalent:

TARGET=worker-1

# Bound how long YOU wait. The Pod still gets its full grace period.
kubectl drain "${TARGET}" --ignore-daemonsets --delete-emptydir-data --force --timeout=60s

--timeout gives up and returns non-zero, leaving the node cordoned and partially drained. Nothing was cut short; you simply stopped waiting. In a rotation script this is the flag that keeps one slow Pod from consuming the whole maintenance window, and its non-zero exit is what tells the script to stop rather than move on to node two.

TARGET=worker-1

# Shorten what the POD gets. This overrides its request.
kubectl drain "${TARGET}" --ignore-daemonsets --delete-emptydir-data --force --grace-period=10

--grace-period overrides the Pod’s own terminationGracePeriodSeconds. Here it is harmless, because this container ignores SIGTERM and was never going to use the time. For a database flushing to disk, it is a data-integrity decision taken on the workload’s behalf without asking. The inherited runbook’s --grace-period=0 means “SIGKILL everything immediately, cluster-wide, on every rotation”.

Task 7 — Confirm the node is empty, then return it to service

TARGET=worker-1

kubectl get pods -A -o wide --field-selector "spec.nodeName=${TARGET}"

What remains should be DaemonSet Pods only — the ones --ignore-daemonsets told the drain to skip. That is a drained node: empty of everything that could move, still running the things that serve the node itself.

This is where a real maintenance window does its work. This lab deliberately does not reboot the node: a reboot in a nested environment introduces a class of failure this lab is not about, and the drain is what is being taught. If you are in A-physical mode and want the full rehearsal, reboot now and wait for the node to return Ready before continuing.

Look at what the drain did to the pinned workloads:

kubectl -n drain-lab get pods -o wide
kubectl -n drain-lab describe pod -l app=cache | sed -n '/Events/,$p'

cache and slow-shutdown are Pending. Their replacement Pods can only run on the target node, and the target node is cordoned, so the scheduler has nowhere to put them — 0/3 nodes are available: 1 node(s) were unschedulable. legacy-import is not listed at all; --force deleted it and nothing exists to recreate it.

The drain reported success and three of your four workloads are down. That gap — between “the node is empty” and “the workload is healthy” — is the single most important thing on this page.

TARGET=worker-1

kubectl uncordon "${TARGET}"
kubectl get nodes
kubectl -n drain-lab get pods -o wide -w

cache and slow-shutdown schedule within seconds of the uncordon. The web replicas that moved to other nodes do not come back. Kubernetes has no rebalancer: once a Pod is placed, nothing moves it because a better node appeared. After a full rotation your workload is unevenly distributed until something forces a reschedule, such as kubectl rollout restart.

Task 8 — The refusal ledger

Write this table out from your own notes before reading it here. Each row is a flag from the inherited runbook and what it actually bought:

Only the first row is free. The line your predecessor left runs the other three against every node in the estate, every rotation, without anyone deciding.

Validation

Save as validate.sh and run it. It exits non-zero while anything is still wrong, which is the point: a validation you can pass by squinting is not a validation.

#!/usr/bin/env bash
set -euo pipefail
NS=drain-lab
TARGET=${TARGET:-worker-1}

sched=$(kubectl get node "$TARGET" -o jsonpath='{.spec.unschedulable}')
if [ -n "$sched" ]; then
  echo "FAIL $TARGET is still cordoned"
  exit 1
fi
echo "ok   $TARGET is schedulable"

status=$(kubectl get node "$TARGET" \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
if [ "$status" != "True" ]; then
  echo "FAIL $TARGET is not Ready"
  exit 1
fi
echo "ok   $TARGET is Ready"

ready=$(kubectl -n "$NS" get deploy web -o jsonpath='{.status.readyReplicas}')
if [ "${ready:-0}" -ne 4 ]; then
  echo "FAIL web has ${ready:-0}/4 ready replicas"
  exit 1
fi
echo "ok   web has 4/4 ready replicas"

allowed=$(kubectl -n "$NS" get pdb web-pdb -o jsonpath='{.status.disruptionsAllowed}')
if [ "${allowed:-0}" -lt 1 ]; then
  echo "FAIL web-pdb allows ${allowed:-0} disruptions; a rotation would stall"
  exit 1
fi
echo "ok   web-pdb allows $allowed disruption(s)"

pending=$(kubectl -n "$NS" get pods --field-selector status.phase=Pending \
  -o name | wc -l)
if [ "$pending" -ne 0 ]; then
  echo "FAIL $pending Pod(s) still Pending"
  exit 1
fi
echo "ok   no Pending Pods"

if kubectl -n "$NS" get pod legacy-import >/dev/null 2>&1; then
  echo "FAIL legacy-import still exists; --force did not do what you think"
  exit 1
fi
echo "ok   legacy-import is gone, as --force promised"

Five different claims are being proved, and they are not the same claim. The node is schedulable again (you cleaned up after yourself). The node is Ready (the maintenance did not break it). web has all four replicas (the drain moved work without losing it). The PDB now permits a disruption (the next node in the rotation will not stall on the same budget). And legacy-import is gone — proved positively, because the cost of --force should appear in the validation rather than being discovered later.

Expected Outcome

worker-1     Ready, schedulable, DaemonSet Pods only during the drain
drain-lab
├── deploy/web            4/4 Ready, replicas redistributed, none returned by itself
├── pdb/web-pdb           maxUnavailable: 1, ALLOWED DISRUPTIONS 1
├── deploy/cache          1/1 Ready, /scratch empty — the emptyDir did not survive
├── deploy/slow-shutdown  1/1 Ready, replaced after a 10s grace period
└── pod/legacy-import     does not exist

validate.sh exits 0, and you have a written refusal ledger naming each flag in the inherited runbook and what it destroys.

Troubleshooting

The drain refuses with a DaemonSet name you do not recognise. Every kubeadm cluster runs kube-proxy as a DaemonSet, and the CNI ships one too. That is normal and is why --ignore-daemonsets appears in essentially every real drain command. Confirm the owner before assuming: kubectl -n kube-system get pod POD -o jsonpath='{.metadata.ownerReferences[0].kind}'.

You have only one schedulable node. The evicted web replicas will go Pending rather than moving, and the validation’s readyReplicas check will fail. Every refusal task still works. To finish, uncordon first and let the replicas reschedule before running validate.sh.

The drain hangs with no message at all, on a Pod you did not create. Read its grace period before assuming a fault. A long, silent wait is usually the kubelet honouring a terminationGracePeriodSeconds the workload asked for, and the drain prints nothing while it waits.

ALLOWED DISRUPTIONS is 0 but EXPECTED PODS is also 0. That is a different failure: the PDB’s selector matches no Pods at all, so the budget is inert and protecting nothing. Compare kubectl -n drain-lab get pdb web-pdb -o jsonpath='{.spec.selector}' against kubectl -n drain-lab get pods --show-labels. A budget that matches nothing never blocks a drain, which is why nobody finds it until the outage.

A web Pod is Running but not Ready, and the PDB will not budge even after the patch. ALLOWED DISRUPTIONS counts healthy Pods. With maxUnavailable: 1 and one replica already unhealthy, the budget is legitimately exhausted and the drain is correctly refusing. That is the PDB working; fix the unhealthy Pod, do not widen the budget.

The node stayed cordoned after you gave up. Expected. A failed drain never uncordons. kubectl uncordon is the only thing that clears it, and forgetting is how a cluster silently loses capacity node by node.

kubectl drain reports the node drained but Pods are still listed on it. Check whether those Pods set spec.nodeName directly. A Pod placed without the scheduler ignores the cordon entirely, because the cordon is a scheduler input and that Pod never asked the scheduler.

Cleanup

Two things to undo: the namespace, and the node’s unschedulable field. Missing the second is the mistake that matters.

$ kubectl uncordon worker-1 && kubectl delete namespace drain-lab

TARGET=worker-1

kubectl get node "${TARGET}" -o jsonpath='{.spec.unschedulable}{"\n"}'
kubectl get node "${TARGET}" -o jsonpath='{.spec.taints}{"\n"}'
kubectl get namespace drain-lab 2>&1 | grep -q NotFound && echo "namespace gone"
rm -f drain-lab-web.yaml drain-lab-pinned.yaml validate.sh

The first command should print an empty line: spec.unschedulable is cleared rather than set to false, so an empty result is the correct one. The second should not contain node.kubernetes.io/unschedulable — uncordon removes the built-in taint alongside the flag, and a leftover taint there means something other than the cordon put it on.

Nothing in kube-system was modified, no DaemonSet was touched, and no node was rebooted, so there is nothing else to restore.

Production notes

Cordon and drain are two decisions, and separating them buys you time. Cordon is reversible, instant, and disturbs nothing that is running. On a node behaving suspiciously, cordoning stops the bleeding — no new work lands there — while you decide, without a clock running, whether to drain. Running kubectl drain because you are worried is a much larger commitment than it looks.

Exit codes are the whole safety mechanism in a rotation. A drain that fails leaves the node cordoned and half-empty. A loop that ignores the exit code moves to the next node and cordons that one too. Every rotation script needs set -euo pipefail, an explicit --timeout, and a stop-on-first-failure rule, because the failure mode of getting this wrong is losing capacity across the estate faster than you notice.

Audit your PDBs before the window, not during it. One command across the cluster tells you which node rotations will stall:

kubectl get pdb -A

Any row with ALLOWED DISRUPTIONS of 0 on a healthy workload will block a drain indefinitely. Any row with EXPECTED PODS of 0 is a budget protecting nothing. Both are cheaper to find at 14:00 the day before than at 02:30 mid-rotation.

Hold is a first-class outcome. When a PDB blocks you and the workload is not yours, the honest options are: get the owner to authorise a budget change, or stop. Both need a name and a time — who owns the answer, and when the hold expires. Widening someone else’s availability contract at 02:30 because it was blocking your window is how a maintenance task becomes an incident, and the change is invisible afterwards because nothing in the cluster records who loosened it or why.

Draining a control-plane node is a different procedure. The static Pods that make up the control plane are mirror Pods owned by the kubelet, not by any controller, so a drain skips them rather than evicting them — the API server on that node keeps running. Stopping it means stopping the kubelet, which is a separate decision with quorum implications, and is covered by the control-plane and etcd material rather than by this lab.

What You Learned

• kubectl drain refuses in two places, and they need different responses. Three client-side filters fire once, together, before anything is evicted, and are answered with a flag. The PDB refusal comes from the API server, per Pod, forever, and has no honest flag.
• A failed drain leaves the node cordoned. The cordon is the drain’s first action and is never rolled back on failure. A rotation script that ignores exit codes removes nodes from the scheduling pool silently.
• Each override flag is a signature on a specific loss. --ignore-daemonsets costs nothing, --delete-emptydir-data costs the volume’s contents, --force costs the Pod itself. Inheriting all three as a fixed string means signing for all of them on every node, forever.
• ALLOWED DISRUPTIONS is the diagnosis; the retry message is not. Zero disruptions on a fully healthy workload is a permanent block, and no amount of waiting will change it. The same message on a recovering workload clears by itself.
• A drained node is not a healthy workload. Three of the four workloads here were down at the moment the drain reported success. Capacity, node affinity and local storage all decide whether evicted Pods have anywhere to go, and the drain command reports on none of them.
• --timeout bounds your patience; --grace-period bounds the Pod’s shutdown. They look interchangeable in a runbook and are not. One of them makes a decision on the workload’s behalf about how much of its shutdown it gets to finish.
• Nothing rebalances afterwards. Uncordoning returns capacity, not workload. Pods that moved during a rotation stay where they went until something forces a reschedule.