Lab 10: Node affinity and scheduling constraints

Objective

By the end of this lab you will have driven a single Deployment through five placement regimes on the same cluster — unconstrained, nodeSelector, required node affinity, an impossible required rule, and preferred affinity — and then layered a node taint on top of the last one. The point is not the YAML. The point is that after Task 8 you can look at one FailedScheduling line and say, without guessing, which filter plugin produced each clause of it and therefore what has to change.

You will also produce the observation that most operators never make deliberately: that a running Pod does not care whether its node still satisfies the rule that placed it there.

Architecture

One control-plane node and three workers. The workers are the interesting part; cp-1 participates only as the node that keeps not being chosen, which is itself a lesson in Task 2.

flowchart TB
    S[kube-scheduler on cp-1] --> F[Filter plugins]
    F --> N0["cp-1<br/>taint: control-plane:NoSchedule"]
    F --> N1["worker-1<br/>disk=ssd"]
    F --> N2["worker-2<br/>disk=ssd"]
    F --> N3["worker-3<br/>disk=hdd"]
    N1 --> SC[Score plugins]
    N2 --> SC
    N3 --> SC
    SC --> B[Bind]

The labels in that diagram do not exist yet. You add them in Task 3, and Cleanup removes them.

Requirements

• A disposable kubeadm cluster running Kubernetes 1.34.x, with one control-plane node (cp-1) and three worker nodes (worker-1, worker-2, worker-3), all Ready, with a working CNI.
• kubectl 1.34.x on your workstation with a cluster-admin kubeconfig. Node labels and taints are cluster-scoped objects; a namespace-scoped role cannot do this lab.
• Roughly 500 mCPU and 1 GiB of schedulable memory free across the workers. The workloads are nginx Pods with 50 mCPU requests, so the cluster does not need to be large — it needs to be empty enough that the scheduler’s decisions are driven by your constraints rather than by capacity.
• Internet access, or a registry mirror, for nginx:1.27.2. Nothing else is pulled.
• No out-of-band access requirement. Nothing here touches SSH, the primary interface, or the firewall, so a mistake cannot lock you out of a node. It can make workloads unschedulable, which Cleanup addresses.

Two workers will get you through Tasks 1–4 and 6–8, but Task 5 needs a third node to be legible: the whole demonstration is that one node changes state while the others do not. If you only have two workers, add one before starting.

Scenario

An nginx-backed web tier runs six replicas and is currently placed wherever the scheduler feels like putting it. Storage has told you the pods want SSD where possible. Somebody has already written a nodeSelector into a different service’s manifest and it has been Pending for two days, which is why you are being asked to do this properly rather than by copying theirs.

You have a disposable four-node cluster to work it out on.

Tasks

Task 1: Capture the starting state and create the namespace

The cluster is not fresh — kubeadm has already labelled and tainted things, and your CNI may have added labels of its own. Capture what is there before you change it, because Cleanup restores rather than deletes.

mkdir -p "$HOME/k8s-scheduling-lab"
cd "$HOME/k8s-scheduling-lab"

kubectl get nodes -o wide | tee nodes.before.txt
kubectl get nodes --show-labels | tee node-labels.before.txt

Taints are not labels and are not in --show-labels. Capture them separately:

cd "$HOME/k8s-scheduling-lab"

kubectl get nodes -o json \
  | jq -r '.items[] | .metadata.name + " " + (.spec.taints // [] | tostring)' \
  | tee node-taints.before.txt

If jq is not available, kubectl get nodes -o jsonpath produces the same record without the pretty-printing:

cd "$HOME/k8s-scheduling-lab"

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.taints}{"\n"}{end}' \
  | tee node-taints.before.txt

Then create a namespace so every workload in this lab is deletable in one command:

$ kubectl create namespace sched-lab

Task 2: Baseline placement, and why nothing lands on cp-1

Write web-baseline.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: sched-lab
  labels:
    app: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 64Mi

Apply it and record where the six Pods land:

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-baseline.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o wide

Reduce that to a distribution you can put in placement.md:

kubectl -n sched-lab get pods \
  -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

$ kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c

      2 worker-1
    2 worker-2
    2 worker-3

The split across workers is a scoring outcome and it is not guaranteed to be even; if you get 3/2/1 you have not done anything wrong. What is guaranteed is that none of the six is on cp-1, and it is worth being precise about why, because it is not affinity:

kubectl describe node cp-1 | grep -A 2 '^Taints:'

kubeadm applies node-role.kubernetes.io/control-plane:NoSchedule at init. Your Pods carry no toleration for it, so the TaintToleration filter plugin eliminates cp-1 before scoring ever runs. This is the repulsion model at work and you have been relying on it since Lab 1 without configuring anything.

Record the baseline row in placement.md: no constraint · feasible nodes: worker-1, worker-2, worker-3 · observed 2/2/2.

Task 3: nodeSelector, and what actually moves a Pod

Label the nodes. disk is not a well-known key — you own it, and owning it means nothing sets it for you:

$ kubectl label node worker-1 disk=ssd && kubectl label node worker-2 disk=ssd && kubectl label node worker-3 disk=hdd

Verify with -L, which promotes a label to a column instead of dumping the whole set:

kubectl get nodes -L disk

Now copy web-baseline.yaml to web-nodeselector.yaml and add four lines to the Pod template — nodeSelector is a peer of containers, inside spec.template.spec:

    spec:
      nodeSelector:
        disk: ssd
      containers:
        - name: nginx
          image: nginx:1.27.2

Apply it and watch the rollout:

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-nodeselector.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

All six now sit on worker-1 and worker-2. Be exact about the mechanism: the scheduler did not move anything. Adding nodeSelector changed the Pod template, so the Deployment controller rolled out replacement Pods, and the replacements were placed under the new constraint. Nothing in Kubernetes relocates a bound Pod because its node stopped matching. Task 5 makes that concrete.

Task 4: The same intent as required node affinity, plus composition

Copy to web-affinity-required.yaml, drop the nodeSelector, and express the rule with matchExpressions:

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: disk
                    operator: In
                    values: ["ssd", "nvme"]
      containers:
        - name: nginx
          image: nginx:1.27.2

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-required.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Same placement, more expressive rule: In ["ssd", "nvme"] is a set, so adding NVMe nodes later needs no manifest change.

Confirm the rule is on the Pods and not merely in the Deployment. This matters because kubectl describe pod does not print node affinity — it prints Node-Selectors and Tolerations and nothing else about placement, which is a common source of “the rule isn’t applied” panics:

POD=$(kubectl -n sched-lab get pods -l app=web -o jsonpath='{.items[0].metadata.name}')

kubectl -n sched-lab get pod "$POD" -o jsonpath='{.spec.affinity.nodeAffinity}'
echo

Now demonstrate the composition rules by changing the feasible set rather than by reading the spec. Add a second term, and give that term two expressions:

          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: disk
                    operator: In
                    values: ["ssd", "nvme"]
              - matchExpressions:
                  - key: disk
                    operator: In
                    values: ["hdd"]
                  - key: role
                    operator: In
                    values: ["overflow"]

Save it as web-affinity-or.yaml. Before applying, predict the feasible set. The second term needs disk=hdd and role=overflow; worker-3 has the first and not the second, so the term is unsatisfied and the feasible set is still two nodes. Apply and scale to nine to prove it:

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-or.yaml
kubectl -n sched-lab scale deployment/web --replicas=9
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Nine Pods, two nodes. Now satisfy the second expression and scale again:

kubectl label node worker-3 role=overflow
kubectl -n sched-lab scale deployment/web --replicas=12
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

The three new Pods land on worker-3, because the second term is now satisfied and terms are ORed. You have just observed both rules: OR between terms, AND between expressions inside a term. Record both rows in placement.md.

Task 5: IgnoredDuringExecution — the drift you cannot see

Take the label straight back off worker-3:

$ kubectl label node worker-3 role-

Check the Pods on worker-3:

kubectl -n sched-lab get pods -o wide --field-selector spec.nodeName=worker-3

They are all still Running. This is what IgnoredDuringExecution means, and it is the half of the field name that operators skip: the rule is evaluated once, at binding time, and never again. The Deployment’s Pod template still demands role=overflow for anything on an hdd node. No node satisfies that. And twelve Pods are healthy.

Prove that the rule is still live for new Pods by deleting one:

VICTIM=$(kubectl -n sched-lab get pods -o name \
  --field-selector spec.nodeName=worker-3 | head -1)

kubectl -n sched-lab delete "$VICTIM"
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

The replacement lands on worker-1 or worker-2. The survivors on worker-3 do not move.

Write the answer to this in placement.md, in your own words: why is this drift invisible until a drain, and what would you monitor to see it sooner? A usable answer names something you could actually query — for example, comparing each Deployment’s required affinity keys against kubectl get nodes -L output on a schedule, and alerting when the feasible node count for a workload drops below its replica count.

Task 6: An impossible constraint, and reading the message

Copy to web-affinity-impossible.yaml and require a label no node has:

          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: disk
                    operator: In
                    values: ["nvme"]

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-impossible.yaml
kubectl -n sched-lab get pods -o wide | head -8

The rollout does not fail. It stalls: the Deployment brings up surge Pods that cannot be scheduled and refuses to terminate the old ones, so the old Pods keep serving and kubectl rollout status hangs. Let it hang for thirty seconds, press Ctrl-C, and read the event instead:

PENDING=$(kubectl -n sched-lab get pods --field-selector status.phase=Pending \
  -o jsonpath='{.items[0].metadata.name}')

kubectl -n sched-lab describe pod "$PENDING" | tail -12

$ kubectl -n sched-lab describe pod $PENDING | tail -12

Events:
Type     Reason            Age   From               Message
----     ------            ----  ----               -------
Warning  FailedScheduling  61s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

Take it apart clause by clause, and write the attribution into placement.md:

The last row is the most useful one and the most ignored. “Preemption is not helpful” is the scheduler telling you the problem is a predicate, not a resource shortage — so adding nodes of the same kind will not fix it either.

Confirm the diagnosis in one command rather than by rereading the YAML:

kubectl get nodes -l disk=nvme

An empty result is the whole story. Constraint asks for a label the cluster does not have; the fix is to label a node or change the constraint, and which one is correct depends on whether NVMe nodes are supposed to exist.

Task 7: preferred — what “soft” actually buys you

Copy to web-affinity-preferred.yaml and turn the same impossible rule into a preference:

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: disk
                    operator: In
                    values: ["nvme"]

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-preferred.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Every Pod schedules, spread across all three workers. The preference matched nothing, so no node received the 100-point bonus, and the placement was decided entirely by the scheduler’s other scoring plugins. A preference that cannot be satisfied is silent — there is no event, no condition, and no warning. The workload is running and quietly getting none of what you asked for.

Now make the preference meaningful and watch it bite:

kubectl label node worker-3 disk=nvme --overwrite
kubectl -n sched-lab rollout restart deployment/web
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

worker-3 now attracts a disproportionate share — not all of it, because the 100-point bonus is added to a score that other plugins also contribute to, and because worker-3 has finite capacity. That is the honest description of weight: points added during scoring, not a guarantee and not a percentage. A weight: 1 would have been swamped by the other plugins and produced no visible change at all.

Put worker-3 back before continuing:

kubectl label node worker-3 disk=hdd --overwrite

Task 8: A taint on top, and two rejection reasons in one event

Return to the required rule that worked, then repel the nodes it selects:

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-required.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s

$ kubectl taint nodes worker-1 tier=batch:NoSchedule && kubectl taint nodes worker-2 tier=batch:NoSchedule

Check the running Pods first:

kubectl -n sched-lab get pods -o wide | head -5

Still running. NoSchedule is a scheduling decision, not a runtime contract — the kubelet does not re-admit a Pod when the node’s taints change. Only NoExecute evicts, and you are deliberately not using it here.

Force new Pods and read the event:

kubectl -n sched-lab scale deployment/web --replicas=15

PENDING=$(kubectl -n sched-lab get pods --field-selector status.phase=Pending \
  -o jsonpath='{.items[0].metadata.name}')

kubectl -n sched-lab describe pod "$PENDING" | tail -8

$ kubectl -n sched-lab describe pod $PENDING | tail -8

  Warning  FailedScheduling  12s   default-scheduler  0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {tier: batch}, 1 node(s) didn't match Pod's node affinity/selector.

Three clauses, three different nodes, two different plugins. worker-1 and worker-2 are feasible for the affinity rule and rejected by the taint; worker-3 tolerates everything and is rejected by the affinity rule. Neither half is wrong on its own, and neither half alone explains the outage — which is exactly why the message enumerates them separately.

Add the toleration to web-affinity-required.yaml, as a peer of affinity:

      tolerations:
        - key: tier
          operator: Equal
          value: batch
          effect: NoSchedule

cd "$HOME/k8s-scheduling-lab"

kubectl apply -f web-affinity-required.yaml
kubectl -n sched-lab rollout status deployment/web --timeout=180s
kubectl -n sched-lab get pods -o custom-columns=NODE:.spec.nodeName --no-headers \
  | sort | uniq -c

Fifteen Pods across worker-1 and worker-2. A toleration does not attract a Pod to a tainted node — it only stops the node being eliminated. The affinity rule is still what chose those two.

Validation

Each of these must hold before you call the lab done.

1. placement.md contains at least five rows: baseline, nodeSelector, required affinity, the OR/AND composition pair, and preferred. Each row names the feasible node set and the observed distribution.

2. The impossible-constraint event is recorded verbatim, with every clause attributed to a filter plugin.

3. The current placement matches the toleration state:

   kubectl -n sched-lab get pods -o wide --no-headers \
     | awk '{print $7}' | sort | uniq -c

   Fifteen Pods, on worker-1 and worker-2 only, none Pending.

4. Node state matches what you set:

   kubectl get nodes -L disk,role
   kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.taints}{"\n"}{end}'

   worker-1 and worker-2 carry disk=ssd and the tier=batch:NoSchedule taint; worker-3 carries disk=hdd and no lab taint; no node carries role=overflow.

5. The scheduling failures you produced are in the event log, and nothing else is:

   kubectl -n sched-lab get events --field-selector reason=FailedScheduling \
     --sort-by=.lastTimestamp

6. You can state, without looking it up, which of the two rules you wrote would have left a workload Pending after a node reboot and which would have left it silently mis-placed.

Expected Outcome

k8s-scheduling-lab/
├── node-labels.before.txt
├── node-taints.before.txt
├── nodes.before.txt
├── placement.md
├── web-affinity-impossible.yaml
├── web-affinity-or.yaml
├── web-affinity-preferred.yaml
├── web-affinity-required.yaml
├── web-baseline.yaml
└── web-nodeselector.yaml

On the cluster: a sched-lab namespace with one web Deployment at 15/15 ready, all Pods on the two disk=ssd nodes, tolerating tier=batch:NoSchedule. Three worker nodes carrying a disk label, two of them tainted. No Pending Pods.

Troubleshooting

Every Pod is Pending immediately after Task 3, including on the SSD nodes. Check the label actually landed: kubectl get nodes -L disk. A typo in the key (dissk) or in the value produces the identical didn't match Pod's node affinity/selector message as a missing label, because to the scheduler they are the same thing.

rollout status never returns and there are no Pending Pods. Look for ImagePullBackOff instead. A cluster without registry access fails here rather than at scheduling, and the symptom is a stalled rollout either way.

Pods land on worker-3 in Task 4 when you predicted they would not. Check whether role=overflow is still set from an earlier attempt: kubectl get nodes -L role. Labels persist across everything you do in the namespace.

The taint command reports “already has a taint with key tier”. A second taint on the same key replaces the first only if you pass --overwrite; otherwise it errors. Remove it by the full triple and re-add: kubectl taint nodes worker-1 tier=batch:NoSchedule-.

Removing a taint reports “not found”. The removal operator matches on the whole key=value:effect triple. kubectl taint nodes worker-1 tier- removes every taint with that key regardless of value or effect, which is usually what you want in cleanup and is not what you want in production.

Everything schedules in Task 8 despite the taints. The toleration from a previous apply is still in the template. kubectl -n sched-lab get deploy web -o jsonpath='{.spec.template.spec.tolerations}' shows what the Pods actually carry.

Cleanup

The namespace holds the workloads; the labels and taints are cluster-scoped and outlive it. Remove all three, in that order.

$ kubectl delete namespace sched-lab --wait=true

Remove the taints. Use the key-only form so a mistyped value cannot leave one behind:

kubectl taint nodes worker-1 tier- || true
kubectl taint nodes worker-2 tier- || true

Remove the labels the lab added:

kubectl label node worker-1 disk- || true
kubectl label node worker-2 disk- || true
kubectl label node worker-3 disk- || true
kubectl label node worker-3 role- || true

Now verify against the Task 1 capture rather than by eye — this is the step that catches the --overwrite you did in Task 7 and forgot:

cd "$HOME/k8s-scheduling-lab"

kubectl get nodes --show-labels > node-labels.after.txt
diff node-labels.before.txt node-labels.after.txt && echo "labels restored"

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.taints}{"\n"}{end}' \
  > node-taints.after.txt
diff node-taints.before.txt node-taints.after.txt && echo "taints restored"

The Deployments themselves went with the namespace; confirm nothing survived:

kubectl get namespace sched-lab

NotFound is the correct answer. Keep placement.md — it is the deliverable — and remove the rest of the working directory only when you are done with it.

Production notes

Map this to a real change window. A node-label change is a change to the scheduling contract of every workload that mentions that label, and it has no rollout, no canary and no revision history. In production:

• Treat a label or taint change as a cluster-wide change, with the same review a CNI upgrade gets. Before applying, list what depends on it: kubectl get deploy -A -o json filtered on the key you are about to touch.
• Taints are staged, never applied straight to NoExecute. Add NoSchedule first, verify that the workloads which should tolerate it do, then convert. Going straight to NoExecute evicts every Pod without a matching toleration, with the grace period taken from tolerationSeconds.
• Validate constraints against the cluster in CI. The Task 6 failure is entirely preventable: a pipeline step that reads the cluster’s node labels and fails the build when a required affinity key has no matching node would have caught it before the manifest merged.
• Alert on Pod phase, not replica count. A stalled rollout keeps the old Pods serving, so the READY column stays reassuring for ten minutes. Alert on Pending Pods older than a few minutes and on the Deployment’s Progressing condition.

What You Learned

• nodeSelector and required affinity express the same class of rule, and affinity expresses more of it. You migrated one to the other mechanically and got set-based operators and OR composition for free.
• Terms are ORed; expressions inside a term are ANDed. You proved it by changing one label and watching the feasible set change, not by reading it.
• IgnoredDuringExecution is the operative half of the field name. A running Pod survives losing the label that placed it. That is a drift you cannot see until everything reschedules at once.
• A FailedScheduling message is a set of per-plugin reasons, aggregated and counted. You attributed every clause of a three-clause message to the plugin that produced it, and read “preemption is not helpful” as “this is a predicate, not a shortage”.
• Required fails loudly, preferred fails silently. An unsatisfiable preference produces a running workload getting none of what you asked for, and no event anywhere says so.
• A toleration permits, it does not attract. Taints eliminate nodes; affinity selects them; and a single Pending Pod can be rejected by both at once for different nodes.