Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

intermediatekubernetes-taint~30 min

Pod stuck Pending due to taint

Reported symptoms

  • ●The payments-api Deployment has been drifting downwards for three days: 12 desired, 7 Ready this morning, and it loses roughly one more replica every ninety minutes
  • ●The capacity dashboard shows the pay node pool at 34 percent CPU, so the obvious hypothesis — no room — is contradicted by the first graph anyone opens
  • ●The payments team fluent-bit DaemonSet reports 6 desired and 4 current, and logs are missing from two nodes; it is open as a separate logging ticket
  • ●Node-level metrics from those same two nodes are present and completely normal, so the nodes themselves look healthy
  • ●The HorizontalPodAutoscaler shows desired 12 against current 7, which has the team looking at the metrics pipeline
  • ●A rolling rebuild of the pay node pool started on Saturday and is stalled: the drain of the next node is being refused by a PodDisruptionBudget, which is being treated as the blocker
  • ●Nothing has been deployed to payments-api in three weeks — no image change, no manifest change, no Helm release

Evidence

  • · kubectl get pods shows 7 payments-api Pods Running with a node assigned and 5 Pending with none
  • · The FailedScheduling event reads: 0/24 nodes are available: 2 node(s) had untolerated taint {dedicated: payments}, 4 Insufficient cpu, 18 node(s) did not match Pod node affinity/selector
  • · The three counts in that message add to 24, and the count of 2 is exactly the number of nodes rebuilt so far
  • · Listing the pool nodes with their taint key, value and effect as separate columns shows all six agree on key and value and disagree on effect: four NoSchedule, two NoExecute
  • · The two NoExecute nodes are the two youngest nodes in the pool, both created since Saturday
  • · The payments-api toleration reads key dedicated, operator Equal, value payments, effect NoSchedule
  • · The platform node-exporter DaemonSet is running on all six pool nodes; its toleration is operator Exists with no key at all
  • · The node-pool repository shows a commit from Friday changing the pool taint effect from NoSchedule to NoExecute, reviewed as a hardening change
Diagnosis and resolutionclick to reveal

Root cause

A toleration matches a taint only when the key, the operator and value, and the effect all match. Effect is part of the taint identity, not a severity dial applied to an otherwise stable taint. On Friday the node-pool template was changed so that pool nodes register with dedicated=payments:NoExecute instead of dedicated=payments:NoSchedule, on the reasoning that evicting a stray Pod is strictly stronger than merely refusing to place one. It is not stronger; it is different. NoExecute and NoSchedule are two distinct taints that happen to share a key and a value, and every toleration in the cluster that named the effect NoSchedule stopped matching the pool the moment the first node came up under the new template. Because the nodes register with the taint already applied, the NoExecute effect had nothing to evict at join time, so the change produced no eviction storm and no event anyone would look at. And because the pool is being replaced one node at a time with the remaining old nodes still carrying the old taint and still serving, capacity is withdrawn gradually rather than all at once: each rebuilt node removes one node worth of placement without producing a single moment that looks like an incident. The DaemonSet gap has the same cause and the different behaviour of the two DaemonSets is the confirmation — the team fluent-bit tolerates the effect by name and has stopped landing, while the platform node-exporter carries a key-less Exists toleration that matches every taint in existence and lands regardless, which is exactly why node metrics from the two unusable nodes look perfectly healthy.

Remediation

Stop the bleed before repairing anything: pause the pool rebuild, because each further node converted removes another node worth of placement from a tier that is already short. That costs nothing, is reversible, and buys the time to decide properly. Then take the two rebuilt nodes back by hand — removing the NoExecute taint and adding the NoSchedule one is two commands per node, and neither can evict anything, since removing a taint never evicts and NoSchedule never evicts. That restores placement in seconds and fixes nothing durable: the template will recreate the condition on the next node, so the real decision is which side of the mismatch is wrong. If NoExecute was an accident, revert the template and rebuild one node to prove the revert. If NoExecute is genuinely wanted, then every workload that targets the pool has to be enumerated and updated, and the cheap-looking edit — an Exists operator on the key with no effect, which tolerates all effects for that key — costs you the lever you just built, because the workload will then also ignore a NoExecute applied deliberately for maintenance. What must not happen in either case is deleting or overriding the PodDisruptionBudget to unstick the drain. The PDB is not the blocker; it is the only component in this incident that is doing its job, and removing it converts a five-replica shortfall into a full outage.

Verification

Read the FailedScheduling arithmetic rather than the Pod phase. The message splits the cluster into disjoint counts and the repair is proved when the untolerated-taint count is zero and the remaining counts add up to the nodes that are legitimately unavailable — outside the pool, or genuinely full. Confirm all twelve replicas Ready and the team DaemonSet reporting equal desired and current across the pool, since the DaemonSet is the check that covers the nodes the Deployment happens not to have picked. Prove the workload is still evictable: drain a rebuilt node and confirm its Pods move rather than staying put, because a toleration widened far enough to fix the symptom will also swallow the maintenance lever and you will not discover that until the next planned drain. Prove the PodDisruptionBudget still refuses: with the tier at full replicas it should permit a drain, and with the tier short it should refuse again. Finally, rebuild one further node from the pool template and confirm a payments Pod lands on it unattended. Two nodes patched by hand prove only that the two nodes were patched; the template is what produced the incident and the template is what has to pass.

Prevention

Treat effect as part of a taint's identity. Any change to it is a breaking change to every toleration in the cluster that names the old one, and it will read in review as a tightening rather than as a replacement unless someone says otherwise in the diff. Node-pool templates are code with a delayed rollout, so diff each node's actual taints against its pool's declared taints on a schedule and alert on divergence; that check would have fired on Saturday morning. Gate the change rather than the outcome: before a pool rebuild, bring up one node from the new template and assert that a canary Pod of every workload class targeting the pool schedules onto it, which is the cheapest possible test and the only one that exercises the template. Alert on Pods Pending for more than five minutes, carrying the FailedScheduling reason, so a slow bleed produces a page instead of a drift on a dashboard. Keep tolerations narrow but effect-complete, one taint per node with a clear key, and never ship a key-less Exists toleration on anything that is not deliberately meant to run on a broken node — it matches not-ready and unreachable as well, and a Pod carrying it will sit on a dead node rather than be rescheduled. And never override a PodDisruptionBudget to unblock a drain; a PDB refusing is the cluster telling you the drain is unsafe, which on this occasion it was.

Reported symptoms

payments-api is a twelve-replica Deployment pinned by nodeSelector to a six-node pool labelled pool=pay. On Saturday it was at twelve. This morning it is at seven, and it has been shedding roughly one replica every ninety minutes since.

Nothing has been deployed to it in three weeks. No image, no manifest, no Helm release, no configuration change of any kind.

The first graph anyone opens contradicts the obvious hypothesis. The pool is at 34 percent CPU. There is visibly room.

Three other things are live in the channel:

  • The payments team’s fluent-bit DaemonSet shows 6 desired, 4 current. Two nodes have no log shipping. Filed separately, as a logging problem.
  • Node-level metrics from those same two nodes are present and entirely unremarkable, which is why “the nodes are fine” is not in dispute.
  • The HorizontalPodAutoscaler reports desired 12 against current 7, so a third person is looking at the metrics pipeline.

And one thing is stalled. A rolling rebuild of the pay pool began on Saturday — old nodes drained and destroyed, new ones brought up from the pool template. It got through two nodes and stopped: the drain of the third is being refused by a PodDisruptionBudget. That refusal is currently the top item on the incident, on the grounds that it is what is blocking progress.

Evidence provided

Read-only / Safethree counts; they add to 24
$ kubectl -n payments describe pod payments-api-6b47c9f8d-x2ktp | sed -n '/Events/,$p'
Events:
Type     Reason            Age                  From               Message
----     ------            ----                 ----               -------
Warning  FailedScheduling  2m (x39 over 3h)     default-scheduler  0/24 nodes are available: 2 node(s) had untolerated taint {dedicated: payments}, 4 Insufficient cpu, 18 node(s) didn't match Pod's node affinity/selector. preemption: 0/24 nodes are available: 2 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod, 18 Preemption is not helpful for scheduling.

Illustrative output

Read-only / Safesix nodes agreeing on two columns out of three
$ kubectl get nodes -l pool=pay -o custom-columns=NAME:.metadata.name,KEY:.spec.taints[*].key,VALUE:.spec.taints[*].value,EFFECT:.spec.taints[*].effect,AGE:.metadata.creationTimestamp
NAME     KEY         VALUE      EFFECT       AGE
pay-01   dedicated   payments   NoSchedule   2025-06-30T11:02:14Z
pay-02   dedicated   payments   NoSchedule   2025-06-30T11:04:41Z
pay-03   dedicated   payments   NoSchedule   2025-06-30T11:07:09Z
pay-04   dedicated   payments   NoSchedule   2025-06-30T11:09:52Z
pay-05   dedicated   payments   NoExecute    2026-08-16T06:18:03Z
pay-06   dedicated   payments   NoExecute    2026-08-17T18:44:37Z

Illustrative output

Read-only / Safewhat the workload tolerates
$ kubectl -n payments get deploy payments-api -o jsonpath='{.spec.template.spec.tolerations}'
[{"effect":"NoSchedule","key":"dedicated","operator":"Equal","value":"payments"}]

Illustrative output

Read-only / Safewhy the two unusable nodes report healthy metrics
$ kubectl -n monitoring get ds node-exporter -o jsonpath='{.spec.template.spec.tolerations}'
[{"operator":"Exists"}]

Illustrative output

Read-only / SafeFriday
$ git -C node-pools log --oneline -3 -- pools/pay.yaml
8d1f4a2 pay: harden pool isolation, evict strays rather than repel them
3c90bb7 pay: bump node image to 2026.08
a41e77e pay: raise pool size to 6

Illustrative output

Work the evidence before reading on

The whole answer is in the first two blocks, and the first block is one that most people skim because it is long.

  1. The FailedScheduling message splits 24 nodes into three counts. Name the set of nodes behind each count. What is significant about the count that equals 2?
  2. Six nodes in one pool. Three columns of taint. Two of the columns are identical across all six. What does the third column do to a toleration that names it?
  3. Two DaemonSets, same pool, opposite outcomes. One is missing from two nodes; one is on all six. Compare their tolerations and say which property of a toleration explains the difference.
  4. The drain is blocked by a PodDisruptionBudget. Do PodDisruptionBudgets govern scheduling, eviction, or both — and given the answer, is the PDB the cause of the shortfall or a consequence of it?

Before continuing: the tier lost its first replica on Saturday and nobody declared an incident until Tuesday. Which property of this failure made it arrive slowly, and would the same change applied to the running nodes have been better or worse?

Root cause

1. Effect is part of the taint, not a setting on it

A toleration matches a taint when the key matches, the operator and value match, and the effect matches. All three. dedicated=payments:NoSchedule and dedicated=payments:NoExecute are two different taints that happen to share two thirds of their identity.

Friday’s commit reads as a tightening — “evict strays rather than repel them” — and in the mental model that produced it, NoExecute is NoSchedule plus eviction. It is not a superset. It is a replacement, and every toleration in the cluster that spelled out NoSchedule stopped matching the pool the moment the first node came up under the new template.

payments-api tolerates dedicated=payments:NoSchedule. pay-05 and pay-06 carry dedicated=payments:NoExecute. The Deployment is not tolerated on two of the six nodes it is allowed to use, and its nodeSelector forbids the other eighteen.

2. Registering with the taint hid the eviction

The pool template applies the taint through the node’s kubelet configuration — registerWithTaints in the KubeletConfiguration file, or --register-with-taints on older node images — so the taint is present from the moment the node joins.

A NoExecute taint evicts Pods that do not tolerate it. On a node that registers with the taint already applied there is nothing to evict, because nothing has been placed yet. So the single most visible consequence of NoExecute — an eviction storm, with Evicted Pods and a wave of events — never happened. The change looked inert on the day it landed, which is precisely why it stayed unassociated with what followed.

3. The rebuild metered the damage out over days

The pool is being replaced one node at a time. The old nodes still carry NoSchedule, still tolerate the workload, and are still serving. Each node converted removes one node’s worth of placement from a tier whose nodeSelector gives it nowhere else to go.

The result is a shortfall that grows by one node’s worth of replicas every time the rebuild advances, with no single moment that looks like an incident. A tier that drops from twelve to seven over three days does not trip anything that a tier dropping from twelve to zero in one minute would have tripped.

4. The two DaemonSets are the control experiment

The payments team’s fluent-bit names the effect and has stopped landing on the two new nodes. The platform’s node-exporter carries operator: Exists with no key, which matches every taint that exists, and lands everywhere.

That difference is the confirmation of the diagnosis, and it is also the reason the incident was so hard to see: node metrics from pay-05 and pay-06 are present, normal, and describe two machines that are idle because nothing can be scheduled on them. The capacity dashboard reporting 34 percent CPU is not wrong. It is measuring nodes that the workload is not allowed to use.

Resolution

  1. Pause the pool rebuild. Name an owner and an end time — the pause holds until the template decision is made or until 17:00, whichever comes first. This is free, reversible, and it stops the shortfall growing while the rest of the work happens.
  2. Take the two rebuilt nodes back by hand. Remove the NoExecute taint and add the NoSchedule one. Neither command can evict anything: removing a taint never evicts, and NoSchedule has no effect on running Pods. Placement resumes within a scheduling cycle.
  3. Watch the tier recover to twelve and the fluent-bit DaemonSet reach 6 of 6. If either does not, the taint was not the whole story and the FailedScheduling arithmetic will say so.
  4. Now decide which side of the mismatch is wrong, and record the decision. If NoExecute was an accident of wording in review, revert the template. If NoExecute is genuinely wanted, the workloads change instead — and every workload that targets the pool has to be enumerated first, not just the one that paged.
  5. If the workloads change, weigh the widening carefully. An Exists operator on the key with no effect tolerates every effect for that key, which fixes this and costs you the ability to use NoExecute on the pool deliberately for maintenance. Never widen to a key-less Exists.
  6. Rebuild one node from the amended template and confirm a payments Pod lands on it before restarting the rolling rebuild. Two nodes patched by hand say nothing about the template.
  7. Add the divergence check: a scheduled comparison of every node actual taints against its pool declared taints, alerting on any difference. Run it once against a deliberately divergent node so you know it can fail.
  8. Leave the PodDisruptionBudget exactly as it is, and write down in the incident record that it was proposed for deletion and why that would have been wrong.

Verification

  1. The FailedScheduling arithmetic is clean. No Pending payments Pod reports an untolerated-taint count, and any remaining counts map to nodes that are legitimately unavailable — outside the pool, or genuinely full. Checking only that Pods are Running misses a partial repair on a pool this size.
  2. The tier is at twelve Ready and stays there through a full rebuild cycle, not just through the two hand-patched nodes.
  3. The team DaemonSet reports desired equal to current across all six pool nodes. The DaemonSet is the check that covers nodes the Deployment happened not to select, so it fails earlier and more clearly than replica count.
  4. The workload is still evictable. Drain a rebuilt node and confirm its Pods actually move. A toleration widened far enough to make the symptom go away will also swallow the maintenance lever, and the next planned drain is a bad place to find that out.
  5. The PodDisruptionBudget still refuses when it should. At full replicas a drain is permitted; with the tier deliberately scaled short, the same drain is refused.
  6. A node built from the amended template accepts a payments Pod unattended. This is the only step that tests the template rather than the two nodes repaired by hand, and it is the step most likely to be skipped once the graphs look right.
  7. The taint-divergence check fires on a divergent node. Apply the old effect to a scratch node carrying the pool label and require an alert. A check that has only ever passed is untested.

Prevention

  • Effect is part of a taint’s identity. Changing it replaces the taint rather than strengthening it, and it will read in review as a tightening unless the diff says otherwise. Require the review comment to name the tolerations affected.
  • Node-pool templates are code with a delayed rollout. The change lands in git on Friday and reaches the cluster over the following week, one node at a time. Diff each node’s actual taints against its pool’s declared taints on a schedule; that check would have fired on Saturday.
  • Gate the template, not the outcome. Bring up one node from the new template and assert that a canary Pod of every workload class targeting the pool schedules onto it. That is the cheapest test available and the only one that exercises what actually changed.
  • Alert on Pods Pending for more than five minutes, with the FailedScheduling reason attached. A slow bleed needs a page; it will never announce itself on a capacity graph, because the nodes it strands look idle.
  • Keep tolerations narrow but effect-complete, and never key-less. A key-less Exists matches not-ready and unreachable and will pin a Pod to a dead node.
  • One taint per node, with a clear key. A pool carrying several dedicated taints requires a workload to tolerate all of them, and the matching becomes impossible to reason about from either side.
  • Never override a PodDisruptionBudget to unblock a drain. The refusal is a statement about the workload, and the right response is to find out what the statement means.