tolerationSeconds and graceful eviction windows

tolerationSeconds is the field that controls how long a Pod tolerates a NoExecute taint before the kubelet evicts it. A Pod without tolerationSeconds tolerates the taint forever (or until the taint is removed). A Pod with tolerationSeconds: 30 tolerates the taint for 30 seconds, then is evicted. This lesson walks the precedence rules and the operational patterns.

The field semantics

tolerationSeconds is set on the toleration, not on the taint (in API version v1; the taint can also carry a default; the Pod’s value always wins if both are present).

tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 30

The semantics: the clock starts when the kubelet observes the taint on the node. After tolerationSeconds seconds, if the taint is still present, the Pod is evicted. The eviction is graceful — the Pod’s containers receive SIGTERM, the kubelet waits for terminationGracePeriodSeconds (default 30s), then sends SIGKILL.

PreferNoSchedule and NoSchedule taints do not have tolerationSeconds. The field is meaningless for scheduling decisions because the Pod is not admitted to the node on those effects.

Precedence

When a taint and a toleration both specify tolerationSeconds, the more permissive value wins. The Pod can request a shorter window than the taint offers; the Pod can request a longer window only if the taint did not specify one.

flowchart TD
    A[Toleration on Pod] --> B{Pod sets<br/>tolerationSeconds?}
    B -->|Yes| C[Use Pod's value]
    B -->|No| D{Taint specifies<br/>tolerationSeconds?}
    D -->|Yes| C2[Use taint's value]
    D -->|No| E[Use 300s default<br/>for built-in taints]

The cluster defaults for the built-in not-ready and unreachable taints are 300 seconds. The node.kubernetes.io/unschedulable taint (set by kubectl cordon) defaults to 0 in older versions; in 1.34 the default is 300s.

Where the kubelet uses the value

The kubelet’s eviction loop runs the following check for each Pod on the node:

For each Pod:
  for each NoExecute taint on the node:
    if the Pod's tolerations match the taint:
      compute remaining_time = tolerationSeconds - time_since_taint_added
      if remaining_time <= 0:
        evict the Pod

The loop runs every 1 second by default (--node-monitor-period). The total time from “taint added” to “Pod SIGTERM” is approximately tolerationSeconds plus the next monitor tick.

Operational patterns

Pattern 1: spot-instance interruption

Spot-instance nodes can be reclaimed at any time. The cloud provider’s interruption handler (or, on AWS, the node-termination-handler DaemonSet) adds a taint like cloud=spot:NoExecute with a short tolerationSeconds to give the Pods a chance to drain before the node is reclaimed.

tolerations:
  - key: cloud
    operator: Equal
    value: spot
    effect: NoExecute
    tolerationSeconds: 30

The grace period is set to be shorter than the cloud provider’s reclamation window (typically 30s on AWS, 30s on GCP, 5min on Azure). The kubelet evicts the Pods before the node is taken; the controller-manager replaces them on a non-spot node.

Pattern 2: node maintenance with staged eviction

A node upgrade benefits from a two-phase eviction:

# Phase 1: stop new Pods
kubectl cordon node-1

# Phase 2: force the Pods off with a grace period
kubectl taint nodes node-1 maintenance=true:NoExecute

If the Pods do not tolerate the maintenance taint, the kubelet evicts them with the default 300s grace. If the Pods do tolerate it with tolerationSeconds: 60, the kubelet evicts them after 60s. Choose the value based on the workload’s graceful-shutdown time: max(tolerationSeconds, terminationGracePeriodSeconds).

Pattern 3: batch jobs that should ride out transient failures

A batch job (a Spark driver, a CI runner) might tolerate unreachable for 10 minutes:

tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 600

This rides out transient network partitions better than the 300s default. The tradeoff is that the cluster tolerates a slow-fail node for longer; the operator can detect a slow fail with kubectl get nodes and manual remediation.

Diagnosing the eviction delay

A Pod that has not been evicted yet but is on a node with a NoExecute taint:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl describe pod "$POD" | grep -A 5 "Tolerations"

Tolerations:
  node.kubernetes.io/not-ready:NoExecute for 300s
  node.kubernetes.io/unreachable:NoExecute for 300s

The remaining time is not shown directly; the operator must compute it from the Pod’s last transition time and the taint’s application time:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p
NODE=$(kubectl get pod "$POD" -o jsonpath='{.spec.nodeName}')

kubectl get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="Ready")].lastTransitionTime}'
kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].lastTransitionTime}'

A difference of more than tolerationSeconds between the two timestamps means the Pod should already be evicted. If the Pod is still present, the kubelet is lagging or the eviction is being blocked by another mechanism (a Pod finalizer, a stuck volume unmount).

Quiz

Production discipline

• Default tolerationSeconds is 300. Built-in taints default to 300s; do not extend this on latency-sensitive workloads. The right way to make a workload more responsive is to shorten tolerationSeconds on the Pod’s toleration, not to lengthen the cluster default.
• Use tolerationSeconds for expected events. A spot-instance handler can predict the interruption; the taint is then set with a tight window. An unscheduled restart should not be held up by a 300s window.
• Audit tolerationSeconds at every pod-template change. A Helm upgrade that adds tolerationSeconds: 0 to a stateful workload is a data-loss risk; a tolerationSeconds: 86400 on a stateless workload is a capacity leak.
• Watch the eviction monitor period. The kubelet’s node-monitor-period (default 5s) and the eviction-pressure-transition-period (default 5min) shape the actual eviction latency. The tolerationSeconds is the operator’s lever; the monitor period is the cluster’s internal latency.