Resource-driven failures — insufficient CPU, memory, and storage

The most common scheduling failure is a Pod whose requests exceed the node’s available capacity. The scheduler’s NodeResourcesFit plugin (formerly a family of separate plugins) handles the filter and the scoring. This lesson walks the filter logic, the four scoring modes, and the operational patterns that keep a cluster from running out of capacity.

The filter logic

NodeResourcesFit runs in the filter stage. For each node, the plugin computes the node’s available capacity by subtracting the requests of every existing Pod from the node’s allocatable capacity. The Pod under scheduling is feasible on the node if its requests fit within the available capacity.

flowchart LR
    A[Pod under scheduling] --> B[NodeResourcesFit]
    B --> C[Compute node's<br/>available CPU]
    B --> D[Compute node's<br/>available memory]
    B --> E[Compute node's<br/>available ephemeral storage]
    C --> F{All fit?}
    D --> F
    E --> F
    F -->|Yes| G[Node is feasible]
    F -->|No| H[Reject with<br/>Insufficient resource]

The plugin does not consider limits. The filter is purely about requests. The Pod’s spec.containers[].resources.limits are used during runtime enforcement (cgroups) and during eviction; the scheduler never looks at them.

The “available” computation is a sum across the node’s existing Pods. The scheduler reads the node’s Status.Allocatable, which is the node’s reported capacity minus the kubelet’s reserved resources (the kubelet reserves CPU and memory for the system and for the eviction threshold).

# Substitute your own value before running:
NODE=worker-01

kubectl get node "$NODE" -o jsonpath='{.status.allocatable}' | jq

{
  "cpu": "15800m",
  "memory": "63154880Ki",
  "ephemeral-storage": "47113644477",
  "pods": "110"
}

# Substitute your own value before running:
NODE=worker-01

kubectl describe node "$NODE" | grep -A 5 "Allocated resources"

Allocated resources:
  Resource           Requests      Limits
  --------           --------      ------
  cpu                9250m (58%)  16000m (101%)
  memory             38Gi (62%)   60Gi (99%)
  ephemeral-storage  0 (0%)       0 (0%)

The “Requests” column is what the scheduler sees. The “Limits” column is what the kubelet enforces. A node that shows 100% of its limits is not overcommitted; the limits are the runtime ceiling, and the requests are the scheduling ceiling.

The four scoring modes

The NodeResourcesFit plugin is also a scoring plugin. The scoring mode is configurable by the cluster operator:

The scoring mode is set via the scheduler’s profile configuration. The default is LeastAllocated, which is the right choice for most general-purpose clusters. The MostAllocated mode is used by clusters that want to aggressively bin-pack to reduce the node count.

Reading the failed-scheduling event

A resource-driven failure looks like:

0/5 nodes are available: 2 Insufficient memory, 3 Insufficient cpu.

The Insufficient memory and Insufficient cpu strings are the format produced by the NodeResourcesFit filter. The numbers are the count of nodes that failed for each reason. A single node can fail for multiple reasons; the message counts the dominant reasons.

To identify which node failed which resource, the operator must cross-reference the message with the node’s capacity:

kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory

NAME      CPU      MEMORY
node-1    16000m   63154880Ki
node-2    16000m   63154880Ki
node-3    8000m    31577440Ki
node-4    16000m   63154880Ki
node-5    16000m   63154880Ki

A Pod with requests.memory: 64Gi is feasible on node-1, node-2, node-4, and node-5 (each has 64Gi+), but not on node-3 (which has only 32Gi). The message “2 Insufficient memory” would mean the Pod has more than 64Gi of memory request, and only the 32Gi node and one other node fail.

Distinguishing requests from usage

The filter compares requests, not current usage. A node whose existing Pods are requesting 50Gi of memory but using only 20Gi is “50Gi allocated” from the scheduler’s perspective. The scheduler overcommits; the kubelet enforces.

This has two operational consequences:

• A Pod with a small request can land on a busy node. The filter does not care that the node is at 90% of its memory usage; it cares only that the node has the request-shaped capacity. The Pod is scheduled and runs. If the existing Pods grow, the new Pod is evicted.
• A Pod with a large request that actually uses little is over-committed. A request of 16Gi for a workload that uses 1Gi is a 16:1 overcommit. The cluster can show plenty of capacity in kubectl describe node but fail to schedule if the freed Pods were using 1Gi and the new Pod is requesting 16Gi.

The fix is to size the requests to the actual usage. A workload that uses 1Gi but requests 16Gi is reserving 15Gi of capacity that no other Pod can use. Production clusters should monitor request-to-usage ratio and right-size requests.

The four diagnostic moves

1. Read the message. Insufficient memory and Insufficient cpu are the dominant failure modes; Insufficient ephemeral-storage is rare and usually indicates a workload writing too much to the container’s filesystem.
2. Identify the dominant node. The message lists the count, not the node names. Use kubectl describe node to find the node that has the resource.
3. Inspect the Pod’s requests. Many clusters fail because a Pod’s requests are not set, and the scheduler assigns the default (the LimitRange default or the cluster’s default). Set the requests explicitly.
4. Inspect the node’s other Pods. A node that is failing to schedule new Pods but has plenty of capacity for existing Pods is a node that has a runaway workload or a workload that has not been reclaimed after a previous eviction.

The five fix patterns

1. Scale the cluster. Add a node. The new node has fresh capacity; the scheduler can place the Pod there.
2. Reduce the Pod’s requests. The Pod’s actual usage is often smaller than the request. Right-size the request based on the workload’s measured P95.
3. Remove a Pod from the node. The Pod that is consuming the resource must be evicted or rescheduled elsewhere. The fix is rarely “delete the Pod”; it is “address the workload that is mis-using the node.”
4. Set a LimitRange with a sane default. A namespace without a LimitRange allows the Pod to be created with no requests, leading to an opaque “fits” or “doesn’t fit” decision. A LimitRange with explicit defaults is more predictable.
5. Set a ResourceQuota with sufficient headroom. A namespace that is at quota is failing to schedule new Pods because the existing Pods have consumed the quota. The fix is to raise the quota or to free capacity.

Quiz

Production discipline

• Requests are the scheduler’s currency. The scheduler does not see limits. A Pod that uses 1Gi but requests 16Gi is reserving 16Gi of capacity that the cluster cannot use. Audit request-to-usage ratios at every release.
• LeastAllocated is the production default. The default scoring mode favours nodes with the most free capacity. MostAllocated is appropriate only for clusters that want aggressive bin-packing and have a controller like the cluster-autoscaler to drain underused nodes.
• A cluster that is permanently “Insufficient” is a cluster that needs capacity. The cluster-autoscaler is the production answer. The operator should set the autoscaler to react to the scheduler’s failure pattern.
• Ephemeral-storage is the silent capacity leak. A workload that writes to /tmp can fill the node’s filesystem. The Insufficient ephemeral-storage message is rare but the failure is real. Monitor node_stats_fs_* and set requests.ephemeral-storage on Pods that write to the container filesystem.
• Override the scheduler’s filter with a NodeResourceFit-score extension. A custom scoring plugin can prefer nodes with the right CPU/memory shape for the workload. The default scoring is a heuristic and may not match the workload’s actual preference.