Pod stuck Pending due to CPU requests

Reported symptoms

The orders-api release goes out at 10:15. Twenty minutes later the release engineer notices the rollout has not moved.

$ kubectl get deploy -n shop-prod orders-api

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
orders-api   9/12    3            9           214d

Nothing is on fire. The nine surviving replicas are the previous version and they are serving normally, so no alert has fired and no customer has noticed. The Deployment is simply stuck at three quarters of its intended size.

The first hypothesis is that the new image is broken - a crash on startup, a missing config key. It is a reasonable guess and it is wrong: the three new Pods have never started, because they have never been placed on a node.

The second hypothesis is that the cluster is full. That is also easy to dismiss, and dismissing it is the mistake that costs the next hour:

$ kubectl top nodes

NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-1   3100m        38%    9.1Gi           58%
worker-2   2740m        34%    8.4Gi           54%
worker-3   3620m        45%    10.2Gi          65%
worker-4   2480m        31%    7.9Gi           50%
worker-5   2910m        36%    8.8Gi           56%

An hour later a different team ships a new service into another namespace. It schedules in seconds. That settles it for the room: the cluster has capacity, so the problem must be orders-api itself.

The release diff is a version bump and a restructuring of values.yaml that moved several keys around. No CPU value, no memory value, and no replica count appears anywhere in it.

Evidence provided

The scheduler is not silent about this. It has been saying the same thing every few seconds for twenty minutes.

$ kubectl describe pod -n shop-prod orders-api-7d94f6b5c-k8xrt | tail -6

Events:
Type     Reason            Age                  From               Message
----     ------            ----                 ----               -------
Warning  FailedScheduling  22m (x31 over 22m)   default-scheduler  0/5 nodes are available: 5 Insufficient cpu.

Insufficient CPU, on a cluster whose busiest node is at 45 percent. The next command is the one that resolves the contradiction, and it is free to run.

$ kubectl get pod -n shop-prod orders-api-7d94f6b5c-k8xrt -o jsonpath='{.spec.containers[*].resources.requests.cpu}'

1 1 1

$ kubectl get pod -n shop-prod orders-api-6c8b249fd-mv4pq -o jsonpath='{.spec.containers[*].resources.requests.cpu}'

250m 50m 50m

Eight and a half times larger, in a release whose diff contains no resource change. So where did the number come from?

$ kubectl get deploy -n shop-prod orders-api -o jsonpath='{.spec.template.spec.containers[*].resources}'

{} {} {}

$ kubectl get limitrange -n shop-prod -o yaml | grep -A 8 'type: Container'

    - type: Container
    default:
      cpu: "2"
      memory: 1Gi
    defaultRequest:
      cpu: "1"
      memory: 256Mi

$ kubectl describe node worker-1 | grep -A 5 'Allocated resources'

Allocated resources:
Resource   Requests      Limits
--------   --------      ------
cpu        6100m (80%)   12400m (163%)
memory     9.4Gi (66%)   22Gi (154%)

Work the evidence before reading on

The five workers have 7600m of allocatable CPU each. Requests across the estate sit between 76 and 84 percent. Usage sits between 31 and 45 percent.

1. Which of those two figures does the scheduler use, and which does kubectl top report? Which one can a Pending Pod possibly be about?
2. Add up the free CPU across all five nodes. Now ask what the largest single free block on any one node is. Why does the first number not help a Pod that needs three CPU?
3. The Deployment declares no resources and the Pods request three CPU. Which component wrote that number into the Pod, and at what point in the request lifecycle?
4. The other team’s service scheduled fine an hour later. What does that tell you about the cluster, and what does it not tell you?

Before continuing: explain why exactly three Pods are Pending rather than twelve, and what would have happened if the rollout had been able to continue.

Root cause

1. The scheduler filters on requests; kubectl top reports usage

These are different numbers and nothing reconciles them.

NodeResourcesFit computes each node’s available capacity by subtracting the requests of every Pod already on it from the node’s allocatable, and asks whether the incoming Pod’s requests fit in what remains. It does not look at limits and it does not look at what any container is currently doing. A node running at four percent CPU whose Pods have requested all of it is full, as far as scheduling is concerned.

So kubectl top nodes is the wrong instrument for a Pending Pod. It was the first thing three people ran, it showed an idle cluster, and it sent the investigation towards the image.

2. The request changed without anyone changing a request

The values.yaml restructuring moved the resources block to a key the chart does not read. The chart rendered a Deployment whose containers carry no resources field at all - which is valid, and which passes every review that looks at the diff, because the diff shows a key moving rather than a value changing.

The namespace has a LimitRange. Its defaultRequest sets cpu: 1, and the LimitRange admission plugin applies defaults per container, writing them into the Pod spec as it is stored. A three-container Pod picks up three CPU of requests.

# What the chart rendered            # What the API server stored
resources: {}                        resources:
                                       requests:
                                         cpu: "1"
                                         memory: 256Mi
                                       limits:
                                         cpu: "2"
                                         memory: 1Gi

There is no event for this and no warning. kubectl get deploy -o yaml still shows nothing about CPU, because the mutation happens to the Pod, not to the Deployment template. The only place the number is visible is on an admitted Pod - which is why comparing a Pending Pod against a surviving old Pod is the decisive check and takes ten seconds.

3. Free capacity is not the same as a free block

The cluster has roughly nine CPU of unrequested capacity. It cannot place a three-CPU Pod.

Five nodes at 7600m allocatable, requested between 76 and 84 percent, leaves somewhere between 1200m and 1800m free on each. Scheduling is a per-node decision: the Pod needs three CPU on one node, and no node has it. The event says so precisely - 0/5 nodes are available: 5 Insufficient cpu - five nodes considered, all five rejected by the same filter.

This is also why the other team’s service scheduled without trouble. It requested 100m, which fits comfortably in the slack on any worker, and it lives in a namespace with a different LimitRange. Its success said nothing at all about whether a three-CPU Pod could be placed.

4. The rolling update decided how bad it got

Three Pods are Pending rather than twelve because the Deployment’s rolling update strategy caps how far ahead of itself the controller can run. With the default 25 percent surge and 25 percent unavailable on twelve replicas, it created three oversized Pods and retired three small ones, then waited for the new ones to become available before doing anything else. They never did.

The result is a Deployment that is stable, half-updated, serving from the old ReplicaSet, and quietly running at nine replicas instead of twelve until the progress deadline expires and marks it failed.

Resolution

1. Compare a Pending Pod against a surviving Pod from the previous ReplicaSet. Two jsonpath queries, no risk, and they establish the whole diagnosis - the same Deployment producing Pods eight times larger than the ones it produced yesterday.
2. Confirm where the number came from before changing anything: the Deployment template carries an empty resources object, and the namespace LimitRange carries a matching defaultRequest. The pairing is the proof; either fact alone is circumstantial.
3. Restore the explicit resources block in the chart values so the rendered manifest declares real requests, and verify the rendering locally before applying it.
4. If the rollout must complete before the chart change can ship, apply an imperative override with kubectl set resources per container as a stated hold. Write down that the next Helm release silently reverts it, name an owner, and set an expiry in hours.
5. Do not delete or relax the LimitRange to unblock the rollout. It is doing its job - defaulting a container that declared nothing - and removing it converts the Pods to BestEffort rather than fixing their sizing.
6. Before rolling forward, check that the honest requests fit. Twelve replicas at 350m places very differently from twelve at three CPU, and the largest free block per node is the number that decides it.
7. Roll forward and watch the scheduler place the Pods within a cycle or two of the change. If they stay Pending, the requests are still wrong and the event will say so.
8. Once the Deployment is whole, re-read the allocated-requests figure on each node. It should fall by roughly eight CPU across the estate, which is the amount that was never really needed.

Verification

1. Read the requests back from an admitted Pod, not from the chart or the Deployment. The chart is what you wrote; the Pod is what the API server stored after admission, and the gap between them is the entire incident.
2. kubectl rollout status returns success rather than the progress-deadline error, and kubectl get deploy shows 12/12 ready and 12 up to date.
3. No Pod in the namespace is left Pending. Check by phase across the namespace rather than by looking at the one Deployment, since an oversized request can strand a Job or a CronJob that nobody is watching.
4. The LimitRange defaulted nothing on the new Pods. If a container still shows exactly the default value, that container is still missing its declaration and will break again on the next node that fills up.
5. Node allocated requests have fallen to a figure consistent with what is running, and the gap between requested and used CPU is defensible rather than a factor of two.
6. Prove the new CI check fails. Submit a rendered container with the resources block removed and require the check to reject it. A guard that has only ever passed has not been tested.
7. Do not accept the absence of FailedScheduling events as evidence on its own; they expire after about an hour. Verify against PodScheduled in status.conditions, which persists.

Prevention

• Require explicit CPU and memory requests on every container, and enforce it in CI against the rendered manifest. This defect did not exist in values.yaml and did not exist in the chart; it existed only after templating, which is the only place worth checking.
• Treat a LimitRange defaultRequest as a floor for workloads someone forgot, never as a sizing mechanism. Remember that it applies per container, so a one-CPU default costs three CPU in a three-container Pod, and sidecars are the containers least likely to need it.
• Diff what the API server will store, not what you wrote. A server-side dry-run apply runs the admission chain and shows mutations like this one before the release goes out.
• Alert on Pods in Pending with reason Unschedulable. This incident never degraded the service and never paged anyone, while quietly holding the Deployment at three quarters of its intended capacity - which is exactly the state you want to discover before the next traffic peak rather than during it.
• Publish cluster capacity as requested CPU alongside used CPU. A dashboard built on usage reports an idle cluster that cannot schedule anything, and everyone will believe the dashboard.
• Record the largest free block on any single node next to aggregate free capacity. Scheduling is a per-node decision; the difference between those two numbers is fragmentation, and it is what turns nine free CPU into nowhere to put a three-CPU Pod.