Deployment rollout stuck

Reported symptoms

At 09:10 a release engineer bumps the checkout image from 2.7.4 to 2.7.5. It is the smallest change in the queue: no config, no schema, no new dependency.

At 10:00 it is still deploying.

kubectl rollout status says 0 out of 20 new replicas have been updated. Zero. Not one Pod of the new version has been created in fifty minutes, and there is nothing wrong to look at:

$ kubectl get deployment checkout -n checkout-prod

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
checkout   20/20   0            20          214d

Twenty of twenty Ready. Twenty Available. Every dashboard is green, every synthetic check passes, and the customers are being served perfectly — by the old version, which is exactly as healthy as it was yesterday.

Two other things are odd in that namespace, and neither has been connected to this:

• The search-api HPA has been pinned at 12 replicas for two days with CPU above 90 per cent and maxReplicas: 30. The team assumed a metrics-server problem.
• An engineer could not start an ephemeral debug Pod there yesterday and assumed RBAC.

Somebody has tried kubectl rollout restart. It printed a success message and produced nothing at all.

The questions worth asking:

• What do you diagnose when the rollout has produced no Pod to diagnose?
• The CI pipeline said the deploy succeeded. What did it actually check?
• Which of the three problems in this namespace is a separate problem?

Evidence provided

$ kubectl get rs -n checkout-prod -l app=checkout

NAME                  DESIRED   CURRENT   READY   AGE
checkout-6d4c7b9f58   20        20        20      9d
checkout-7c9f4d6b85   5         0         0       51m

$ kubectl describe rs checkout-7c9f4d6b85 -n checkout-prod | sed -n '/Conditions/,$p'

Conditions:
Type             Status  Reason
----             ------  ------
ReplicaFailure   True    FailedCreate

Events:
Type     Reason        Age                    From                   Message
----     ------        ----                   ----                   -------
Warning  FailedCreate  4m11s (x22 over 51m)   replicaset-controller  Error creating: pods "checkout-7c9f4d6b85-" is forbidden: exceeded quota: compute, requested: requests.cpu=1600m, used: requests.cpu=32, limited: requests.cpu=32

$ kubectl describe resourcequota compute -n checkout-prod

Name:            compute
Namespace:       checkout-prod
Resource         Used    Hard
--------         ----    ----
limits.cpu       61      64
limits.memory    118Gi   128Gi
pods             186     200
requests.cpu     32      32
requests.memory  58Gi    64Gi

$ kubectl get deployment checkout -n checkout-prod -o jsonpath='{.spec.strategy.rollingUpdate}{.spec.template.spec.containers[0].resources.requests}'

{"maxSurge":"25%","maxUnavailable":0}{"cpu":"1600m","memory":"2Gi"}

$ kubectl describe deployment checkout -n checkout-prod | sed -n '/Conditions/,/Events/p'

Conditions:
Type             Status  Reason
----             ------  ------
Available        True    MinimumReplicasAvailable
Progressing      False   ProgressDeadlineExceeded
ReplicaFailure   True    FailedCreate

The namespace change log for the last ten days holds three entries, by three people, none of them related to each other:

1. Nine days ago, checkout CPU request raised from 1500m to 1600m to stop the container being throttled at peak.
2. Two days ago, checkout-worker scaled from 2 replicas back to 6, ending a migration.
3. This morning, checkout image 2.7.4 to 2.7.5.

Work the evidence before reading on

Nothing has failed. Every controller in this picture is doing precisely what it was told.

1. Multiply the replica count by the CPU request and compare it with the requests.cpu row of the quota. Then work out how much a single surge Pod needs.
2. maxUnavailable: 0 and a namespace with zero free CPU. In which order must the controller act, and what does that order require?
3. Three tickets in one namespace: a stuck rollout, a pinned HPA, and a debug Pod that would not start. How many causes is that?

Before continuing: the fastest fix is to raise the quota. What does that change and what does it not change?

Root cause

1. The quota is exhausted on exactly one row

Twenty Pods at 1600m is 32 cores of requests.cpu, and the quota permits 32. Not approximately — exactly. Every other row has room, which is why no utilisation alert fired: the namespace looks about 90 per cent used if you average the rows, and 100 per cent used only on the row that matters.

Nine days ago the same Deployment ran twenty Pods at 1500m, which is 30 cores, and there were two cores spare. The request bump consumed both. It fitted at the time because checkout-worker was scaled down to 2 during a migration; scaling it back to 6 two days ago took the rest.

Each of those changes was correct and each was made by somebody with no reason to look at the others.

2. maxUnavailable: 0 turns “no headroom” into “no rollout”

maxUnavailable: 0 is the safe setting, and it is chosen deliberately for a checkout tier: no old Pod is removed until a new Pod is Ready, so the service never runs below its replica count during a deployment.

Its price is that the rollout can only ever move by creating first. The controller sets the new ReplicaSet to the surge budget — 25 per cent of 20, which is 5 — and waits for those Pods to exist before it touches the old ReplicaSet.

The first of those Pods needs 1600m and the namespace has nothing. So the rollout does not proceed slowly. It does not proceed at all, and it will still not be proceeding tomorrow.

3. The rejection lands on an object nobody looks at

The Deployment was admitted without complaint. Quota counts Pods, and a Deployment is not a Pod, so kubectl apply succeeded and the CI pipeline recorded a successful deploy. This is worth stating plainly because it is the reason the pipeline is not a control here at all.

The ReplicaSet controller then tried to create Pods and was refused by the quota admission plugin. It records that as a FailedCreate event and a ReplicaFailure condition — on the ReplicaSet.

The habitual diagnostic path is describe pod, then events, then logs. All three need a Pod. There is no Pod, has never been a Pod, and will never be a Pod, and the path simply runs out of objects. A stuck rollout with a healthy looking Deployment sends you to the ReplicaSet, not to Pods that were never created.

4. The other two tickets are the same ticket

A quota is a namespace-wide budget, so everything that needs to create a Pod in checkout-prod is blocked: the rollout, the search-api HPA that has been trying to scale for two days, and the debug Pod that looked like an RBAC refusal. Three teams, three hypotheses, one row of one object.

Resolution

1. Say out loud that there is no outage. The old version is serving twenty of twenty; every option below is available at leisure, and that is the most useful fact in the room.
2. Merge the three tickets. The pinned HPA and the failed debug Pod are the same quota, and closing them separately guarantees this recurs.
3. Capture the ReplicaSet condition and event text before changing anything. It is the only record of what was refused and it disappears when the ReplicaSet is replaced.
4. Choose deliberately between the four responses and write down the cost of the one you pick: raise the quota, set maxUnavailable: 1, reclaim quota from elsewhere in the namespace, or hold until a window.
5. If you raise the quota, check free allocatable CPU on the eligible nodes first, and raise it to a derived number rather than a round one.
6. If you set maxUnavailable: 1, accept 19 of 20 replicas during the rollout and confirm with whoever owns the tier that this is acceptable at the current hour.
7. Apply the chosen change and watch the new ReplicaSet, not the Deployment. Desired, current and ready converging is the signal; the Deployment was reporting Available the whole time.
8. Leave headroom on purpose afterwards. Landing exactly on the new limit reproduces this incident on the next rollout.

Verification

1. The new ReplicaSet reports desired, current and ready equal, and no longer carries a ReplicaFailure condition.
2. kubectl rollout status exits cleanly instead of timing out, and the Deployment shows UP-TO-DATE equal to READY.
3. The Deployment Progressing condition is True with reason NewReplicaSetAvailable. Available was True throughout and verifies nothing.
4. The quota shows deliberate free space: the unused requests.cpu exceeds the surge budget of the largest Deployment in the namespace. This is the check that can fail, and it is the one that decides whether this happens again next week.
5. The search-api HPA has resumed scaling. This is the evidence that the three tickets shared a cause, and it costs one command.
6. An ephemeral debug Pod starts in the namespace.
7. If the quota was raised, the new Pods are Running rather than Pending, confirmed at the node level. A Pending Pod means the quota was not the binding constraint any more and the scheduler is.

Prevention

• Put rollout headroom in the quota and show the arithmetic. A namespace at 100 per cent quota is not full, it is frozen: the largest Deployment in it can never be updated again. The free requests.cpu has to exceed maxSurge multiplied by the per-Pod request, and that sentence belongs in a comment next to the quota.
• Alert on utilisation, not on rejection. A rejection surfaces hours later as somebody else’s stuck rollout. Utilisation above roughly 90 per cent on any single row is the signal, and note that the row matters — averaging the rows hid this one completely.
• Alert on ReplicaFailure on any ReplicaSet. One rule catches stuck rollouts, blocked HPAs and rejected Jobs, and it fires on the object that actually knows.
• Alert on Progressing being False, never on Available alone. The condition that was true about this incident flipped forty minutes before a human noticed.
• Make the pipeline wait. A deploy step that reports success when kubectl apply returns is reporting that the API server accepted some YAML. Wait on kubectl rollout status with a timeout and fail the step when it expires.
• Treat a resource request change as a quota change. A hundred millicores looks like rounding and becomes two cores across twenty replicas, taken from a budget shared with everything else in the namespace.