CrashLoopBackOff

Reported symptoms

The billing error-rate alert fires at 08:05. Six replicas, two of them in CrashLoopBackOff, four Running:

NAME                       READY   STATUS             RESTARTS        AGE
billing-6c9d84f7b5-2xk4m   1/1     Running            0               26d
billing-6c9d84f7b5-4tqzn   0/1     CrashLoopBackOff   9 (2m14s ago)   6h
billing-6c9d84f7b5-8hzpl   1/1     Running            0               26d
billing-6c9d84f7b5-j7wvc   1/1     Running            0               26d
billing-6c9d84f7b5-q2rmb   0/1     CrashLoopBackOff   9 (1m52s ago)   6h
billing-6c9d84f7b5-vd8kf   1/1     Running            0               26d

Everything about this list says the six Pods are interchangeable. They share a ReplicaSet, so they share a Pod template. They share an image digest. They share a node pool. Two of them are broken and four are not.

The change log offers nothing useful. kubectl rollout history deployment/billing -n prod shows revision 14, created 26 days ago. The application team has not merged to the release branch this week. The only entry in the platform change log for the last 48 hours is a node drained at 02:10 for a kernel patch - and the ages line up: the two failing Pods are six hours old, the four healthy ones are 26 days old.

At 08:20, before anyone had read the logs, an engineer deleted billing-6c9d84f7b5-2xk4m on the theory that a stuck Pod might clear. Its replacement went into CrashLoopBackOff within ninety seconds. The incident channel now believes the fault is spreading, and the natural next move - restarting the rest - is the worst available action.

Evidence provided

$ kubectl describe pod billing-6c9d84f7b5-4tqzn -n prod | sed -n '/Containers:/,/Conditions:/p'

Containers:
billing:
Image:          registry.example.com/billing@sha256:9b2f...c41a
State:          Waiting
Reason:       CrashLoopBackOff
Last State:     Terminated
Reason:       Error
Exit Code:    1
Started:      Tue, 11 Aug 2026 08:22:41 +0000
Finished:     Tue, 11 Aug 2026 08:22:41 +0000
Ready:          False
Restart Count:  9

$ kubectl get events -n prod --field-selector involvedObject.name=billing-6c9d84f7b5-4tqzn --sort-by=.lastTimestamp

LAST SEEN   TYPE      REASON      MESSAGE
6h          Normal    Scheduled   Successfully assigned prod/billing-6c9d84f7b5-4tqzn to node-04
6h          Normal    Pulled      Container image already present on machine
6h          Normal    Created     Created container billing
6h          Normal    Started     Started container billing
2m          Warning   BackOff     Back-off restarting failed container billing

$ kubectl logs billing-6c9d84f7b5-4tqzn -n prod -c billing --previous

2026-08-11T08:22:41Z INFO  billing 3.4.1 starting
2026-08-11T08:22:41Z INFO  loading configuration from /etc/billing/billing.yaml
2026-08-11T08:22:41Z FATAL config: /etc/billing/billing.yaml: unknown field retry_budget

$ kubectl exec -n prod billing-6c9d84f7b5-8hzpl -c billing -- grep -c retry_budget /etc/billing/billing.yaml

0
command terminated with exit code 1

$ kubectl get configmap billing-config -n prod -o jsonpath={.data.config} | grep -n retry_budget

14:  retry_budget: 0.25

$ kubectl get configmap billing-config -n prod --show-managed-fields -o yaml | grep -E 'manager:|time:'

    manager: kubectl-client-side-apply
time: "2026-07-21T14:32:08Z"

$ kubectl get deployment billing -n prod -o yaml | sed -n '/volumeMounts:/,/volumes:/p'

        volumeMounts:
- mountPath: /etc/billing/billing.yaml
name: config
subPath: config
volumes:
- configMap:
name: billing-config
name: config

Work the evidence before reading on

The application is telling you precisely what it objects to, so the interesting question is not what is wrong with the config. It is why four Pods do not care.

1. A healthy Pod’s config file does not contain retry_budget. The ConfigMap does. Both statements are true right now, and the healthy Pod mounts that ConfigMap. What can make a mounted file differ from the object it is mounted from?
2. The last write to the ConfigMap was 21 days ago. The oldest Pods are 26 days old and the failing ones are 6 hours old. Sort those three numbers and say which Pods were created before the write and which after.
3. Deleting a healthy Pod produced another crasher. If the fault were spreading between Pods, what would you expect a deletion to do? Is there any reading of this evidence in which the delete was the cause rather than the reveal?
4. rollout history shows nothing for 26 days. What kind of change to a workload does not appear in rollout history?

Before continuing: six Pods, one ConfigMap, one image. Name the event that broke the service and the date it happened - they are not the same date.

Root cause

1. The breaking change landed 21 days before the outage

Someone added retry_budget to billing-config on 21 July, ahead of a release that was going to use it. The application version running in production, 3.4.1, rejects unknown fields when it parses its configuration and exits non-zero.

That edit should have broken production immediately, and it did not, for a reason so ordinary it is easy to look straight past: editing a ConfigMap does not restart anything. The Deployment’s Pod template did not change, so there was no new ReplicaSet, no rollout, no revision, and no event. Six processes that had already read their configuration at startup continued to run on what they had read. A process does not re-read a file it is not watching, and this one is not watching.

So the cluster entered a state where the running configuration and the declared configuration disagreed, and stayed there for three weeks with no observable symptom of any kind.

2. The drain converted latent divergence into an outage

At 02:10 a node was drained for a kernel patch. Two billing Pods on that node were evicted, and the ReplicaSet created two replacements on other nodes.

A new Pod projects the ConfigMap as it exists at that moment. Those two Pods therefore started with retry_budget present, parsed it, rejected it, and exited - repeatedly, into the kubelet’s exponential backoff, which is what CrashLoopBackOff means.

The drain is not the cause. It is an entirely routine operation that would have been safe on any day before 21 July, and the same reveal would eventually have come from a node failure, an eviction under memory pressure, a cluster autoscaler scale-down, or a scale-up adding a seventh replica. The change log looks empty because the change that mattered was not the kind of change anybody logs.

3. subPath is why the healthy Pods’ files are stale too

This is the piece of evidence that makes the diagnosis land, and it deserves to be stated exactly.

The Deployment mounts a single ConfigMap key, config, with subPath, onto the filename /etc/billing/billing.yaml that the application insists on - which is the one genuinely good reason to reach for subPath. The kubelet does not update a subPath mount when its source ConfigMap changes. A directory mount is refreshed within the kubelet’s sync period; a subPath mount is resolved once and left alone.

That is why grep retry_budget inside a healthy Pod finds nothing while the same grep against the ConfigMap finds it on line 14. Without subPath, the file inside the healthy Pods would have been updated weeks ago and the process would still have been running happily on what it read at startup - the outage would have been identical and the disk evidence would have been gone. subPath did not cause this incident. It is the reason you can see it.

4. The delete did not spread anything

Deleting a healthy Pod created a new Pod. A new Pod reads the current ConfigMap. The current ConfigMap is the broken one. The engineer converted a working replica into a failing one, and the service went from four healthy replicas to three.

Under this failure mode every restart is a one-way door. That reframes the entire response: the four old Pods are not “the ones that have not failed yet”, they are the only remaining instances of a configuration that works, and they are irreplaceable until the ConfigMap is fixed.

Resolution

1. Freeze anything that can restart a Pod in this namespace: pause the HPA or note its current floor, suspend node maintenance, and tell the incident channel explicitly that kubectl delete pod and rollout restart are forbidden until the ConfigMap is fixed. This is the first action because the failure mode punishes exactly the reflexes people have.
2. Capture the working configuration while it still exists: kubectl exec -n prod <healthy-pod> -c billing -- cat /etc/billing/billing.yaml > /tmp/billing-running.yaml. If the last four Pods are lost before the ConfigMap is corrected, this file is the only record of what production was actually running.
3. Diff it against the object: kubectl get configmap billing-config -n prod -o jsonpath={.data.config} > /tmp/billing-declared.yaml and diff /tmp/billing-running.yaml /tmp/billing-declared.yaml. The diff is the incident in one screen, and it belongs in the write-up.
4. Decide between reverting the config and rolling the application forward. Reverting affects only the failing Pods and leaves the four survivors untouched; rolling forward to a version that accepts the field replaces all six and converts a partial outage into a full deployment under incident conditions. Revert unless the field is load-bearing for something already live.
5. Decide explicitly whether to hold. Running at four replicas with restarts frozen is a legitimate position if the release that needs the field ships within hours - but only with a named owner, a stated risk (any eviction costs a replica), and an end time. Record the decision; an unrecorded hold is just a delay with better manners.
6. Revert the field in the source repository first, not in the cluster. A kubectl edit fixes production and leaves the next GitOps sync or kubectl apply to put it straight back.
7. Apply the corrected ConfigMap and confirm the object no longer contains the field.
8. Recreate the failing Pods deliberately: kubectl delete pod on the two crashers only. They are already contributing nothing, so this is the one restart that costs you nothing, and it is what makes them project the corrected configuration.
9. Watch one replacement Pod through to Ready before touching the second, so that if the revert was wrong you learn it while three healthy Pods are still serving.
10. Only once the service is stable, fix the mechanism: add a checksum annotation or a configMapGenerator hash to the Pod template so config edits produce a rollout, and change the mount from subPath to a directory. Both are ordinary changes that deserve an ordinary review, not incident-time edits.

Verification

1. The failing Pods recovered. Both replacements reach 1/1 Running with Restart Count 0, and kubectl logs shows the startup line followed by the application serving rather than a fatal line.
2. A newly created Pod survives startup. Delete one of the old healthy Pods on purpose and watch its replacement reach Ready. Until a Pod that started from the current ConfigMap is healthy, nothing has been verified - and skipping this step is what leaves the same landmine armed.
3. The file matches the object. Read /etc/billing/billing.yaml inside a freshly created Pod and diff it against kubectl get configmap billing-config -o jsonpath output. They must be identical; before the fix they were not.
4. Restart counts stay flat for longer than the backoff. The kubelet backoff grows to five minutes, so watch for at least ten before calling it stable. A Pod quiet for sixty seconds has proven almost nothing.
5. The service is serving, not merely running. Error rate returns to its pre-incident baseline and the readiness endpoint succeeds from all six replicas. Ready is a claim by the probe; the error rate is the claim by the users.
6. The mechanism change works. After adding the checksum annotation, make a trivial config edit in a non-production environment and confirm it produces a new Deployment revision and a normal rolling update. If it does not, the annotation is decorative.
7. The check can fail. In the same non-production environment, apply a config the running version rejects and confirm the rollout stops with unavailable replicas rather than silently arming a future incident. This is the whole point of the change.

Prevention

• Make config changes roll out. A configMapGenerator hash suffix or a checksum annotation on the Pod template turns a ConfigMap edit into a Pod template change, which means a revision, a rolling update, a health gate and a rollout undo. Everything this incident lacked follows from that one change.
• Accept the cost of that honestly. Once config edits trigger rollouts they are no longer a hot-tuning knob, and a team that genuinely needs runtime tuning needs a reload path the application supports - a watched file, a SIGHUP handler, or a reloader sidecar - rather than an in-place edit nobody observes.
• Mount ConfigMaps as directories. subPath freezes the file at container creation and hides the divergence from anyone who looks. Use it only when the application demands an exact filename in a directory it also writes to, and write down why.
• Validate config against the binary in CI. A field the application rejects is catchable by running the application’s own config parser in the pipeline. This is cheap, deterministic, and would have failed the pull request in July.
• Consider immutable: true for release-baked config. It converts an in-place edit into an API error, which forces the change through the release path where it belongs.
• Restart on a schedule. A Deployment that has not restarted in 26 days is carrying an unknown quantity of latent divergence between what it is running and what is declared. A periodic rolling restart during working hours turns that into a routine, observed event instead of something a node failure discovers at 02:10.
• Alert on the shape, not just the symptom. Pods of one ReplicaSet in mixed states, with a large age gap between the healthy and the failing, is a specific and unusual signal. It says “new Pods cannot start” long before the error rate does.