Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

advancedworkload-identity~45 min

A Pod cannot get its secret, then can, then cannot again an hour later

Reported symptoms

  • ●At 09:02 UTC the migrated payments deployment is applied to the new cluster and every Pod sits in ContainerCreating, never starting a container
  • ●The identical manifest has been running unchanged on the old cluster for three years, which the migration team reads as proof that the manifest is not the problem
  • ●After a change at 09:11 the Pods start immediately, and the team records the incident as resolved before checking whether the application can authenticate
  • ●The application is running but its own readiness endpoint reports that it cannot reach its configuration store, and the secret manager logs a rejected login for that workload identity
  • ●A second change at 09:40 works: the Pods start at 09:44, authenticate, and serve customer traffic normally for an hour
  • ●At 10:44 UTC every Pod in the deployment simultaneously fails to authenticate to the secret manager, with no deployment, no restart and no configuration change in the intervening hour

Evidence

  • · Describing a stuck Pod shows the container waiting on a volume whose named Secret cannot be found by the API server
  • · Listing Secrets in the namespace returns no Secret of the service account token type at all; only the TLS and image pull Secrets exist
  • · Reading the ServiceAccount object shows it has no secrets list, so nothing was ever bound to it on this cluster
  • · The committed Deployment manifest mounts a Secret volume by name and reads the token from it, a pattern that only ever worked because older clusters generated that Secret automatically
  • · The new cluster reports a server version of 1.37; the source cluster was built on a release that predates the change to token generation
  • · After the first repair, decoding the payload of the mounted token shows an audience array containing only the API server identifier, and not the audience the secret manager is configured to require
  • · The secret manager rejects that login on audience grounds while a token minted by hand with the correct audience authenticates successfully from a debug Pod at the same moment
  • · After the second repair, listing the token file inside a running container at 10:50 shows a modification time of 10:32, later than the container start time of 09:44, proving the kubelet replaced the file and the process did not read it again
Diagnosis and resolutionclick to reveal

Root cause

Three failures, one underlying cause: a manifest written for a credential mechanism that no longer exists, repaired twice under time pressure by people reasoning from the previous mechanism. Auto-generated ServiceAccount token Secrets stopped being created in v1.24 and the gate that controlled that behaviour was removed in v1.27, so on a current cluster there is nothing to mount and the Pod never starts. The first repair pointed the workload at the token the admission controller projects into every Pod. That mount succeeds, so the Pods start, and the incident looks closed. But a projected token carries an audience, and by default that audience is the API server. The secret manager is a different relying party with its own audience, and it correctly refused a token that was not minted for it. The second repair declared a projected token volume with the right audience and an explicit one hour lifetime, and it was correct. What it did not change is the application, which reads the token file once during start-up and holds the string for the life of the process. The kubelet requests a replacement once the token passes eighty per cent of its lifetime, or twenty four hours, whichever comes first, and rewrites the file in place. Reloading is the application''s responsibility and nobody had ever needed to implement it, because the credential on the old cluster was a static Secret that never expired. That is the defect that made this an outage rather than a migration hiccup: it recurs on the hour, forever, and each recurrence looks like a brand new fault.

Remediation

Do not create a token Secret by hand to make the mount work. It is the fastest possible fix, it is still supported, and it reintroduces exactly the credential the platform spent three releases removing: a token with no expiry, bound to nothing, that cannot be revoked because there is no revocation API for service account tokens at all. Equally, do not relax the audience the secret manager requires. An audience check is the only thing preventing a token minted for the API server from being replayed against every other relying party that trusts the same issuer, and switching it off converts one workload identity into a cluster-wide one. Work in this order. First, declare a projected service account token volume on the Pod with the audience the secret manager expects and an explicit expiry rather than an inherited default, mounted at a path that is clearly not the API server credential. Second, delete the legacy Secret volume, the volume mount and the manifest comment that explains it, so nobody restores the old pattern next quarter. Third, and this is the step that actually ends the incident, change the application to read the token file at every authentication attempt rather than caching the string. Reading a small file from a tmpfs mount on each login is cheap, and it removes an entire class of failure. If the application cannot be changed today, a scheduled reload every five minutes is enough to stay ahead of rotation and is explicitly good enough per the upstream guidance. Roll the deployment so every Pod picks up the new volume, and remember that deleting a Pod is also how a bound token is invalidated.

Verification

A successful start proves almost nothing here, because the workload started successfully twice while still broken. Verify from the relying party rather than from the cluster. Read the secret manager''s audit records and confirm a successful login attributed to this workload identity, then confirm there are no rejected logins for it after the roll. Independently, decode the payload of the token the Pod is actually mounting and confirm the audience array contains the secret manager and the expiry is the value you declared rather than an inherited one, which distinguishes a token you configured from a token you inherited. Then wait. The only verification that matters here runs on a clock: leave the workload serving for at least ninety minutes and confirm it is still authenticating after the token file has been replaced at least once. Check that by listing the token file inside a running container and observing a modification time later than the container start time, while the application continues to succeed. That combination, a rotated file and an unbroken service, is the only evidence that the reload path works. Finally, delete one Pod deliberately and confirm the replacement authenticates on its own, because a workload that only works when a human is watching a rollout is not fixed.

Prevention

Declare token lifetimes rather than inheriting them. Set the expiry explicitly on every projected token, and understand that the API server extends admission-injected tokens by default, up to a year, for compatibility with workloads exactly like this one. That default is why the old cluster appeared to work forever, and it is why the phrase it worked before is a signal to check a flag rather than evidence of a regression. Pin the audience for every relying party and never let a third party accept the API server audience. Fail the pipeline on any manifest that references a Secret of the service account token type, which is a one-line lint rule and would have caught this before the migration window opened. Alert on any Pod in ContainerCreating for more than five minutes, and alert on rejected logins at the secret manager broken down by workload identity, warning at three failures in five minutes so a rotation failure pages before a user notices. Soak every migrated workload in staging for at least two hours before promotion, so it crosses at least two token rotations rather than none; a thirty minute smoke test cannot observe an hourly failure. Finally, write down that service account tokens cannot be revoked, so the response to a suspected leak is to delete the Pod, and any credential design that assumes revocation needs a different mechanism.

Reported symptoms

The payments-api workload is moving from a cluster built three years ago onto a freshly installed one. The manifests were copied across unchanged, which is the whole point of the exercise. The application authenticates to the estate secret manager using its Kubernetes workload identity and reads its database configuration from there.

The migration window opened at 09:00 UTC. What follows is one incident that presented as three.

  • 09:02. The Deployment is applied. Every Pod sits in ContainerCreating and never starts a container. The manifest has run unchanged for three years, so the manifest is assumed innocent and the new cluster is assumed guilty.
  • 09:11. Somebody points the workload at the token that every Pod already has, at the standard projected path. The Pods start within seconds. The channel records the issue as resolved.
  • 09:14. The application is running and its readiness endpoint says it cannot reach its configuration store. The secret manager is logging a rejected login for this workload identity, once per retry.
  • 09:40. A second change declares a proper projected token with the audience the secret manager requires. Pods start at 09:44, authenticate, and serve customer traffic. The window closes.
  • 10:44. Every Pod in the deployment fails to authenticate, at the same second, with no deployment, no restart and no change in the intervening hour.

The last one is the interesting one. Nothing happened at 10:44. The thing that happened was at 09:44, exactly sixty minutes earlier.

Evidence provided

Read-only / Safe09:04 UTC, on the new cluster
$ kubectl -n payments describe pod -l app=payments-api
Read-only / Safethe same namespace, measured the other way
$ kubectl -n payments get secret
Read-only / Safethe negative result that settles the first failure
$ kubectl -n payments get serviceaccount payments-api -o yaml

The manifest that has run unchanged for three years is this:

      volumes:
        - name: sa-token
          secret:
            secretName: payments-api-token
      containers:
        - name: api
          volumeMounts:
            - name: sa-token
              mountPath: /var/run/secrets/payments
              readOnly: true

That manifest never created anything. It consumed a Secret that older clusters produced automatically for every ServiceAccount. On this cluster nothing produces it, so the mount has nothing to mount.

Read-only / Safe09:20 UTC: why a running Pod still cannot log in
$ TOKEN=$(kubectl -n payments exec deploy/payments-api -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
printf '%s' "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq '.aud, .exp'
Configuration changethe control experiment that isolates the audience
$ kubectl -n payments create token payments-api --audience openbao --duration 10m

The corrected volume, applied at 09:40:

      volumes:
        - name: bao-token
          projected:
            sources:
              - serviceAccountToken:
                  path: bao-token
                  audience: openbao
                  expirationSeconds: 3600
Read-only / Safe10:50 UTC: the file moved and the application did not
$ kubectl -n payments exec deploy/payments-api -- ls -l --time-style=full-iso /var/run/secrets/bao/bao-token

Work the evidence before reading on

Two of the three failures were introduced by the repair for the previous one. That is worth sitting with before reading the answer.

  1. The namespace contains no Secret of the service account token type and none was deleted. What changed between the two cluster generations, and why did nothing in the migration plan catch it?
  2. A Pod that starts successfully and a Pod that can authenticate are different claims. Which of the two did the 09:11 change actually deliver, and what would have exposed the difference in one command?
  3. A token minted by hand with a different audience works. Given that, which components can you exonerate, and what single field is the entire second failure?
  4. Nothing happened at 10:44. The token file changed at 10:32 and the container started at 09:44. Reconstruct the arithmetic, and say whose responsibility the missing step is.

Before continuing: name the check that would have failed at 09:45, one minute after everyone agreed the migration was finished.

Root cause

The credential the manifest expects has not been generated for several releases

Older clusters created a long-lived token Secret for every ServiceAccount automatically. That stopped in v1.24, and the feature gate that controlled it was removed in v1.27, so on this 1.37 cluster the behaviour is not merely off, it is gone. The workload identity mechanism is now the TokenRequest API: the admission controller adds a projected volume, the kubelet fetches a time-bound token for the specific Pod, and refreshes it before it expires.

A manifest written against the old mechanism does not fail loudly at apply time. It fails at mount time, which is why the Pod is stuck in ContainerCreating rather than crash-looping with a useful message.

The projected token is minted for one audience, and the secret manager is a different one

Every Pod already carries a token at the standard path. Its audience defaults to the API server, because that is what it is for. The secret manager is a separate relying party which validates the audience claim and refuses a token that was not minted for it.

That refusal is the system working. An audience-bound token that any relying party would accept is a bearer token for the entire estate: present the API server token to the secret manager, to the message broker, to anything else federated to the same issuer. The audience is what stops one identity from becoming all of them.

The application caches a credential the platform is actively replacing

The corrected volume asks for a one hour token. The kubelet requests a replacement once the token is past eighty per cent of its lifetime, or older than twenty four hours, and rewrites the file in place. Reloading it is explicitly the application’s job.

The application reads the file once, in its start-up path, and keeps the string. So the timeline is exact: container start at 09:44, file replaced at 10:32, cached token expires at 10:44, every Pod fails together because every Pod started together.

Resolution

  1. Rule out the two shortcuts explicitly before anyone proposes them. Do not create a token Secret by hand: it is a static credential with no expiry and no revocation path. Do not relax the audience requirement at the secret manager: that check is what stops a token minted for the API server being replayed against every other relying party trusting the same issuer.
  2. Declare a projected serviceAccountToken volume on the Pod with the audience the secret manager expects and an explicit expirationSeconds, mounted at a path clearly distinct from the API server credential. Inheriting the default expiry is what makes a workload behave differently on two clusters for reasons nobody can see in the manifest.
  3. Delete the legacy Secret volume, its volume mount, and any comment in the manifest that explains the old pattern. A commented-out block is an invitation to restore it during the next incident.
  4. Change the application to read the token file at each authentication attempt rather than caching the string at start-up. The file lives on a tmpfs mount, so re-reading it costs nothing measurable, and this is the step that actually ends the incident.
  5. If the application cannot be changed in this window, reload the file on a schedule instead. Every five minutes is comfortably ahead of a rotation that happens at eighty per cent of an hour, and the upstream guidance says plainly that a fixed schedule is usually good enough.
  6. Roll the deployment so every Pod picks up the new volume, and stagger the roll rather than replacing every Pod in the same minute. Pods that start together rotate together and therefore fail together, which is why 10:44 looked like an event.
  7. Record which credential mechanism this workload used on the source cluster and which it uses now, in the migration checklist, and repeat that question for every remaining workload before its window opens.
  8. Leave the incident open until the ninety minute soak in the verification step has passed. Closing it at 09:45 is exactly what happened the first time.

Verification

  1. Read the secret manager audit records and confirm a successful login attributed to this workload identity, then confirm there are no rejected logins for it after the roll. The relying party is the independent witness; the cluster only knows that it handed out a file.
  2. Decode the payload of the token the Pod is actually mounting and confirm the audience array contains the secret manager and the expiry matches the value you declared. This distinguishes a token you configured from a token you inherited.
  3. Confirm the Pods are running and their own readiness endpoint reports the configuration store reachable. At 09:14 the Pods were running and could not authenticate, and only the second signal knew.
  4. Wait at least ninety minutes with the workload serving. Nothing shorter can observe an hourly failure, and a thirty minute smoke test passed cleanly on the broken build.
  5. List the token file inside a running container and confirm its modification time is later than the container start time while the application is still authenticating. A rotated file plus an unbroken service is the only evidence that the reload path works.
  6. Delete one Pod deliberately and confirm its replacement authenticates without help. A workload that only works while somebody is watching a rollout is not fixed.
  7. Confirm no Secret of the service account token type has appeared in the namespace during the incident. Under pressure, somebody usually creates one, and it will still be there next year.

Prevention

  • Declare the expiry, never inherit it. Set expirationSeconds explicitly on every projected token. The API server extends admission-injected tokens by default, up to a year, to keep legacy workloads alive during migrations, so a workload can run for months without ever observing a rotation.
  • Treat it worked before as a question, not as evidence. On the source cluster this workload held a token that never expired. The sentence describes a credential mechanism that no longer exists, and it is the single most misleading input to this incident.
  • Pin the audience at every relying party. No third party should accept the API server audience. The check costs nothing and it is the only boundary between one workload identity and all of them.
  • Lint the manifests. Fail the pipeline on any reference to a Secret of the service account token type. It is a one-line rule and it would have caught this before the migration window opened.
  • Alert on the two signals this incident produced. Any Pod in ContainerCreating for more than five minutes, and rejected logins at the secret manager grouped by workload identity, warning at three failures in five minutes.
  • Soak for two hours, not thirty minutes. A migrated workload must cross at least two token rotations in staging before promotion. Nothing shorter can see this class of failure, and every workload in the estate is about to be migrated the same way.