Kubernetes workload identity and certificate incidents

Every Pod that talks to the API server carries a credential nobody issued by hand. Since v1.22 that credential is a bound, projected ServiceAccount token: a JWT the kubelet obtains through the TokenRequest API, refreshes on a schedule, and ties to the lifetime of the Pod. It is a considerable improvement on what came before, and it has three sharp edges that surface only during an incident. This lesson is about those edges and about the cluster certificates that fail on the same schedule.

What the admission controller actually injects

The ServiceAccount admission controller adds a projected volume to the Pod named kube-api-access with a random suffix. It has three sources, and only the first is a credential.

sources:
  - serviceAccountToken:
      path: token
      expirationSeconds: 3600
  - configMap:
      name: kube-root-ca.crt
      items:
        - key: ca.crt
          path: ca.crt
  - downwardAPI:
      items:
        - path: namespace
          fieldRef:
            fieldPath: metadata.namespace

The token source produces the JWT. The ConfigMap source supplies the CA bundle that verifies a connection to the API server through kubernetes.default.svc, and nothing wider than that. The downward API source writes the namespace so a client library can construct its own object paths. A Pod that must not hold an API credential at all sets automountServiceAccountToken: false and keeps none of this.

The token’s own properties are the part worth memorising. expirationSeconds defaults to one hour and cannot go below 600. The audience defaults to the API server’s identifier, and setting it to something else is how you mint a token intended for an external relying party rather than for Kubernetes itself. The token is bound to the Pod, so it stops working when the Pod is deleted, regardless of the clock.

Rotation is the kubelet’s job and reloading is yours

The kubelet proactively requests a new token once the current one is older than 80% of its total time to live, or once it is older than 24 hours, whichever comes first. It rewrites the file in place. It does not signal the process, and nothing restarts the container.

sequenceDiagram
    participant K as kubelet
    participant A as kube-apiserver
    participant P as Pod process
    K->>A: TokenRequest for the Pod
    A-->>K: JWT, bound to the Pod
    K->>P: write /var/run/secrets/.../token
    Note over K: age past 80 percent of TTL, or past 24h
    K->>A: TokenRequest again
    A-->>K: fresh JWT
    K->>P: overwrite the same path
    Note over P: process must re-read the file

The consequence is the most common Kubernetes identity incident there is: an application that read the token once at start-up runs happily for an hour and then begins receiving 401 responses, with no deployment, no configuration change and no obvious trigger. The documented expectation is that the application reloads the token, and that reading it from disk on a fixed schedule, for example every five minutes, is usually good enough. Any client library that caches the bearer token for the process lifetime is broken on a modern cluster, and the symptom will appear long after the change that introduced it.

There is no revocation API

If you decide a bound token is no longer trustworthy, there is no endpoint that invalidates it. The documentation is explicit: delete the Pod, because deleting a Pod expires its bound tokens. That is the entire mechanism.

Tokens can be bound to a Pod, a Secret or a Node, with node binding generally available since v1.33, and the bound object’s name and UID are carried as private claims inside the JWT. When the referenced object or the ServiceAccount is pending deletion, authentication with that token fails from 60 seconds after the deletion timestamp. That 60-second window is the fastest revocation the platform offers, and it only applies where a binding exists.

Two legacy details still bite. Auto-generated token Secrets stopped being created in v1.24 and the gate that controlled it was removed in v1.27, so any procedure that says “create a ServiceAccount and read the token from its Secret” is describing a cluster nobody runs any more. A legacy token cleaner has been stable since v1.30; it checks every 24 hours, marks unused auto-generated token Secrets invalid with a kubernetes.io/legacy-token-invalid-since label, and eventually deletes them, with the thresholds defaulting to one year. Use of an invalidated legacy token is recorded as an audit annotation and counted by a dedicated metric, which makes it the cleanest way to find the last few workloads still using one.

kubectl create token deployer \
  --namespace payments \
  --audience vault.example.com \
  --duration 1800s

That is the supported way to obtain an ad-hoc token, and the --duration you ask for is a request rather than a promise: the server may return a token with a longer or a shorter lifetime.

Signers, approval, and the lifetime you actually get

Cluster certificates are requested through the certificates.k8s.io/v1 CertificateSigningRequest API, where spec.signerName is required. Approval and signing are separate steps; approving a request does not issue anything by itself.

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: web-01-serving
spec:
  signerName: kubernetes.io/kubelet-serving
  expirationSeconds: 157680000
  usages:
    - digital signature
    - server auth
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNU

That request asks for five years and will not get it. The built-in signers use the minimum of spec.expirationSeconds and the controller manager’s --cluster-signing-duration, which defaults to 8760h0m0s. The request is not rejected, it is silently shortened, and the team discovers the real lifetime a year later.

Approval behaviour differs per signer and this is where kubelet incidents come from. Only kubernetes.io/kube-apiserver-client-kubelet may be auto-approved by the controller manager. The kubernetes.io/kubelet-serving signer is never auto-approved, for stated security reasons, so serving-certificate rotation depends on a human or a custom approving controller. That is why kubelet serving certificates are the ones that expire: the gate that enables their rotation has been beta since v1.12 and remains beta, and the CSRs it files sit pending until somebody acts. Requests are garbage collected an hour after they are approved, denied or failed, and 24 hours after they are created if still pending, so an unnoticed backlog disappears on its own.

If you write a custom approver, the documented safety checks are not optional: accept only requests from nodes, where the username has the node form and the groups contain the node group; accept only server authentication usages; and accept only IP and DNS subject alternative names belonging to the requesting node, with no URI or email entries at all.

What kubeadm renews, and what it refuses

kubeadm renews all control-plane certificates during an upgrade, and --certificate-renewal defaults to true on both upgrade apply and upgrade node. On a cluster that is upgraded more often than once a year, expiry never happens. On a cluster that has been left alone, it happens all at once.

kubeadm certs check-expiration

# Run on EVERY control-plane node, then restart the static Pods.
kubeadm certs renew all
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml
sleep 25
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml

Renewal reads the CA key from disk, so it works on a cluster whose API server is already refusing connections. It uses the existing certificates as the authoritative source of common name, organisation and subject alternative names, and does not consult the kubeadm configuration ConfigMap. Moving a manifest out of the directory and back is the documented way to restart a static Pod; the kubelet notices within roughly its file check interval.

Two exclusions define the rest of the work. kubelet.conf is deliberately not renewed, because the kubelet rotates its own client certificate into /var/lib/kubelet/pki, and recovering a node whose rotation failed is a separate procedure that regenerates that kubeconfig from a working control plane. And kubeadm does not support rotation or replacement of the CA certificates at all. There is no command; the CA transition is a designed, disruptive project that this course treats separately.

Production discipline

1. Make token reload a review item. Any in-house API client must re-read the token file periodically; a token cached at start-up is an incident with an hour-long fuse.
2. Measure the exp claim, not the manifest. With extended expiration on by default, the Pod spec is not evidence of the credential lifetime you are actually issuing.
3. Alert on pending CSRs. Kubelet serving requests are never auto-approved and are garbage collected after 24 hours pending, so an unwatched queue silently empties itself.
4. Upgrade more often than once a year. Regular upgrades renew every control-plane certificate as a side effect and remove the whole class of expiry incident.
5. Rehearse the CA transition separately. kubeadm will not rotate a CA, so the procedure must be written, tested and scheduled rather than discovered during an outage.

Cross-course references

• Kubernetes for Production Sysadmins - Parts LX (ServiceAccounts) and LXXVI (Cluster Certificates) cover the platform operation of identities and certificates that this lesson examines purely as credentials with lifetimes and revocation properties.
• Observability for Production Sysadmins - Part LXIV (TLS Monitoring) covers turning the expiry windows described here into an alert that fires with weeks of margin instead of an outage.
• Linux for Production Sysadmins - Part XXIV (Time Synchronisation) covers the clock discipline that decides whether a token or certificate is judged valid on the node evaluating it.

Quiz