Cert rotation in /etc/kubernetes/pki — the file-by-file walk

A cert rotation is a coordinated set of file writes followed by a static pod restart. This lesson walks the file-by-file behaviour of kubeadm certs renew, the relationship between the cert files and their consumers, and the static pod manifest hash that triggers the restart.

The PKI file map

The PKI directory has a one-to-one mapping between files and certificates:

/etc/kubernetes/pki/
├── ca.crt                  # cluster CA certificate
├── ca.key                  # cluster CA private key
├── apiserver.crt           # API server serving cert
├── apiserver.key           # API server private key
├── apiserver-kubelet-client.crt  # API server's kubelet client cert
├── apiserver-kubelet-client.key
├── front-proxy-ca.crt      # front-proxy CA certificate
├── front-proxy-ca.key      # front-proxy CA private key
├── front-proxy-client.crt  # aggregator client cert
├── front-proxy-client.key
├── sa.key                  # ServiceAccount signing key
├── sa.pub                  # ServiceAccount verifier
└── etcd/
    ├── ca.crt              # etcd CA certificate
    ├── ca.key              # etcd CA private key
    ├── server.crt          # etcd member serving cert
    ├── server.key
    ├── peer.crt            # etcd peer cert
    ├── peer.key
    ├── healthcheck-client.crt  # etcd healthcheck client cert
    └── healthcheck-client.key

Each .crt is consumed by something. The map:

flowchart LR
    subgraph ca.crt
      CA[cluster CA]
    end
    subgraph apiserver.crt
      AS[API server serving cert]
    end
    subgraph apiserver-kubelet-client.crt
      AKC[API server kubelet client cert]
    end
    subgraph front-proxy-client.crt
      FP[aggregator client cert]
    end
    subgraph sa.key
      SA[SA signing key]
    end
    CA --> AS
    CA --> AKC
    AS --> P1[kube-apiserver static pod]
    AKC --> P1
    FP --> P1
    SA --> P1
    P1 --> Restart[kubelet restarts static pod]

What gets written during renew

kubeadm certs renew all writes:

1. A new cert file for each .crt (e.g., apiserver.crt is overwritten with the new cert).
2. The corresponding .key file is unchanged (default behavior).
3. The kubeconfigs in /etc/kubernetes/ that reference the cert (admin.conf, controller-manager.conf, scheduler.conf, kubelet.conf).
4. The static pod manifests in /etc/kubernetes/manifests/. The manifests are not changed in content, but their hash changes (because the cert file is referenced via a ConfigMap or a downward-mounted file).

Let me trace one renewal:

sudo kubeadm certs renew apiserver

What happens:

1. kubeadm reads the current apiserver.crt, regenerates it with extended validity, and writes the new cert to /etc/kubernetes/pki/apiserver.crt.
2. The apiserver.key is unchanged.
3. The static pod manifest for the API server (/etc/kubernetes/manifests/kube-apiserver.yaml) is rewritten — not because the manifest content changed, but because kubeadm updates the manifest hash used by the kubelet to detect changes.
4. The kubelet sees the manifest change, terminates the existing API server pod, and starts a new one with the new manifest.
5. The new pod mounts the new cert and serves it via TLS.

sequenceDiagram
    participant O as Operator
    participant K as kubeadm
    participant FS as Filesystem
    participant KL as kubelet
    participant AS as API server pod
    O->>K: kubeadm certs renew apiserver
    K->>FS: write /etc/kubernetes/pki/apiserver.crt (new)
    K->>FS: update /etc/kubernetes/manifests/kube-apiserver.yaml hash
    K->>KL: notify via file watch
    KL->>AS: terminate pod
    KL->>AS: start new pod with new cert
    AS-->>O: API server serving new cert

The static pod manifest hash

The kubelet watches the static pod manifests in /etc/kubernetes/manifests/. When a manifest’s content hash changes, the kubelet terminates the existing pod and starts a new one. The hash is computed by the kubelet, not stored explicitly; kubeadm forces a hash change by rewriting the manifest file with the same logical content but different metadata (e.g., a comment or annotation).

# Before renewal
sudo crictl ps | grep kube-apiserver
# 1a2b3c4d...  kube-apiserver  Running  0  24h

# Run renewal
sudo kubeadm certs renew apiserver

# After renewal
sudo crictl ps | grep kube-apiserver
# 5e6f7g8h...  kube-apiserver  Running  0  1m   # new pod ID, restarted

The new pod ID confirms the restart.

The renewal of multiple certs

kubeadm certs renew all runs the renewal for each cert in sequence. The order is:

1. CA certs (ca, etcd-ca, front-proxy-ca) — only if their validity is below threshold.
2. Leaf certs (apiserver, apiserver-kubelet-client, front-proxy-client, etcd-server, etcd-peer, etcd-healthcheck-client).
3. Kubeconfigs (admin.conf, controller-manager.conf, scheduler.conf, kubelet.conf).

Each leaf cert renewal triggers a static pod restart. The etcd-related certs (etcd-server, etcd-peer, etcd-healthcheck-client) restart the etcd static pod.

The orphan files

If a cert file is deleted without being renewed, the cluster breaks:

sudo rm /etc/kubernetes/pki/apiserver.crt
# The API server's static pod is broken; the kubelet
# cannot reconcile; the cluster is unreachable.

The recovery is to restore from the etcd snapshot (which includes the certs) or re-issue the cert via kubeadm init phase certs apiserver.

The kubelet’s view

The kubelet does not have a cert in /etc/kubernetes/pki/. The kubelet’s client cert is in /var/lib/kubelet/pki/kubelet-client-current.pem:

ls -la /var/lib/kubelet/pki/

kubelet-client-2026-08-16.pem
kubelet-client-current.pem -> kubelet-client-2026-08-16.pem

The kubelet rotates its own cert via the CSR flow at 75% of validity. The cert file is symlinked to the current cert; the kubelet rewrites the symlink when the new cert is in place.

The kubeconfig’s cert ref

The kubeconfigs in /etc/kubernetes/*.conf are YAML files with a certificate-authority-data field that contains the CA cert bundle. After renewal, the bundle is still the same CA (the CA was not regenerated), so the kubeconfigs’ CA-data is unchanged. The client-certificate-data field is updated to the new client cert.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded CA bundle>
    server: https://lb:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
users:
- name: kubernetes-admin
  user:
    client-certificate-data: <base64-encoded client cert>
    client-key-data: <base64-encoded client key>

The client-certificate-data and client-key-data are regenerated by kubeadm certs renew admin.conf.

Cross-course references

• The Linux course covers file permissions and inode semantics for the PKI directory.
• The Observability course covers the kubelet’s metric kubelet_certificate_expiration_seconds.
• The VyOS course covers CA chain validation when integrating external services.

Quiz

Production discipline

• Don’t delete cert files manually. The cluster depends on them.
• Use kubeadm init phase certs <name> to re-issue. Never hand-edit certs.
• Trace the manifest hash. The kubelet’s restart is driven by the manifest, not by the cert file directly.
• Verify the restart with crictl. The new pod ID confirms the renewal took effect.
• Document the PKI directory in the runbook. Every consumer, every file, every path.
• Test the renewal on staging. Catch the manifest-hash issue before production.

The rotation is a coordinated set of file writes followed by a static pod restart. Operating it well is understanding which files are written and which consumers restart.