high riskcluster affecting~45 min

Runbook: Renew Cluster Certificates

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture every cert expiry: kubeadm certs check-expiration
· Capture the current date and the earliest expiry: date -u and kubeadm certs check-expiration
· Capture the kubelet certs on every node: openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates
· Capture the etcd certs on every control-plane node: ls /etc/kubernetes/pki/etcd/
· Capture the API server cert SANs: openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep -A2 "Subject Alternative Name"
· Confirm a change ticket is open and the change window current

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Plan the rotation: identify which certs expire within the next 30 days
2Renew the control-plane certs: kubeadm certs renew all (on a single control-plane node for HA, then copy to others)
3For HA clusters: copy the renewed CA to other control-plane nodes and restart the static pods
4Renew the kubelet serving cert on every node: kubeadm certs renew kubelet-serving or by re-bootstrapping
5Restart the API server, scheduler, controller-manager, and etcd static pods (kubelet does this when their manifest files are touched, or by crictl stop)
6Restart kubelet on every node so it picks up the new client cert
7Confirm the cert chain: from a remote client, openssl s_client -connect <api-vip>:6443 -showcerts shows the new CA
8Validate every component: kubectl get nodes, kubectl get pods -A, and a workload smoke test

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubeadm certs check-expiration reports every cert valid for at least 90 days
✓openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates on every node shows the new expiry
✓curl -k https://<api-vip>:6443/healthz returns 200
✓kubectl get nodes returns every node Ready
✓kubectl auth can-i get pods returns yes
✓A test workload runs end-to-end (create, expose, hit, delete)
✓etcdctl ... endpoint health --cluster -w table reports every member healthy
✓kubectl get --raw=/healthz returns ok
✓No x509 errors in API server, kubelet, or etcd logs since the rotation

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a kubelet cert rotation broke the kubelet auth to the API server, restore the previous cert from /etc/kubernetes/pki.bak or the bootstrap path
↶If the API server fails to start with the new cert, restore the previous cert files and restart the static pod
↶If etcd fails to start with the new cert, restore /etc/kubernetes/pki/etcd from backup
↶For an HA cluster, if rotation succeeded on one node but failed on another, restore the working certs to the failing node
↶Capture the kubeadm output, the cert files, and the logs to the change ticket before any rollback

6 · Escalation

When the runbook isn't enough, contact:

· CA cert expired (not just leaf): this requires a coordinated rotation of every component; escalate to platform ownership
· Kubelet cannot re-bootstrap (cluster CA changed): re-bootstrap the node manually with the new CA
· API server unreachable after rotation: a node is still using the old CA; restart kubelet and the API server on every node
· External clients cannot connect (kubeconfigs use the old CA): distribute the new CA to every client
· A workload that uses a Kubernetes API client secret cannot authenticate: the SA token was rotated; see kubernetes-cxxi-06-service-account-tokens

Cluster certificates have finite lifetimes. kubeadm renews control-plane certificates every year by default; kubelet serving certificates every year; client certificates every year. The runbook rotates them on demand and verifies every component still trusts the chain.

1. Check current expiry

Read-only / SafeCheck current expiry


# Columns: CERTIFICATE, EXPIRES, RESIDUAL TIME

2. Back up the existing certificates

Read-only / SafeBack up the existing certificates


sudo cp -r /etc/kubernetes /etc/kubernetes.bak-$(date -u +%Y%m%dT%H%M%SZ) || true
ls -d /etc/kubernetes*

A backup is the rollback path for a bad rotation.

3. Renew control-plane certificates

Read-only / SafeRenew control-plane certificates


# Confirm the renewal
sudo kubeadm certs check-expiration
sudo ls -l /etc/kubernetes/pki/

kubeadm certs renew all renews:

apiserver, apiserver-kubelet-client, front-proxy-client
etcd-server, etcd-peer, etcd-healthcheck-client
The CA certs are NOT renewed (they are 10-year certs by default)

4. For HA clusters: copy the renewed certs

Read-only / SafeFor HA clusters: copy the renewed certs


sudo rsync -a /etc/kubernetes/pki/ <cp2>:/etc/kubernetes/pki/
sudo rsync -a /etc/kubernetes/pki/ <cp3>:/etc/kubernetes/pki/
sudo rsync -a /etc/kubernetes/admin.conf <cp2>:/etc/kubernetes/admin.conf
sudo rsync -a /etc/kubernetes/admin.conf <cp3>:/etc/kubernetes/admin.conf

# The kubelet and kube-apiserver static pod manifest files
sudo rsync -a /etc/kubernetes/manifests/ <cp2>:/etc/kubernetes/manifests/
sudo rsync -a /etc/kubernetes/manifests/ <cp3>:/etc/kubernetes/manifests/

In HA, every control-plane node must have identical cert files; only the etcd member name in the manifest differs.

5. Restart the control plane

Read-only / SafeRestart the control plane


ssh "$n" -- bash -c '
  sudo crictl stop $(sudo crictl ps -a -q --name kube-apiserver) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name kube-scheduler) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name kube-controller-manager) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name etcd) 2>/dev/null || true
'
done

sleep 30
for n in <cp1> <cp2> <cp3>; do
ssh "$n" -- sudo crictl ps -a | grep -E 'kube-apiserver|etcd' || echo "$n: not running"
done

6. Renew kubelet certificates

Read-only / SafeRenew kubelet certificates


for n in $(kubectl get nodes -o name | cut -d/ -f2); do
ssh "$n" -- bash -c '
  sudo kubeadm certs renew kubelet-serving 2>&1 || true
  sudo systemctl restart kubelet
'
done

# Verify
for n in $(kubectl get nodes -o name | cut -d/ -f2); do
ssh "$n" -- sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates
done

kubeadm certs renew kubelet-serving renews only the serving cert. The kubelet client cert (used to talk to the API server) is renewed by re-bootstrapping or by manually replacing it.

7. Validate

Read-only / SafeValidate


sudo kubeadm certs check-expiration
kubectl get nodes -o wide
kubectl get pods -A | wc -l

# Validate the chain from a remote client
openssl s_client -connect <api-vip>:6443 -showcerts < /dev/null 2>&1 | grep -E "subject=|issuer=|verify return'

# A workload smoke test
kubectl run smoketest --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never --command -- sleep 30
kubectl wait --for=condition=Ready pod/smoketest --timeout=60s
kubectl delete pod smoketest --wait=false

8. Distribute the new CA to clients

If external clients (CI/CD, kubectl on workstations, monitoring) use the cluster CA, distribute the new CA to them.

Read-only / SafeDistribute the new CA to clients


sudo cat /etc/kubernetes/pki/ca.crt > /tmp/ca.crt.new

# For every client that uses the cluster
scp /tmp/ca.crt.new admin@bastion.internal:/etc/kubectl/ca.crt
scp /tmp/ca.crt.new ci-runner.internal:/etc/kubectl/ca.crt

# Update kubeconfig on workstations
scp /tmp/ca.crt.new user@workstation.internal:~/.kube/ca.crt
ssh user@workstation.internal "kubectl config set clusters.kube.certificate-authority-data "$(base64 -w0 ~/.kube/ca.crt)"'

Common pitfalls

Symptom	Cause	Action
API server fails to start with cert error	Manifest references a cert file that was not renewed	Re-create the manifest with the right paths
Kubelet auth fails after renewal	Kubelet still has the old client cert	Re-bootstrap or replace the client cert manually
External client cannot connect	kubeconfig has the old CA	Distribute the new CA
`etcd` member cannot rejoin cluster	Cert mismatch between members	Restore the matching cert from a control-plane node
Renewal succeeds but the cluster is still using old cert	Static pods not restarted	Restart the static pods explicitly

A cert rotation succeeds only when every component trusts the new chain. The runbook verifies the chain from a remote client, not just from the control plane.