Certificate management — kubeadm PKI, rotation, validation

Every kubeadm-managed cluster has a PKI that governs authentication and encryption: CA certs, server certs, peer certs, client certs, and the ServiceAccount signing keys. This lesson walks the PKI layout, the renewal procedure, and the production discipline of certificate expiry prevention.

The PKI layout

/etc/kubernetes/pki/
├── ca.crt         # cluster CA (1 year)
├── ca.key
├── apiserver.crt  # API server (1 year)
├── apiserver.key
├── apiserver-kubelet-client.crt # API server's kubelet auth (1 year)
├── apiserver-kubelet-client.key
├── sa.key         # SA signing key (no expiry)
├── sa.pub         # SA verifier (no expiry)
├── front-proxy-ca.crt      # front proxy CA (1 year)
├── front-proxy-ca.key
├── front-proxy-client.crt  # front proxy client (1 year)
├── front-proxy-client.key
└── etcd/
    ├── ca.crt     # etcd CA (1 year)
    ├── ca.key
    ├── server.crt # etcd server (peer + client) (1 year)
    ├── server.key
    ├── peer.crt   # peer cert (1 year)
    ├── peer.key
    ├── healthcheck-client.crt # for etcdctl health check (1 year)
    └── healthcheck-client.key

Inspecting certs

Every cert can be inspected:

sudo openssl x509 -in /etc/kubernetes/pki/ca.crt \
  -noout -dates -subject

notBefore=Aug  1 10:00:00 2026 GMT
notAfter=Aug  1 10:00:00 2027 GMT
subject=CN = kubernetes

The validity is 1 year by default. Certs with 30 days remaining should be in the calendar for renewal.

$ sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates -subject

notBefore=Aug  1 10:00:00 2026 GMT
notAfter=Aug  1 10:00:00 2027 GMT
subject=CN = kube-apiserver

The kubeadm certs renew command

The renewal tool:

sudo kubeadm certs renew all

This rotates every cert in the cluster:

CERTIFICATE                          EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                          Aug 01 10:00:00 2027 UTC  <no-residual>                        no
apiserver                           Aug 01 10:00:00 2027 UTC  <no-residual>   ca                      no
apiserver-kubelet-client            Aug 01 10:00:00 2027 UTC  <no-residual>   ca                      no
controller-manager.conf             ...                                              no
scheduler.conf                      ...
...

After renewal, the static pods (API server, controller- manager, scheduler) need a restart to pick up the new certs. kubeadm certs renew does this automatically.

Renewing a specific cert

sudo kubeadm certs renew apiserver

This renews a single cert. The static pod restart is automatic.

Renewing after kubeadm upgrade

sudo kubeadm upgrade apply v1.34.1
sudo kubeadm certs renew all

The upgrade may change the cluster’s CA; the certs renew follows to ensure all components have current certs.

The CSR flow for node certs

The kubelet’s client cert is requested via CSR:

# Watch pending CSRs
kubectl get csr
# NAME                  AGE   REQUESTOR                 CONDITION
# node-csr-...          5m    system:bootstrap:abc...   Pending

# Manual approval (auto-approval is the default)
kubectl certificate approve node-csr-...

The controller-manager auto-approves node CSRs by default. Workers’ certs are signed by the same CA.

The kube-controller-manager’s kubelet signing

The kube-controller-manager’s --cluster-signing-cert-file controls which CA signs node certs:

--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key

Default is the cluster’s CA. Customisation is rare.

The cert validity and SAN

Every cert has a Subject Alternative Name (SAN) that determines which hostnames / IPs it identifies:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt \
  -text | grep -A 2 'Subject Alternative Name'

X509v3 Subject Alternative Name:
  DNS:cp-1, DNS:cp-2, DNS:cp-3, DNS:lb, DNS:localhost,
  IP Address:10.0.1.10, IP Address:10.0.1.11, IP Address:10.0.1.12,
  IP Address:10.0.1.254, IP Address:127.0.0.1

The cert identifies all control-plane hosts and the LB. A new host requires a cert update.

The etcd cert rotation

Etcd certs rotate independently:

# Renew etcd-server certs
sudo kubeadm certs renew etcd-server

# This restarts the etcd member
# Wait for it to come back: kubectl get pods -n kube-system

The peer certs (etcd-peer) are renewed similarly.

The CA rotation

Rotating the CA itself:

sudo kubeadm certs renew ca

This is rare but possible. It rotates the CA; every cert must be re-signed with the new CA.

The SAN rotation (adding a host)

For a new control-plane host with a new SAN:

# Substitute your own value before running:
NEW_HOST_DNS=cp-04.example.com

sudo kubeadm init phase certs apiserver \
  --apiserver-san="$NEW_HOST_DNS"

This re-signs the apiserver cert with the new SAN list.

The dashboard’s view

Production clusters run dashboards / alerts that track cert expiry. A common Prometheus alert:

- alert: KubeCertExpiringSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 30 * 24 * 3600
  for: 1d
  labels:
    severity: warning
  annotations:
    summary: "Cert expiry on {{$labels.instance}} within 30 days"

The blackbox_exporter is used to probe the API server’s TLS endpoint and extract cert expiry.

The cert validity in the SaaS age

Cloud-managed certificates rotate automatically. On-prem clusters require operator attention:

# 30 days before expiry: a calendar reminder
# 7 days before expiry: rotate certs
# Post-rotation: verify cluster health

A 1-year cert is the kubeadm default. Some teams reduce to 90 days for compliance.

The kubelet client cert rotation

The kubelet’s client cert (different from the API server cert) is auto-rotated by the kubelet’s CSR flow:

sequenceDiagram
    participant K as kubelet
    participant AS as API server
    K->>K: cert about to expire (75% of validity)
    K->>AS: submit CSR
    AS-->>K: signed cert
    K->>K: install new cert

The auto-rotation is configurable; the default is at 75% of the cert’s validity.

The discipline

• Calendar reminder. All kubeadm certs expire; track in the calendar.
• Test cert renewal. Run kubeadm certs renew on staging first.
• Verify the cluster after renewal. kubectl get nodes, get pods -n kube-system, get endpointslices.
• Document the SAN list. When adding hosts, ensure the SAN is updated.
• Use external monitoring. A Prometheus alert on cert expiry catches forgotten certs.
• Don’t rotate SA keys without planning. It invalidates all SA tokens.

Quiz

Production discipline

• Calendar reminder for cert expiry. All kubeadm certs expire; plan ahead.
• Renew followers first, leader last. Avoid simultaneous unavailability.
• Test renewal on staging. Catch misconfiguration before production.
• SAN list in version control. When adding hosts, the SAN list is updated.
• External monitoring via Prometheus / alert. Catch forgotten certs.
• Avoid SA key rotation without planning. It invalidates all SA tokens.

Cert management is the cluster’s hygiene discipline. Operating it well is keeping certs current and validated.