Cluster certificates — the PKI that holds the cluster together

Every kubeadm-managed cluster has a Public Key Infrastructure (PKI) that governs both authentication and encryption. The PKI is small enough to enumerate completely — about twenty certificate and key files in a single directory — and each one has a defined purpose and expiry. This lesson walks the layout, the certs, and what each one does.

The PKI directory

kubeadm init creates the cluster’s PKI in /etc/kubernetes/pki/:

ls /etc/kubernetes/pki/

ca.crt
ca.key
apiserver.crt
apiserver.key
apiserver-kubelet-client.crt
apiserver-kubelet-client.key
front-proxy-ca.crt
front-proxy-ca.key
front-proxy-client.crt
front-proxy-client.key
sa.key
sa.pub
etcd/
  ca.crt
  ca.key
  server.crt
  server.key
  peer.crt
  peer.key
  healthcheck-client.crt
  healthcheck-client.key

Sibling files in /etc/kubernetes/ include the kubeconfigs that bind identities to client certs:

admin.conf
controller-manager.conf
scheduler.conf
kubelet.conf

flowchart LR
    A["/etc/kubernetes/pki/"] --> B["Cluster CA (ca.crt, ca.key)"]
    A --> C["API server (apiserver.crt, apiserver.key)"]
    A --> D["API server kubelet client"]
    A --> E["Front-proxy CA + client"]
    A --> F["ServiceAccount signing key"]
    A --> G["etcd/ (CA, server, peer, healthcheck)"]
    H["/etc/kubernetes/*.conf"] --> I["admin, controller-manager, scheduler, kubelet"]

The cluster CA

The cluster CA (ca.crt, ca.key) is the trust anchor for the entire cluster. It signs:

• The API server’s serving certificate.
• The API server’s kubelet client certificate.
• The kubelet’s client certificate (signed on demand via the kube-controller-manager’s CSR signers).
• The admin.conf, controller-manager.conf, scheduler.conf, and kubelet.conf kubeconfigs.

sudo openssl x509 -in /etc/kubernetes/pki/ca.crt \
  -noout -subject -issuer -dates

subject=CN = kubernetes
issuer=CN = kubernetes
notBefore=Aug 16 10:00:00 2026 GMT
notAfter=Aug 16 10:00:00 2027 GMT

Self-signed (issuer == subject). 1-year validity.

The API server certificate

The API server’s serving certificate (apiserver.crt) is what clients present against when they connect to the API server:

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt \
  -noout -subject -ext subjectAltName

subject=CN = kube-apiserver
X509v3 Subject Alternative Name:
    DNS:cp-1, DNS:cp-2, DNS:cp-3, DNS:lb,
    DNS:localhost, DNS:kubernetes,
    IP Address:10.0.1.10, IP Address:10.0.1.11,
    IP Address:10.0.1.12, IP Address:10.0.1.254,
    IP Address:127.0.0.1

The SAN list is critical: every hostname or IP that a client might use to reach the API server must be in the SAN, or the TLS handshake fails. Adding a new control-plane host requires re-signing this cert with the new SAN.

The API server kubelet client cert

The API server authenticates to the kubelet using the apiserver-kubelet-client cert. This is a client cert (presented to the kubelet, not the other way around), and its CN is what the kubelet’s webhook authorization checks.

The front-proxy CA and client

The front-proxy CA authenticates the API server’s aggregated-API path. The aggregator uses the front-proxy-client cert to authenticate to upstream extension API servers (metrics-server, custom metrics API, ingress controllers with admission). Without this cert, the aggregator cannot forward requests to extension API servers.

The ServiceAccount signing key

The sa.key and sa.pub pair are used by the API server to sign and verify ServiceAccount tokens:

ls -la /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub

These keys do not have a certificate expiry — they are cryptographic keys, not X.509 certs. Rotating them invalidates every existing SA token across the cluster, which is correct but disruptive. Plan SA-key rotation for major cluster operations.

The etcd certificates

The etcd/ directory contains a separate PKI for the etcd cluster:

• ca.crt / ca.key — etcd’s own CA.
• server.crt / server.key — the etcd member’s serving cert (used for peer traffic and client traffic).
• peer.crt / peer.key — the cert etcd uses to authenticate to peer members.
• healthcheck-client.crt / healthcheck-client.key — the cert used by etcdctl endpoint health and liveness probes.

etcd’s CA is independent of the cluster CA. Replacing the cluster CA does not affect etcd.

The kubeconfigs

The kubeconfigs in /etc/kubernetes/ are not certificates themselves; they bundle a client certificate with configuration:

The cert lifecycle

Every cert (except the SA keys) has a 1-year default validity. The lifecycle is:

1. Generated at kubeadm init (or kubeadm join).
2. In use for the duration of the validity.
3. Renewed by kubeadm certs renew (or by external tooling).
4. Replaced by the kubelet; the static pods are restarted; control plane resumes.

The cert is the cluster’s hygiene discipline. Operating it well is knowing what each cert does, when it expires, and how to rotate it without downtime.

Cross-course references

• The Linux course covers X.509 certificate anatomy (man 5 x509v3_config) — the underlying ASN.1 structure Kubernetes certs reuses.
• The Observability course covers certificate expiry alerting via Prometheus probe_ssl_earliest_cert_expiry.
• The OPNsense course covers the CA chain when integrating external services with the cluster.

Quiz

Production discipline

• Inventory every cert. Run kubeadm certs check-expiration monthly (Lesson 3) and store the output in version control.
• Document the SAN list. When adding a control-plane host, the SAN list grows; document the change in the runbook.
• Treat the cluster CA as sacred. Rotating it is a planned operation, not a default.
• Test renewal on staging. kubeadm certs renew is non-disruptive but not free; rehearse it.
• Alert on cert expiry. 30/15/7-day warnings via Prometheus probe_ssl_earliest_cert_expiry.
• Avoid SA key rotation without planning. It breaks every workload’s token cache.

The PKI is small, enumerable, and finite. Operating it well is keeping every cert in that enumeration current and accounted for.