Certificate expired · Break/Fix · Kubernetes

Reported symptoms

The cluster is three control-plane nodes and twelve workers, built with kubeadm and never upgraded. It has been in production for a year and has been, by every measure anyone tracks, boring.

At 10:00 UTC it stopped answering. The incident channel filled up faster than the facts did:

Every kubectl command, from every workstation and from the CI runner, fails immediately: Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid.
An engineer adds --insecure-skip-tls-verify out of habit and gets a completely different error: error: You must be logged in to the server (Unauthorized). The channel spends twenty minutes on whether an RBAC change went out overnight. It did not.
The product is fine. Every application Pod is serving, no container has restarted, and not one external check has flapped.
crictl ps on cp-1 shows kube-apiserver Running, 0 restarts, up 47 days. Whatever is broken, the control plane has not crashed.
The kubelet journal on all fifteen nodes is filling with x509 errors against the API server — and those same nodes keep every Pod they already had running perfectly.
The Grafana cluster dashboards went blank at 10:02. The first hypothesis in the channel is a Prometheus outage, and two people go and look at Prometheus.

At 10:35 somebody finds the answer, or half of it. kubeadm certs check-expiration shows an expiry of 10:00 this morning on every row. They run kubeadm certs renew all on cp-1, re-run check-expiration, and get 364 days on every line.

Nothing changes. Every symptom above is still true at 10:40, and now the team has a working remediation, a passing verification, and a cluster that is still down.

Evidence provided

Read-only / Safecp-1 at 10:05 — one date, every leaf certificate

$ sudo kubeadm certs check-expiration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Aug 18, 2026 10:00 UTC   <invalid>        no
apiserver                  Aug 18, 2026 10:00 UTC   <invalid>        no
apiserver-etcd-client      Aug 18, 2026 10:00 UTC   <invalid>        no
apiserver-kubelet-client   Aug 18, 2026 10:00 UTC   <invalid>        no
controller-manager.conf    Aug 18, 2026 10:00 UTC   <invalid>        no
etcd-healthcheck-client    Aug 18, 2026 10:00 UTC   <invalid>        no
etcd-peer                  Aug 18, 2026 10:00 UTC   <invalid>        no
etcd-server                Aug 18, 2026 10:00 UTC   <invalid>        no
front-proxy-client         Aug 18, 2026 10:00 UTC   <invalid>        no
scheduler.conf             Aug 18, 2026 10:00 UTC   <invalid>        no

Illustrative output

Read-only / Safecp-1, cp-2, cp-3— run on every control-plane node; all three agree with each other and with the reference clock, which rules out skew before anything is changed

$ timedatectl show -p NTPSynchronized -p TimeUSec --value

Read-only / Safecp-1 — with verification on

$ curl -s --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:6443/livez

curl: (60) SSL certificate problem: certificate has expired

Illustrative output

Read-only / Safecp-1 — the same endpoint, with verification off

$ curl -sk https://127.0.0.1:6443/livez

ok

Illustrative output

Read-only / Safecp-1 at 10:40, AFTER kubeadm certs renew all reported 364 days

$ openssl s_client -connect 127.0.0.1:6443 -showcerts < /dev/null 2>/dev/null | openssl x509 -noout -subject -dates

subject=CN = kube-apiserver
notBefore=Aug 18 09:55:00 2025 GMT
notAfter=Aug 18 10:00:00 2026 GMT

Illustrative output

Read-only / Safeworker-04 — a different expiry entirely

$ sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

notBefore=May 21 08:14:00 2026 GMT
notAfter=Nov 21 08:14:00 2026 GMT

Illustrative output

Read-only / Safecp-1 at 10:40 — the renewal did not restart anything

$ sudo crictl ps | grep kube-apiserver

3b7a91ce2f04d  registry.k8s.io/kube-apiserver:v1.34.1  47 days ago  Running  0  kube-apiserver-cp-1

Illustrative output

Work the evidence before reading on

The expiry is not the interesting part. The interesting part is that the correct remediation was applied and verified, and the cluster stayed down.

curl succeeds with -k and fails with --cacert. What does that pair tell you about which component is broken, and which is fine?
Two clients, two completely different errors, one cluster. Trace a TLS handshake and work out which certificate each error is about, and why skipping verification changes the answer.
kubeadm certs check-expiration says 364 days. openssl s_client says the certificate expires at 10:00 this morning. Both commands ran on the same host, seconds apart, and both are correct. What is each one actually reading?
The workers are the only part of the cluster with a valid identity, and their certificate has an expiry three months from now rather than today. What is different about how it was issued?

Before continuing: name the step that stands between a renewed file and a renewed process, and say why no amount of re-running check-expiration can tell you whether it happened.

Root cause

1. One issue date, one expiry

kubeadm init generates the cluster PKI in a single pass and gives every certificate the same one-year validity. That is deliberate: it makes renewal one planned event a year instead of a continuous background process. The cost is that the cluster has exactly one hard deadline, and everything crosses it in the same minute — the API server serving certificate, its client certificates to etcd and to the kubelets, and the client certificates embedded in admin.conf, controller-manager.conf and scheduler.conf.

The certificate authorities did not expire; they are issued for years, and check-expiration lists them separately. Only the leaves went.

2. Why two clients report two different problems

A TLS handshake has an order, and the error you get is from whichever check fails first.

The client connects, the API server presents its serving certificate, and the client verifies it against the cluster CA. That certificate expired at 10:00, so verification fails and kubectl reports x509: certificate has expired. The client never gets far enough to present its own credentials.

Add --insecure-skip-tls-verify and the client stops checking the server. Now the handshake proceeds to client authentication, the client presents the expired certificate from admin.conf, the API server rejects it, and the request arrives with no identity attached. The API server answers Unauthorized.

Both messages are accurate. Neither one says “the PKI expired”, and the second one looks exactly like an RBAC regression, which is why it cost twenty minutes.

3. Why the workers were fine

The kubelet client certificate is not part of the init-time PKI. The kubelet requests it through a CSR that the kube-controller-manager signs, and it renews it automatically at 75 percent of its validity — which is why kubelet-client-current.pem on worker-04 expires in three months rather than this morning.

So the nodes still had a valid identity and were only failing because they could not verify the API server. Their Pods kept running throughout, because the kubelet does not need the API server to keep containers that already exist alive.

4. Why the renewal did not fix anything

This is the part worth carrying away.

kubeadm certs renew rewrites files: the certificates under /etc/kubernetes/pki/ and the kubeconfigs beside them. It does not reach into a running process. The control-plane components read their key material once, at startup, and the kube-apiserver process on cp-1 started 47 days ago. It went on serving the certificate it loaded then.

And kubeadm certs check-expiration reads the same files the renewal just wrote. It confirmed that the write succeeded. It cannot confirm, and does not claim, that any process picked the new material up.

The file, the process, and the check that cannot tell them apart

There is no kube-apiserver.service. On a kubeadm cluster the API server, controller-manager, scheduler and etcd are static Pods: manifests in /etc/kubernetes/manifests/ that the kubelet runs directly. systemctl restart kube-apiserver returns Unit kube-apiserver.service not found, and systemctl restart kubelet is a real unit but does not by itself force the static Pods to be recreated.

The kubelet re-reads that directory on its fileCheckFrequency, 20 seconds by default, and recreates a Pod whose manifest has changed or reappeared. So the way to restart a control-plane component is to move its manifest out of the directory, wait, and move it back:

cd /etc/kubernetes/manifests
sudo mkdir -p /root/manifests-parked
sudo mv kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml etcd.yaml /root/manifests-parked/
sleep 30
sudo mv /root/manifests-parked/*.yaml /etc/kubernetes/manifests/

The general lesson is bigger than certificates. Any remediation that writes a file and any verification that reads the same file form a closed loop that proves only that the write happened. To verify a credential, ask the thing that uses it:

openssl s_client -connect 127.0.0.1:6443 -showcerts < /dev/null 2>/dev/null | openssl x509 -noout -dates

That reads the certificate off the wire, from the process, which is the only place the answer actually lives.

Check the clock before you renew anything

Clock skew produces this incident symptom for symptom, from a perfectly valid certificate. Validity is evaluated against the local clock, so a node running far enough ahead rejects a certificate that from where it stands has not been issued yet, and a node running behind accepts one that everybody else considers dead.

The two are separated in one command per node, before anything is changed:

timedatectl status

If the clocks disagree, renewing certificates costs you the change window and fixes nothing, and you will then be debugging a cluster with a new PKI and the original fault still in place. Confirm the clock first. It is thirty seconds and it decides which incident you are in.

Resolution

Confirm the clock on all three control-plane nodes with timedatectl status before touching the PKI. Agreement between the nodes and with the reference clock is what separates expiry from skew, and only one of those two is fixed by renewing.
Pick one control-plane node and work only on that one. With the cluster already down there is no availability left to protect, but etcd quorum still is: parking all four manifests stops that node etcd member too, and taking a second node down at the same time turns a credential outage into a quorum outage.
Renew on that node: sudo kubeadm certs renew all. This rewrites the certificates under /etc/kubernetes/pki/ and the kubeconfigs in /etc/kubernetes/, including admin.conf.
Recreate the static Pods, which is the step the renewal does not perform. Move the four manifests out of /etc/kubernetes/manifests/, wait past the kubelet fileCheckFrequency of 20 seconds, confirm with sudo crictl ps that they are gone, then move them back.
Verify on the node by reading the served certificate, not the file: openssl s_client -connect 127.0.0.1:6443 must present a notAfter about a year out, and sudo crictl ps must show a kube-apiserver container that is minutes old rather than 47 days old.
Prove the chain now verifies: curl -s --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:6443/livez returns ok with verification enabled. This is the check that was failing, and -k deliberately cannot answer it.
Use the renewed admin.conf on the node itself to confirm the cluster is back: sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes. One healthy API server restores the whole control plane.
Repeat the renewal and the manifest recreation on the second node, verify it, then the third. One at a time. A node missed here reports its own expired dates and becomes the next incident.
Redistribute operator kubeconfigs. Every engineer copy of admin.conf is a copy, and renewing on the node does not touch it. Where a fresh credential is needed rather than a copy of the admin one, kubeadm kubeconfig user generates it on the control-plane node.
Do not restore etcd and do not run kubeadm reset. The datastore is intact and current; a restore would discard every write since the snapshot in exchange for fixing a problem it does not touch.

Verification

The served certificate is new on every control-plane node: openssl s_client -connect 127.0.0.1:6443 -showcerts < /dev/null 2>/dev/null | openssl x509 -noout -dates shows a notAfter roughly a year away. This is the check that would have caught the failed remediation at 10:40.
The processes actually restarted: sudo crictl ps shows kube-apiserver, kube-controller-manager, kube-scheduler and etcd with start times from the maintenance rather than from 47 days ago.
The chain verifies without -k: curl -s --cacert /etc/kubernetes/pki/ca.crt https://127.0.0.1:6443/livez returns ok on each node.
sudo kubeadm certs check-expiration reports roughly 364 days on every control-plane node, not just the first one. Run it in a loop over all three and compare, because divergence means a host was missed or was repaired out of band.
kubectl get nodes from a refreshed client kubeconfig returns every node Ready, and no client needs --insecure-skip-tls-verify to get an answer.
The kubelet journals stop producing x509 errors: sudo journalctl -u kubelet --since "10 minutes ago" | grep -c x509 returns 0 on a sample of workers. That is the nodes confirming they accept the new serving certificate.
The control plane is reconciling and not merely answering reads: kubectl get lease -n kube-system shows current holders for the controller-manager and scheduler leases, and a throwaway Deployment can be created, scaled and deleted.
A certificate-expiry alert exists and has been tested by evaluating it against the now-known expiry date, so the next anniversary arrives as a ticket rather than as a page.

Prevention

Alert on residual days, not on the expiry timestamp. The remaining time is the actionable number; a warning at 30 days and a page at 7 turns the anniversary into a maintenance window. An alert that fires on the day is not an alert, it is a postmortem.
Inventory every control-plane node, not one. Run kubeadm certs check-expiration across all of them and compare. Diverging dates mean a host was re-bootstrapped or restored out of band, and it will expire on its own private schedule.
Know why most clusters never hit this. kubeadm upgrade renews the PKI as a side effect, so a cluster on a regular upgrade cadence keeps resetting the clock without anyone deciding to. A cluster that has skipped upgrades for a year has two problems, and the certificates are the one that shows up first.
Write the static Pod recreation into the renewal runbook as a numbered step. It is the step this incident turned on, and it is the step most naturally written as a footnote.
Verify credentials by asking the process, not the filesystem. A verification that reads the file the remediation just wrote is a closed loop. openssl s_client reads what is actually being served.
Write down the recovery path for an expired admin.conf somewhere outside the cluster. The moment you need it is the moment you have no working kubectl to look it up with, and the wiki page may well be behind an ingress on the cluster that is down.
Keep NTP healthy and monitored on the control plane, for the same reason the clock check comes first in the diagnosis.

What if only the etcd certificates had expired?

The failure would have looked like a completely different incident, and it is worth being able to tell them apart on sight.

etcd has its own certificate authority, independent of the cluster CA, with its own server, peer and healthcheck-client certificates. If those expire while the cluster PKI is still valid, kubectl authenticates perfectly and then every request fails, because the API server cannot reach the store behind it. The signature is a 503 rather than a TLS error, and kubectl get --raw "/readyz?verbose" names the failing check directly instead of leaving you to guess.

The reason to keep the two apart is that the remediations diverge. An expired cluster PKI is repaired one control-plane node at a time with the API server as the thing you are trying to restore. An etcd certificate problem is repaired with a quorum you must not lose in the process, and the ordering constraints are much tighter.

The shared lesson is the same in both: renewing writes files, and until the static Pod is recreated the running process is still using the old material. The component that has to be restarted is different; the trap is identical.

Why this matters

This incident had a correct diagnosis at 10:35 and stayed down for another hour, because the remediation and the verification were reading and writing the same file and nothing in that loop touched the running process.

The habit worth taking away is to ask, of any verification, where the answer comes from. A check that reads the artefact the fix just produced can only confirm the fix ran. A check that asks the consumer of that artefact — the process serving the port, the client completing a handshake, the node accepting the connection — is the one that can still fail, and that is what makes it worth running.

The second habit is smaller and cheaper: read the exact error text before forming a theory. x509: certificate has expired and Unauthorized were two views of one fault, and the difference between them was a flag one engineer typed from muscle memory. The flag did not change the cluster. It changed which of the two expired certificates the handshake noticed first.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. One issue date, one expiry

2. Why two clients report two different problems

3. Why the workers were fine

4. Why the renewal did not fix anything

Resolution

Verification

Prevention