cert rotation and clock skew — the cryptographic drift

Control-plane certificates issued by kubeadm are valid for one year, so a cluster that is never upgraded reaches that date with nobody watching, and on the day kubectl, the controllers, and the kubelets all fail authentication at once. Clock skew produces the same symptom from a perfectly good certificate, because validity is checked against the local clock: a node running minutes ahead of the CA rejects a certificate that, from where it stands, has not been issued yet. Running kubeadm certs check-expiration and comparing date across the nodes separates the two.

The certificate rotation

The cluster’s certificates have a finite lifetime. The default is 1 year. The certificates must be rotated before they expire.

# Check the certificate's expiration
kubeadm certs check-expiration

# Renew the certificates
kubeadm certs renew all

The certificate rotation is the cluster’s cryptographic health.

The clock skew

The cluster’s clocks must be synchronized. The default synchronization is NTP (Network Time Protocol). The clock skew is the difference between the local clock and the reference clock.

# Check the clock
date

# Check the NTP
timedatectl status

# Check the NTP servers
ntpq -p

The clock skew is the cluster’s time synchronization.

The diagnostic

The canonical diagnostic:

# 1. Check the certificate's expiration
kubeadm certs check-expiration

# 2. Check the clock
date

# 3. Check the NTP
timedatectl status
ntpq -p

# 4. Check the API server's logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200

# 5. Check the kubelet's logs
journalctl -u kubelet -n 200

The diagnostic is the certificates, the clock, and the authentication.

Common failures

• Certificate expired. The certificate is past its expiration date. The remediation is to renew the certificate.
• Clock skew. The local clock is ahead of or behind the reference clock. The remediation is to sync the clock.
• NTP failure. The NTP server is unreachable. The remediation is to fix the NTP server.

flowchart TD
    A[Crypto failing] --> B{Certificate valid?}
    B -->|No| C[Renew the certificate]
    B -->|Yes| D{Clock skew?}
    D -->|Yes| E[Sync the clock]
    D---|No| F{NTP OK?}
    F -->|No| G[Fix the NTP]
    F -->|Yes| H[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Renew the certificates, then restart the control
# plane. kubeadm runs the four components as static Pods, so
# move each manifest out of the kubelet's watch directory and
# back again; there are no systemd units to restart.
kubeadm certs renew all
mv /etc/kubernetes/manifests/{kube-apiserver,kube-controller-manager,kube-scheduler,etcd}.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/{kube-apiserver,kube-controller-manager,kube-scheduler,etcd}.yaml /etc/kubernetes/manifests/

# Option 2: Sync the clock
chronyc makestep

# Option 3: Fix the NTP server
# (NTP-specific)

The remediation is the cryptographic recovery.

The kubeadm certs check-expiration

A real kubeadm certs check-expiration output:

CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Aug 16, 2027 04:23 UTC   364d            no
apiserver                  Aug 16, 2027 04:23 UTC   364d            no
apiserver-etcd-client      Aug 16, 2027 04:23 UTC   364d            no
apiserver-kubelet-client   Aug 16, 2027 04:23 UTC   364d            no
ca                         Aug 16, 2027 04:23 UTC   364d            no
etcd-ca                    Aug 16, 2027 04:23 UTC   364d            no
etcd-healthcheck-client    Aug 16, 2027 04:23 UTC   364d            no
etcd-peer                  Aug 16, 2027 04:23 UTC   364d            no
etcd-server                Aug 16, 2027 04:23 UTC   364d            no
front-proxy-ca             Aug 16, 2027 04:23 UTC   364d            no
front-proxy-client         Aug 16, 2027 04:23 UTC   364d            no

The output is the certificate’s expiration.

Production discipline

A cryptographic drift is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The cryptographic health is the cluster’s authentication; the remediation is the cryptographic recovery.

• Check the certificate’s expiration. kubeadm certs check-expiration prints the residual time for every control-plane certificate, etcd’s included.
• Renewing is not restarting. The components read their certificates at startup, so the renewed material only takes effect once each static Pod under /etc/kubernetes/manifests/ has been recreated.
• Check the clock and the NTP source. A valid certificate still fails verification when the local clock has drifted; chronyc makestep fixes the skew, and the unreachable NTP server is the cause behind it.

Quiz