Cluster-wide credential rotation — the security incident's recovery

Once one credential is assumed leaked, every credential issued alongside it is suspect, and a cluster holds more of them than most operators expect: ServiceAccount tokens, control-plane certificates, image pull secrets, and node SSH keys. Rotating them is not a single command but a sequence that restarts the API server, invalidates the token every running workload is holding, and breaks image pulls if the registry secrets are missed. This lesson sets out what has to be rotated, in what order, and what stops working at each step.

The credential rotation

The credential rotation is the cluster’s security hygiene. The credential rotation invalidates the leaked credentials and issues new ones.

flowchart TD
    A[Credential rotation] --> B[ServiceAccount tokens]
    B --> C[Certificates]
    C --> D[API server]
    D --> E[Image pull secrets]
    E --> F[SSH keys]

The credential rotation is the cluster’s security hygiene.

The ServiceAccount rotation

The ServiceAccount rotation is the cluster’s most common credential rotation. The ServiceAccount tokens are rotated by deleting and recreating the secrets.

# Substitute your own values before running:
NS=production
SA_NAME=deploy-bot
SA_TOKEN_SECRET=deploy-bot-token-8k4rz

# Rotate the ServiceAccount token
kubectl delete secret "$SA_TOKEN_SECRET" -n "$NS"

# Force the ServiceAccount to issue a new token
kubectl create token "$SA_NAME" -n "$NS"

The ServiceAccount rotation is the cluster’s most common.

The certificate rotation

The certificate rotation is the cluster’s most critical. The certificates are rotated by kubeadm certs renew.

# Renew the certificates
kubeadm certs renew all

# Restart the control plane. kubeadm runs the API server,
# controller-manager, scheduler and etcd as static Pods, so
# there is no systemd unit for any of them: move the manifests
# out of the kubelet's watch directory and back again.
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/*.yaml /root/manifests-parked/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /root/manifests-parked/*.yaml /etc/kubernetes/manifests/

The certificate rotation is the cluster’s most critical.

The diagnostic

The canonical diagnostic:

# 1. Check the credentials
kubectl get secrets -A | grep -i token

# 2. Check the certificates
kubeadm certs check-expiration

# 3. Check the API server's logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200

# 4. Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

# 5. Check the events
kubectl get events -A --field-selector type=Warning

The diagnostic is the credentials, the certificates, the API server’s logs, the audit logs, and the events.

Common failures

• ServiceAccount token leaked. The token is leaked. The remediation is to rotate the token.
• Certificate expired. The certificate is expired. The remediation is to renew the certificate.
• API server compromised. The API server is compromised. The remediation is to rotate the API server’s certificate.
• Image pull secret leaked. The image pull secret is leaked. The remediation is to rotate the image pull secret.

flowchart TD
    A[Credential rotation] --> B{Token leaked?}
    B -->|Yes| C[Rotate the token]
    B -->|No| D{Certificate expired?}
    D -->|Yes| E[Renew the certificate]
    D---|No| F{API server compromised?}
    F -->|Yes| G[Rotate the API server's certificate]
    F -->|No| H{Image pull secret leaked?}
    H -->|Yes| I[Rotate the image pull secret]
    H -->|No| J[Unknown]

The remediation

The remediation depends on the credential:

# Substitute your own values before running:
NS=production
SA_NAME=deploy-bot
SA_TOKEN_SECRET=deploy-bot-token-8k4rz
PULL_SECRET=regcred
REGISTRY_SERVER=registry.example.com
REGISTRY_USER=ci-puller

# Option 1: Rotate the ServiceAccount token
kubectl delete secret "$SA_TOKEN_SECRET" -n "$NS"
kubectl create token "$SA_NAME" -n "$NS"

# Option 2: Renew the certificate, then recreate the static
# Pods so the components read the new files
kubeadm certs renew all
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/*.yaml /root/manifests-parked/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /root/manifests-parked/*.yaml /etc/kubernetes/manifests/

# Option 3: Rotate the API server's certificate
kubeadm certs renew apiserver

# Option 4: Rotate the image pull secret. Prompt for the password
# so it never lands in shell history.
read -rsp 'registry password: ' REGISTRY_PASSWORD; echo
kubectl delete secret "$PULL_SECRET" -n "$NS"
kubectl create secret docker-registry "$PULL_SECRET" -n "$NS" \
  --docker-server="$REGISTRY_SERVER" \
  --docker-username="$REGISTRY_USER" \
  --docker-password="$REGISTRY_PASSWORD"

The remediation is the credential rotation.

Production discipline

A cluster-wide credential rotation is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the security layer, identify the credentials, apply the remediation. The security is the cluster’s protection; the remediation is the credential rotation.

Quiz