API server unavailable

Reported symptoms

The page came from the CI system at 21:53. Nothing else paged.

The cluster is three control-plane nodes, cp-1 to cp-3, behind a keepalived VIP with HAProxy in front of port 6443, and eighteen workers. Engineers reach it through the VIP; the CI runner has an older kubeconfig that names cp-2 directly.

By 22:10 the incident channel had four descriptions of the same outage and no agreement on what was broken:

• Engineers running kubectl get Unable to connect to the server: EOF, instantly, from every workstation.
• The CI runner gets connection refused instead. Two different errors from one cluster is the first thing somebody argues about.
• The product is completely healthy. Every application Pod is still serving, the external checks have not flapped, and no customer has noticed anything.
• Nothing is being created. A Deployment scaled to 12 replicas at 21:40 still has 4 Pods. A CronJob due at 22:00 did not run.

The first three remediation attempts each made sense and each failed:

1. systemctl restart kube-apiserver on cp-1. It returns Unit kube-apiserver.service not found. Someone concludes the control plane was never installed on these hosts and starts looking for a rebuild runbook.
2. The HAProxy stats page shows all three backends DOWN with Layer4 connection problem. The network team is asked to check the fabric between the load balancer and the control plane.
3. The kubelet is checked on all three nodes. It is active (running), its journal is clean, and the container runtime is healthy. That is read as proof the nodes are fine.

The only change in the window is a ticket titled enable API audit logging, closed successful at 21:38. The engineer who ran it verified with kubectl get nodes after each of the three nodes and recorded three passes.

Evidence provided

Everything below is run over SSH, because there is no working kubectl and will not be one until the incident is over.

$ for n in cp-1 cp-2 cp-3; do printf '%s ' $n; ssh $n 'sudo ss -lntp | grep -c :6443'; done

cp-1: 0
cp-2: 0
cp-3: 0

$ sudo crictl ps -a | grep kube-apiserver

8f1c2d3e4a5b6  registry.k8s.io/kube-apiserver:v1.34.1  12 seconds ago  Exited (1)  9  kube-apiserver-cp-3

$ sudo crictl logs --tail=5 8f1c2d3e4a5b6

$ ls -l /etc/kubernetes/audit-policy.yaml

-rw-r--r-- 1 root root 812 Aug 17 21:34 /etc/kubernetes/audit-policy.yaml

$ grep -nE 'audit|volumeMounts|hostPath' /etc/kubernetes/manifests/kube-apiserver.yaml

23:    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
24:    - --audit-log-path=/var/log/kubernetes/audit.log
61:      volumeMounts:
62:      - mountPath: /etc/kubernetes/pki
71:      hostPath:
72:        path: /etc/kubernetes/pki

$ sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health --cluster

https://10.0.1.10:2379 is healthy: successfully committed proposal
https://10.0.1.11:2379 is healthy: successfully committed proposal
https://10.0.1.12:2379 is healthy: successfully committed proposal

$ sudo kubeadm certs check-expiration | awk 'NR==1 || /apiserver|etcd-server/'

Work the evidence before reading on

Four observations are in tension, and resolving that tension is the whole exercise.

1. Nothing is listening on 6443 on any node, yet the kubelet is healthy and the runtime is healthy. Which component is missing, and which component was supposed to start it?
2. The API server’s own log says a file does not exist. ls on the same host, seconds later, says it does. Both are telling the truth. What is different about where each one is standing?
3. etcd is healthy on all three members and the certificates are valid. What does that rule out, and just as importantly, what does the fact that etcd is reachable tell you about the network the network team is being asked to investigate?
4. The change was verified three times and passed three times, and the outage started after the third pass. What can kubectl get nodes observe, and what can it not?

Before continuing: name the check that would have failed at 21:06, fifteen minutes into a change that nobody knew had already broken a node.

Root cause

1. There is no API server process to be down

systemctl restart kube-apiserver failing with Unit not found is not evidence of a broken install. On a kubeadm cluster the API server, the controller-manager, the scheduler and etcd are all static Pods: YAML manifests in /etc/kubernetes/manifests/ that the kubelet reads directly and runs without the API server’s involvement. The kubelet is the only systemd unit in the picture.

That is also why the kubelet looks healthy. It is doing exactly what it is told: read the manifest, start the container, watch it exit, back off, start it again. A kubelet whose static Pod crash-loops is a working kubelet.

2. The container cannot see a file the host can

The change added two flags to the manifest and copied audit-policy.yaml to each node. It did not add a volumes entry and a matching volumeMounts entry for it.

A static Pod is still a Pod. It runs in its own mount namespace and sees only the paths the manifest mounts into it — which, on the manifest kubeadm writes for the API server, is essentially /etc/kubernetes/pki and the host CA bundle directories. /etc/kubernetes/audit-policy.yaml is not among them, so from inside the container that path does not exist.

kube-apiserver reads the audit policy at startup and treats a policy file it cannot load as fatal. It logs the path, exits non-zero, and never binds to 6443. The kubelet restarts it into the same failure.

This is why ls and the API server disagree. They are not looking at the same filesystem.

3. Everything else stayed up, which is normal

The control plane is what changes state. The data path is kube-proxy rules, CNI datapath and the containers themselves, all of which keep running with no API server at all. So the product stays healthy while nothing new can be created, no Deployment rolls, no Pod is rescheduled and no node can join. That combination — healthy traffic, frozen cluster — is the signature of a control plane outage and it is worth recognising on sight, because it tells you where not to look.

The HAProxy Layer4 connection problem is the load balancer correctly reporting that nothing accepted its TCP connection. It is a symptom of the backend being absent, not of the network being broken, and etcdctl reaching all three members over the same fabric a minute later is the cheap proof of that.

Resolution

1. Take a copy of the manifest before touching it: sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.incident. There is no API server and no version history to recover it from, and the file you are about to hand-edit is the only one on the node.
2. Pick one node and fix only that node. The fastest safe repair is to delete the two audit flags, because it needs no new YAML structure under pressure. Adding the volume and volumeMount is the other option and is correct, but it is four lines of indentation-sensitive YAML typed into a live control plane at 22:00.
3. Let the kubelet notice. It re-reads the manifest directory on its fileCheckFrequency, 20 seconds by default; you do not need to restart the kubelet and restarting it does not speed this up.
4. Verify locally before doing anything else: sudo ss -lntp | grep :6443 shows a listener, and curl -sk https://127.0.0.1:6443/livez returns ok. If it does not, read sudo crictl logs on the newest kube-apiserver container rather than editing again.
5. Confirm the cluster is back from a client. One healthy API server restores the whole control plane: kubectl works, the controllers reconnect, and the work queued since 21:35 starts draining. Watch kubectl get deployment for the scale-up that never happened.
6. Now repeat on the second node, verify it node-locally, then the third. One at a time, with the local check between each, is the same discipline the change should have used.
7. Confirm the load balancer agrees: the HAProxy stats page must show three backends UP, not one. A cluster running on a single API server has no redundancy left and is one bad edit from a second outage.
8. Decide the fate of the audit-logging change deliberately rather than by default. Reverting it is a legitimate end state for an overnight incident; hold it, give it an owner and a change window, and re-land it with the mount and a verification that reads the audit log rather than the ticket.

Verification

1. A listener exists on each repaired node: sudo ss -lntp | grep :6443 returns a kube-apiserver process. This is the check that was false on all three nodes, and it is node-local by construction.
2. curl -sk https://127.0.0.1:6443/livez returns ok on each control-plane node, run on the node itself. /livez answers whether the process is alive.
3. curl -sk https://127.0.0.1:6443/readyz?verbose returns 200 and no line begins with a minus sign. The verbose form names the individual failing check, which is the difference between "the API server is unhappy" and knowing which subsystem to look at.
4. The restart count is flat for ten minutes: sudo crictl ps sampled twice, ten minutes apart, shows the same container ID and the same start time. A crash-loop is indistinguishable from health in any single sample, and this is the check that can still fail.
5. HAProxy shows three backends UP. One is service restored; three is redundancy restored, and they are not the same claim.
6. The cluster is reconciling, not merely answering. The Deployment scaled at 21:40 has reached 12 ready replicas, and creating and deleting a throwaway Pod both succeed.
7. If the audit flags were kept rather than reverted, /var/log/kubernetes/audit.log exists on the host and is growing. That is the only check that distinguishes a mounted policy file from one the API server silently could not read — and an unreadable one is now fatal at startup rather than silent, which is the behaviour that caused this outage.

Prevention

• Verify per-node changes with per-node checks. A cluster-level command answered by any surviving member cannot see damage to one member. curl https://127.0.0.1:6443/livez on the node you just changed is the whole fix, and it costs one line in the runbook.
• Alert on the count of ready API server backends, not on reachability. “The API is reachable” is true at three of three and at one of three. Only the count moves when the first node breaks, and that is the alert that would have fired at 21:06.
• Treat static Pod manifests as code. Keep them in the configuration repository, review the diff, and diff them across nodes as a routine check. Drift between control-plane nodes is invisible from inside the cluster.
• Remember that a static Pod is a container. Any flag naming a path needs a hostPath volume and a volumeMount as well. The host shell is not a fair test of what the container can see.
• Copy the manifest before editing it. Once the API server is down there is no kubectl get pod -o yaml to recover the previous content from, and the manifest on disk is the only copy.
• Rehearse the no-kubectl state. Every runbook in the estate is written in kubectl, and none of them work during this incident. Knowing crictl ps -a, crictl logs, journalctl -u kubelet and the manifest layout before you need them is the difference between a twenty-minute outage and a two-hour one.