critical riskcluster affecting~45 min

Runbook: Investigate an API Server Outage

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the failure mode: kubectl cluster-info returns error, hangs, or 5xx
· Capture the API server response: curl -kv https://<api-vip>:6443/healthz --max-time 5
· Capture the API server endpoints on the load balancer: curl -kv https://<api-vip>:6443/api/v1/namespaces
· Capture every control-plane node API server status: kubectl get nodes -o wide and SSH to each control plane to check the API server pod
· Capture etcd health: see kubernetes-rb-investigate-etcd-health
· Capture the timestamps of recent client errors: kubectl get events -A --sort-by=.lastTimestamp | head -20

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm whether the failure is client-side or server-side: curl -kv https://<api-vip>:6443/healthz from outside the cluster
2If the load balancer is the bottleneck: bypass it and reach a control-plane node directly (https://<node-ip>:6443/healthz)
3For API server crashes: kubectl -n kube-system get pods -o wide -l component=kube-apiserver and read the logs
4For slow / unresponsive API server: read /metrics for request duration, inflight requests, and watch streams
5For etcd backend slowness: see kubernetes-rb-investigate-etcd-health
6For admission webhook failures: list MutatingWebhookConfiguration and ValidatingWebhookConfiguration and confirm they are reachable
7For certificate errors: capture the exact x509 message and identify which cert is failing
8Apply the smallest fix: restart the API server pod (if a crash), repair the load balancer (if a VIP issue), remove a misbehaving webhook (if a webhook storm), restore the etcd cluster
9Verify the API server responds: kubectl cluster-info, kubectl get nodes, and an end-to-end workload test

4 · Verification

Confirm the procedure actually fixed the problem.

✓curl -kv https://<api-vip>:6443/healthz returns ok
✓kubectl cluster-info returns both the control plane and CoreDNS URLs
✓kubectl get nodes returns every node Ready
✓kubectl get pods -A returns all namespaces
✓API server logs show no errors in the last 5 minutes
✓Watch streams resume: kubectl get pods -A --watch (start and stop within seconds)
✓Admission webhooks respond: kubectl get validatingwebhookconfiguration -o yaml | head shows reachable webhook URLs
✓etcd cluster is healthy: see kubernetes-rb-investigate-etcd-health

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If restarting the API server pod made things worse, the kubelet on the control-plane node may have to be restarted to recover
↶If an admission webhook fix broke a workload, restore the previous webhook configuration via Git and kubectl apply
↶If a load-balancer fix made things worse, return to the previous VIP configuration
↶Capture the API server logs, the etcd logs, and the LB logs before any rollback that may lose them
↶For a cluster-wide outage, escalate to the kubernetes-rb-recover-failed-control-plane runbook

6 · Escalation

When the runbook isn't enough, contact:

· API server OOMKilled: a workload is sending huge objects; identify and remove the offending client
· etcd is the bottleneck and the cluster has quorum loss: see kubernetes-rb-investigate-etcd-health and the etcd incident runbook
· API server fails to start with a certificate error: see kubernetes-rb-renew-cluster-certs
· HA load balancer reports zero healthy backends: every control-plane API server is down; escalate to the kubernetes-rb-recover-failed-control-plane runbook
· Watch streams failing across the cluster: API server cannot keep up; investigate inflight request count and admission webhook latency

An API server outage stops everything: no scheduling, no Pod creation, no updates, no reads. The urgency is high but the cause is in a small set of places: the API server itself, the load balancer in front of it, etcd behind it, or the admission webhooks intercepting requests.

1. Confirm the failure mode

Read-only / SafeConfirm the failure mode


curl -kv https://<api-vip>:6443/healthz --max-time 5
curl -kv https://<api-vip>:6443/api --max-time 5

# From each control-plane node, bypassing the LB
for n in <cp1> <cp2> <cp3>; do
echo "=== $n ==="
curl -kvs https://$n:6443/healthz --max-time 5 2>&1 | grep -E 'HTTP/|ok'
done

# Inspect API server pods (this may itself fail if the API is down)
kubectl -n kube-system get pods -l component=kube-apiserver -o wide 2>&1 | head

If the LB health check fails but direct control-plane connections succeed, the LB is the problem. If every control plane fails, the API server or its backend is.

2. Read the API server logs

Read-only / SafeRead the API server logs


ssh <cp1> -- sudo crictl ps -a | grep kube-apiserver
ssh <cp1> -- sudo crictl logs <apiserver-container-id> --tail=200
ssh <cp1> -- sudo journalctl -u kubelet --since "10 min ago" | grep -i apiserver | tail

The API server is a static pod managed by kubelet; its manifest lives in /etc/kubernetes/manifests/kube-apiserver.yaml and its logs go to both the container and the kubelet journal.

3. Inspect API server metrics

Read-only / SafeInspect API server metrics


ssh <cp1> -- sudo curl -k https://localhost:6443/metrics | grep -E "apiserver_request_total|apiserver_inflight|apiserver_admission_webhook_admission_duration_seconds' | head -30

# Look for:
# - apiserver_inflight_requests high in one direction (read or write)
# - apiserver_admission_webhook_admission_duration_seconds high
# - apiserver_etcd_object_counts high in one resource type

A high admission webhook latency is the most common cause of an API server that is slow but not crashed. A high etcd object count for one resource type is a controller or client stuck in a write loop.

4. Admission webhook failures

Read-only / SafeAdmission webhook failures


.items[] | {name: .metadata.name, url: (.webhooks[].clientConfig.url // .webhooks[].clientConfig.service)}"
kubectl get validatingwebhookconfigurations -o json | jq -r '.items[] | {name: .metadata.name, url: (.webhooks[].clientConfig.url // .webhooks[].clientConfig.service)}'

# From a control-plane node, can the API server reach the webhook?
ssh <cp1> -- sudo curl -k --max-time 5 https://<webhook-url>/healthz || echo "webhook unreachable"

A misconfigured or unreachable webhook is a frequent cause. Removing or fixing the webhook (carefully) restores the API server.

5. etcd backend

See kubernetes-rb-investigate-etcd-health. The API server cannot serve reads or writes if etcd is unavailable, slow, or has lost quorum.

6. HA load balancer

Read-only / SafeHA load balancer


ssh <lb-node> -- sudo cat /etc/haproxy/haproxy.cfg 2>/dev/null | grep -A10 "kube-apiserver'
ssh <lb-node> -- sudo systemctl status haproxy
ssh <lb-node> -- sudo curl -k https://<lb-vip>:6443/healthz --max-time 5

# Or MetalLB / cloud LB - check the service
kubectl get svc -n kube-system | grep -i kube-apiserver
# For cloud LB, check the LB health in the cloud console

A misconfigured LB health check pointing at /api/v1/namespaces may report healthy when the API server is not. Point the check at /healthz and /readyz instead.

7. Apply the fix

Read-only / SafeApply the fix


ssh <cp1> -- sudo crictl stop <apiserver-container-id>
sleep 30
ssh <cp1> -- sudo crictl ps -a | grep kube-apiserver

# B. Edit the static pod manifest (rarely needed)
ssh <cp1> -- sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
# Kubelet restarts the pod automatically; give it 30 seconds.

# C. Disable a misbehaving webhook (extreme caution)
kubectl delete mutatingwebhookconfiguration <name> --wait=false
# Or set failurePolicy: Ignore in the webhook config

# D. Repair the load balancer
ssh <lb-node> -- sudo vim /etc/haproxy/haproxy.cfg
ssh <lb-node> -- sudo systemctl reload haproxy

8. Verify

Read-only / SafeVerify


kubectl get nodes -o wide
kubectl get pods -A | wc -l

# Confirm watches work
kubectl get pods -A --watch --request-timeout=5s &
WATCH_PID=$!
sleep 5
kill $WATCH_PID 2>/dev/null || true
echo "watch succeeded"

Common pitfalls

Symptom	Cause	Action
API server OOMKilled	A client sending huge objects	Identify and stop the client
API server slow, no errors	Admission webhook timeout	See step 4
API server up, etcd slow	etcd issue	See `kubernetes-rb-investigate-etcd-health`
LB shows healthy but API server errors	LB health check on the wrong endpoint	Point LB at `/healthz` and `/readyz`
API server cannot start (manifest error)	Recent edit broke the manifest	Restore from backup; verify with `kube-apiserver --validate-config`

An API server outage is treated as a cluster-affecting incident. Capture the logs, identify the layer that is failing, fix that layer, verify, then communicate. Do not improvise.