API server health and 503s — the cluster's front door

When kubectl get pods answers with ServiceUnavailable, the cluster has not stopped — running Pods keep serving traffic — but every control loop that would repair it has. Deployments do not roll, the scheduler places nothing, and your own tooling is talking to the component that is down, so the investigation moves to SSH and journalctl on a control-plane node. The 503 itself does not say why: a starting API server, an unreachable etcd, an expired serving certificate, and a failing authentication webhook all produce it, and each needs a different repair.

The API server’s role

The API server is the cluster’s HTTP API. Every cluster operation (kubectl, controllers, kubelet) goes through the API server. The API server is the cluster’s front door.

flowchart LR
    A[kubectl] --> B[API server]
    C[kubelet] --> B
    D[controllers] --> B
    E[scheduler] --> B
    B --> F[etcd]

A failure of the API server is a cluster-wide failure.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
CP_NODE=cp-01.example.com

# 1. Check the API server's health
kubectl get --raw /healthz
kubectl get --raw /readyz

# 2. Check the API server's logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200

# 3. Check the API server's Endpoints object.
# It is named kubernetes and it lives in the default namespace,
# not in kube-system.
kubectl get endpoints kubernetes -n default

# 4. Check the API server's process
ssh "$CP_NODE" "ps aux | grep kube-apiserver"

# 5. Check the etcd
ssh "$CP_NODE" "ETCDCTL_API=3 etcdctl endpoint health"

The diagnostic is the API server’s health, the logs, the endpoints, and the etcd.

503 Service Unavailable

A 503 Service Unavailable response from the API server is the API server’s signal that it is not ready. The common causes:

• API server is starting. The API server is in the process of starting up.
• etcd is unreachable. The API server cannot reach etcd.
• Authentication is failing. The API server cannot authenticate clients.
• Authorization is failing. The API server cannot authorize requests.

flowchart TD
    A[503 response] --> B{API server starting?}
    B -->|Yes| C[Wait for startup]
    B -->|No| D{etcd reachable?}
    D---|No| E[Fix etcd]
    D---|Yes| F{Auth OK?}
    F -->|No| G[Fix auth]
    F -->|Yes| H{Authz OK?}
    H -->|No| I[Fix authz]
    H -->|Yes| J[Unknown]

The diagnostic is the cause.

The 503 response

$ kubectl get pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request

The 503 is the API server’s response.

Common failures

• API server is down. The API server container has crashed. The remediation is to restart the static Pod by moving its manifest aside and back.
• etcd is down. The etcd cluster is unreachable. The remediation is to recover etcd (see Part CXXVI).
• Certificate expired. The API server’s certificate is expired. The remediation is to renew the certificate.
• Network partition. The API server cannot reach the etcd. The remediation is to fix the network.

The remediation

The remediation depends on the cause:

# Option 1: Restart the API server.
# kubeadm runs it as a static Pod, so move the manifest out of
# the kubelet's watch directory and back again.
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Option 2: Recover the etcd
# (see Part CXXVI)

# Option 3: Renew the certificate
kubeadm certs renew apiserver

# Option 4: Fix the network
# (cluster-specific)

The remediation is the API server recovery.

Production discipline

A 503 from the API server is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The API server is the cluster’s front door; the remediation is the API server recovery.

• Check the API server’s health. The health is the cluster’s front door.
• Check the etcd. The etcd is the cluster’s state.
• Check the certificate. The certificate is the API server’s authentication.

Quiz