The API server — the front door of the cluster

The Kubernetes API server (kube-apiserver) is the single front door of the cluster. Every kubectl command, every controller’s reconcile loop, every kubelet’s status update flows through it. This lesson covers what the API server does on each request, how it serves watch streams, how it persists state, and how production failures manifest at this layer.

The request pipeline

Every HTTP/gRPC request to the API server traverses the same pipeline:

flowchart LR
    R[Request<br/>HTTP/gRPC] --> TLS[TLS termination]
    TLS --> AUTHN[Authentication]
    AUTHN --> AUTHZ[Authorization]
    AUTHZ --> MUT[Mutating Admission]
    MUT --> VAL[Validating Admission]
    VAL --> SCHEMA[Schema validation<br/>OpenAPI]
    SCHEMA --> CONV[Conversion<br/>multi-version]
    CONV --> ETCD[etcd write]
    ETCD --> RESP[Response]

A POST /api/v1/namespaces/prod/pods for a new Pod walks every step in order. A GET /api/v1/pods walks the read path, which skips the persistence step.

Step 1: TLS termination

The API server terminates client TLS by default. The server certificate is the same one the cluster’s root CA signs; production clusters rotate it via kubeadm cert renew. Client certificates (for kubectl) or bearer tokens (for ServiceAccount, OIDC) are presented at this step.

A TLS handshake failure surfaces as kubectl: error: unable to connect to server: x509: certificate has expired or is not yet valid. Diagnose with openssl s_client -connect <api>:<port> -showcerts from a node that has the kubelet’s CA bundle.

Step 2: Authentication

The API server tries every configured authenticator in turn until one returns a known identity. Authenticators:

• X.509 client certificates — common for humans and ServiceAccounts.
• Bearer tokens — for ServiceAccounts (projected tokens in 1.24+) and OIDC.
• HTTP basic auth — deprecated; not for production.
• Webhook token authentication — delegate to an external service (e.g., OIDC IdP).

If no authenticator returns a known identity, the request is rejected with 401 Unauthorized. If an authenticator returns “this looks like a token but it does not match anything”, the request is rejected with 403 Forbidden.

Step 3: Authorization

Once authenticated, the API server asks: is this identity allowed to perform this verb on this resource?

Authorizers (in order):

• Node — special-case authoriser for kubelet; allows kubelets to write Pods and Node status for their own node only.
• RBAC — Role/ClusterRole + RoleBinding/ClusterRoleBinding. Default in production.
• Webhook — delegate to an external authoriser (e.g., OPA, Cerbos).
• ABAC — attribute-based; deprecated; rarely used.
• AlwaysAllow / AlwaysDeny — testing only.

If every authoriser denies, the request is rejected with 403 Forbidden. The decision is logged in the audit log (if configured).

Step 4: Mutating admission

Mutating webhooks run before schema validation. They can modify the object (add labels, default fields, inject sidecars). Order: built-in mutators (e.g., namespace defaulting, ServiceAccount token projection), then configured mutating webhooks in the order they are registered.

Failure mode: a webhook that returns an error rejects the request. A webhook that times out is treated as a failure (unless failurePolicy: Ignore is set, which is risky).

Step 5: Validating admission

Validating webhooks run after schema validation. They can only accept or reject; they cannot modify. Order: built-in validators (e.g., Pod Security Standards, conflict detection), then configured validating webhooks.

Production validating admission policies include:

• Pod Security Standards (restricted, baseline, privileged)
• Image registry allowlists (deny latest, deny unknown registries)
• Resource quota enforcement (also a controller)
• Custom policies (OPA/Gatekeeper, Kyverno)

Step 6: Schema validation

The OpenAPI schema for the object’s apiVersion/kind is checked. A field that does not match the schema rejects the request. This is where a typo in spec.replcas: 5 would be caught.

Step 7: Conversion

Internal storage is at the API server’s internal version (often v1 for core, storage.k8s.io/v1 for storage). On write, the API server converts the request to internal; on read, it converts from internal to the requested version. This is how the same etcd key serves multiple API versions.

Step 8: Persistence

The object is encoded (JSON or protobuf) and written to etcd under /registry/<kind>/<namespace>/<name> (namespaced) or /registry/<kind>/<name> (cluster-scoped). The etcd write is a Raft commit — it returns only after a majority of etcd members have accepted it.

The object’s resourceVersion is the etcd revision at which it was written. This is how optimistic concurrency works: subsequent writes must include the resourceVersion they expect to overwrite, or they fail with 409 Conflict.

Watch semantics

A kubectl get pods -w (or a controller’s informer) opens a watch on the API server. Watch semantics:

• The server returns the current state (a list of objects) immediately.
• It then streams every change to those objects, in order, as watch events (ADDED, MODIFIED, DELETED).
• Each event includes the new object’s resourceVersion and the previous resourceVersion (for MODIFIED/DELETED).

sequenceDiagram
    autonumber
    participant C as Client (informer)
    participant API as API server
    participant E as etcd

    C->>API: GET /pods?watch=true&resourceVersion=...
    API->>E: list from resourceVersion
    E-->>API: snapshot
    API-->>C: 200 OK + initial snapshot
    loop Watch stream
        E-->>API: object changed
        API-->>C: watch event {type, object}
    end
    Note over C,API: connection breaks / 410 Gone
    C->>API: re-list from last known resourceVersion
    Note over C,API: resync

Failure modes:

• Connection drops. The client transparently re-lists from the last known resourceVersion. The API server returns the missing events.
• 410 Gone. The client’s resourceVersion is too old for the server’s watch cache. The client must do a full re-list without a resourceVersion and re-sync.
• Slow consumers. The API server closes a watch stream if the client cannot keep up. The client must reconnect.

Watch is the primary mechanism controllers use to observe state. A broken watch makes a controller’s reconcile loop work but slower (it falls back to periodic re-listing).

Storage and the cache

The API server does not query etcd on every read. It maintains an in-memory cache (cache.go) that mirrors etcd state. Reads hit the cache; writes go through to etcd.

flowchart LR
    API[API server process] --> C[Watch cache<br/>in-memory]
    C <-.->|watch events| ETCD[etcd]
    API -->|write| ETCD

The cache size and watch latency are bounded by the API server’s memory and the rate of change in etcd. Production API servers monitor:

• apiserver_storage_objects — count by resource
• apiserver_watch_duration_seconds — watch latency
• apiserver_cache_list_size — list response cache size

A cache that is too large for memory will cause the API server to be OOMKilled. A cache that lags behind etcd will serve stale reads briefly. Production sizing accounts for the peak number of objects in the cluster.

Aggregation layer

The API server supports the aggregation layer: an extension that lets third-party API servers (e.g., metrics-server, cert-manager, custom CRDs) be served under the same https://<apiserver>/apis/<group>/<version>/... URL space.

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  groupPriorityMinimum: 100
  versionPriority: 15

The API server proxies the request to the backing service when a client calls /apis/metrics.k8s.io/v1beta1/.... This is how kubectl top and HPA get their data without each component running its own server.

Aggregation failure modes:

• A misconfigured APIService causes the URL to return 503.
• The backing service’s certificate must be trusted by the API server (front-proxy CA).
• Latency of the backing service is added to every request through that APIService.

API server flags every operator must understand

Production kubeadm-managed API servers run with flags that shape performance and reliability:

The --max-requests-inflight throttle protects etcd from write storms. When exceeded, requests are rejected with 429 Too Many Requests. Production clients (controllers, kubectl) retry on 429 with backoff.

How to inspect the API server

kubectl get --raw /healthz

ok

kubectl get --raw /readyz

ok

kubectl get --raw /metrics | grep apiserver_

# From a control-plane node
journalctl -u kube-apiserver --since "10 min ago" | grep -i "audit\|reject\|warn"

# Substitute your own etcd endpoint:
ETCD_ENDPOINT=https://192.0.2.11:2379

# Check the etcd write rate (write to etcd = write to API server)
etcdctl --endpoints="$ETCD_ENDPOINT" endpoint status --write-out=json

Cross-course references

• The Linux course part XIX-Linux-NetFoundations covers the network primitives the API server depends on (TLS, routing).
• The Observability course part V-Observability-PromArchitecture covers the metrics the operator monitors from the API server.
• The Linux course part XXIV-Linux-Time covers chrony — every API server certificate depends on clocks within tolerance.
• The Docker course part XXVII-Docker-Install covers TLS and certificate management in the same way the API server uses them.

Quiz

Production discipline

• Treat the API server as a single funnel: every cluster change goes through it. Capacity planning for the API server is capacity planning for the entire cluster’s change rate.
• Monitor write latency, throttle rate, and watch cache size. These predict outages.
• Tune --max-requests-inflight and --max-mutating-requests-inflight to the cluster’s change rate and etcd’s commit latency.
• Audit the authentication and authorisation chain at every manifest change. A change that introduces a new authoriser or authentication method must be reviewed for blast radius.
• Back up etcd (Part LXVI) before any API server change that touches storage paths, admission policies, or RBAC.