API server role — the cluster gateway and only writer to etcd

The kube-apiserver is the front door of a Kubernetes cluster. Every kubectl, every controller, every kubelet, every webhook — all of them go through the API server. The API server is also the only writer to etcd that the API contract supports. This lesson walks the API server’s role, the request lifecycle, and the design consequences of the API server’s central position.

The API server in one sentence

The API server is the cluster’s HTTP API endpoint, backed by etcd for state, that every component of the Kubernetes control plane and every user of the cluster accesses for both reads and writes.

flowchart LR
    k[kubectl] -->|https| AS[API server]
    C[controller-manager] -->|https| AS
    S[scheduler] -->|https| AS
    K[kubelet] -->|https| AS
    KX[kube-proxy] -->|https| AS
    AW[admission webhook] -->|https| AS
    AS -->|https| E[etcd]
    AS -->|watch| C
    AS -->|watch| S
    AS -->|watch| K
    AS -->|watch| KX

The only writer property

The cluster’s API contract:

The API server is the only Kubernetes component that writes to etcd through the standard API. Any other component that needs to write must go through the API server.

This property has consequences:

• Mutations flow through the API server. A kubelet that needs to update a Pod’s status submits a PATCH to the API server; the API server validates, persists to etcd, and watches fan out.
• The API server is the only path. No component should connect to etcd directly except for debugging or recovery. Even the kubeadm-managed etcd’s static pod is not special — it’s just a static pod for the local member.
• Authentication and authorisation happen on every request. Because all writes go through the API server, RBAC, audit, and admission controls cover them centrally.

What the API server does

A request flow:

sequenceDiagram
    autonumber
    participant Client
    participant AS as API server
    participant AW as Admission webhook
    participant E as etcd
    Client->>AS: PUT /api/v1/namespaces/prod/pods/web
    AS->>AS: authenticate (TLS cert / token)
    AS->>AS: authorise (RBAC evaluation)
    AS->>AW: admission (mutating)
    AW-->>AS: mutated object
    AS->>AW: admission (validating)
    AW-->>AS: allow
    AS->>AS: schema validation
    AS->>AS: optimistic concurrency (resourceVersion)
    AS->>E: txn If(...){put}
    E-->>AS: committed
    AS->>AS: audit log
    AS-->>Client: 200 OK

The steps are:

1. TLS termination.
2. Authentication (cert, token, OIDC).
3. Authorisation (RBAC; Node authoriser).
4. Mutating admission (webhooks; built-in mutation like defaulting).
5. Object schema validation.
6. Optimistic concurrency (resourceVersion check).
7. Validating admission (webhooks; PodSecurity).
8. Encryption at rest (if configured).
9. etcd write through gRPC.
10. Audit log (if configured).
11. Watch fan-out.
12. Response.

Each step is a place failure can occur. The production discipline is that each step is well-understood and monitored.

What the API server does NOT do

• It does not schedule. The scheduler reads Pods and writes bindings; the API server persists the results but does not choose nodes.
• It does not run controllers. The controller-manager runs the controllers (Deployment, ReplicaSet, EndpointSlice, etc.); the API server records their state changes.
• It does not run kubelets. The kubelet runs on each node and manages containers.
• It does not run Pods. Pods run on nodes, not on the control plane.

The API server is a state machine with an HTTP interface. It does not initiate work; it serves requests and persists state. The work happens because controllers (running in the controller-manager and as custom operators) read state and reconcile.

The watch model

Every API request that creates, updates, or deletes an object generates a watch event. Other components watch the API server for events and reconcile.

flowchart LR
    AS[API server] -->|watch| IC[Informer cache in controller]
    AS -->|watch| KIC[Kubelet informer]
    AS -->|watch| SIC[Scheduler informer]
    IC -->|reconcile| C[Controllers]
    KIC -->|Pod spec| K[Kubelet]
    SIC -->|filter/score| SCH[Scheduler]

This is the observer pattern in distributed systems: the API server is the publisher; the controllers and kubelets are the subscribers. The watch is the event stream.

The watch cache

The API server maintains a coherent, in-memory snapshot of cluster state. Reads (LIST) are served from this snapshot for speed; writes update it.

flowchart LR
    E[etcd] -->|watch event| WC[Watch cache]
    WC -->|serve LIST| R[Reader]
    E -->|serve Txn| W[Writer]

The watch cache is what makes kubectl get fast on clusters with millions of objects. Without it, every LIST would round-trip to etcd.

The API surface

The API server exposes:

• The built-in Kubernetes API groups (apps/v1, core/v1, batch/v1, networking.k8s.io/v1, etc.).
• Aggregated APIs (see next lesson) — metrics.k8s.io, custom.metrics.k8s.io, etc.
• The OpenAPI/Swagger spec at /openapi/v2 and /openapi/v3.
• Discovery at /api, /apis, /api/v1.
• Health checks at /livez, /readyz, /healthz.
• Subject access review at /subjectaccessreview.

The OpenAPI spec is the source of truth for the API’s schema; clients like kubectl use it to generate type-safe clients.

The failure modes

The multi-API-server HA topology

Production clusters run multiple API server instances:

flowchart LR
    LB[Load balancer] --> AS1[API server cp-1]
    LB --> AS2[API server cp-2]
    LB --> AS3[API server cp-3]
    AS1 -->|write| E[etcd]
    AS2 -->|write| E
    AS3 -->|write| E
    E -->|watch| AS1
    E -->|watch| AS2
    E -->|watch| AS3

Each API server is independent; each has its own in-memory cache; each connects to etcd through the same client URL. The watch feed is consistent across instances because etcd serialises writes.

A failure of one API server instance is invisible to clients (the load balancer routes around it). A cluster- wide failure (all 3 API servers down) makes the cluster unreachable.

Quiz

Production discipline

• Route every change through the API server. Direct etcd access is debugging; production writes go through the API.
• Monitor API server latency. The p99 tail is the cluster’s tail latency.
• Run multiple API server instances. A single instance is a single point of failure; HA is at least 2.
• Validate watch consistency. A controller that fails to reconcile is a controller watching at a stale resourceVersion.
• Audit every write. The audit log is the record of who changed what, when, why.

The API server is the cluster’s front door and the only legitimate writer to etcd. Operating it well is operating the cluster.