Kubernetes control plane and worker architecture at a glance

A Kubernetes cluster is not a single binary. It is a federation of components, each with a single responsibility, communicating via gRPC and HTTP, persisting state in etcd, and trusting the network approximately. This lesson walks the full set of components and traces a Pod creation request through every one of them.

The two halves: control plane and workers

A cluster has two functional halves:

flowchart TB
    subgraph CP["Control plane"]
        API[API server<br/>kube-apiserver]
        ETCD[etcd cluster]
        SCH[Scheduler<br/>kube-scheduler]
        CM[Controller manager<br/>kube-controller-manager]
        CCM[Cloud controller manager<br/>cloud-controller-manager]
    end

    subgraph W["Worker nodes (1..N)"]
        KUBELET[kubelet]
        KP[kube-proxy / service dataplane]
        RUNTIME[Container runtime<br/>containerd / CRI-O]
        CNI[CNI plugin]
        CSI[CSI driver]
    end

    API <--> ETCD
    API --> SCH
    API --> CM
    API --> CCM
    KUBELET -->|CRI| RUNTIME
    KUBELET -->|CNI| CNI
    KUBELET -->|CSI| CSI
    KUBELET --> API
    KP --> API

The control plane runs the cluster’s “brain”: the API server, etcd, the scheduler, the controller manager. The workers run the workloads. In a production cluster, control-plane components typically run on dedicated nodes (or on managed infrastructure); workers run on the hosts that host the Pods.

Control-plane components

API server (kube-apiserver)

The front door of the cluster. Every read and every write — from kubectl, from controllers, from kubelet, from webhooks — flows through the API server. The API server:

• Authenticates the request (client cert, bearer token, OIDC, ServiceAccount token).
• Authorizes the request (RBAC, Node authorizer, webhook).
• Runs admission control (mutating then validating admission, including built-in Pod Security Standards).
• Persists the object to etcd (every read of an object reads from etcd through the API server’s cache).
• Returns the response.

The API server is the only component that talks to etcd directly. Every other component goes through the API server.

etcd

A consistent, distributed key-value store. Stores the entire cluster state: every Pod, Service, Deployment, ConfigMap, Secret, RBAC binding, lease — the full API object graph.

flowchart LR
    API[API server] -->|writes go to majority| L[etcd leader]
    L --> F1[follower]
    L --> F2[follower]
    F1 <-.-> F2

etcd uses Raft consensus. A 3-member cluster tolerates 1 member loss; a 5-member cluster tolerates 2. Quorum loss halts writes. The etcd cluster’s failure domain is independent of the worker nodes’ failure domain — losing workers does not lose cluster state, but losing etcd members does.

Scheduler (kube-scheduler)

Watches for Pods that have spec.nodeName unset (i.e., are unscheduled) and assigns them to nodes based on:

• Filters — does this node have enough resources? Does the Pod fit the node’s taints, nodeSelector, affinity rules?
• Scoring — among the feasible nodes, which is “best” by spread, inter-Pod affinity, image locality, etc.?
• Binding — write the chosen node into the Pod’s spec.nodeName.

The scheduler is a control loop: it runs continuously, picking up unscheduled Pods and binding them. It does not start Pods; that is kubelet’s job.

Controller manager (kube-controller-manager)

Runs the core controllers: Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, Service, Endpoints, Node, Namespace, ServiceAccount, PersistentVolume, and many more. Each is a reconcile loop: observe the object, diff against the desired state, act.

The controller manager is one binary but many loops. A failure of the controller manager stalls every controller’s reconcile; the cluster’s spec-vs-status gap will grow until the manager recovers.

Cloud controller manager (cloud-controller-manager)

A separate binary that runs only the cloud-specific controllers: Node lifecycle (cloud knows when a VM is gone), LoadBalancer Service (cloud provisions the LB), Route (cloud updates the route table), PV zone labels (cloud knows the zone of a disk).

The cloud controller manager decouples cloud-specific logic from the upstream controllers, so the same kube-controller- manager binary runs on every distribution.

Worker components

kubelet

The agent on every worker node. Kubelet:

• Watches the API server for Pods assigned to its node (spec.nodeName == <this-node>).
• Pulls images via the runtime (CRI).
• Mounts volumes via CSI.
• Sets up networking via CNI.
• Starts containers via the runtime.
• Runs probes (liveness, readiness, startup).
• Reports status back to the API server.
• Renews the node lease every few seconds.

Kubelet is the only worker component that talks to the API server directly. It is also the most common source of node-level incidents (cert expiry, disk pressure, runtime hang).

kube-proxy / service dataplane

Implements the Service virtual IP and the iptables / IPVS / eBPF rules that route Service traffic to Pods. On every node, kube-proxy (or its replacement) watches the API server for Service and EndpointSlice objects and programs the local node’s dataplane.

flowchart LR
    S[Service: ClusterIP 10.96.0.10:80] --> E[EndpointSlice: Pod A, Pod B, Pod C]
    E -->|kube-proxy programs| IPT[iptables / IPVS / eBPF rules on each node]
    IPT --> P1[Pod A 10.244.1.5]
    IPT --> P2[Pod B 10.244.2.7]
    IPT --> P3[Pod C 10.244.3.2]

Modern clusters (Cilium, Calico eBPF) may replace kube-proxy with a CNI-managed dataplane, but the function — “make Service-to-Pod traffic work” — remains.

Container runtime

Already covered in Part I (kubernetes-i-04). The CRI shim and runtime that kubelet talks to. containerd is the default.

CNI plugin

The networking plugin that connects Pods. CNI is invoked by kubelet when a Pod is created and is responsible for:

• Allocating a Pod IP (from the cluster CIDR)
• Creating the veth pair between the Pod’s netns and the host
• Programming routes on the host (so the Pod is reachable from other nodes)
• Returning the result to kubelet

Common CNIs: Cilium, Calico, Flannel. Each has different features (NetworkPolicy enforcement, eBPF dataplane, BGP).

CSI driver

The storage plugin. CSI is invoked by kubelet (for mount) and by the controller manager (for provision/attach). The driver:

• Creates volumes (in response to a PVC) by talking to the storage backend
• Attaches the volume to the node that the Pod is scheduled on
• Mounts the volume into the Pod’s filesystem

Common CSIs: cloud-provider drivers (EBS, GCE PD, Azure Disk), on-prem drivers (Ceph, NetApp, PowerStore).

A Pod creation, end to end

sequenceDiagram
    autonumber
    participant U as User (kubectl)
    participant API as API server
    participant ETCD as etcd
    participant SCH as Scheduler
    participant CM as Controller manager
    participant K as kubelet (worker)
    participant R as Runtime
    participant N as CNI

    U->>API: POST /api/v1/namespaces/prod/pods (Pod manifest)
    API->>API: Authn, Authz, Admission
    API->>ETCD: persist Pod (resourceVersion=1)
    ETCD-->>API: committed
    API-->>U: 201 Created

    API-->>SCH: watch: Pod ADDED (nodeName unset)
    SCH->>API: GET Pod + nodes
    SCH->>SCH: Filter, score, choose node
    SCH->>API: PUT /pods/web/binding (nodeName=worker-04)
    API->>ETCD: persist (resourceVersion=2)
    API-->>K: watch: Pod ADDED on node worker-04

    K->>R: CRI PullImage
    R-->>K: image ready
    K->>N: CNI ADD (allocate Pod IP, set up veth)
    N-->>K: IP assigned
    K->>R: CRI CreateContainer, StartContainer
    R-->>K: container PID
    K->>API: PATCH /pods/web/status (Running)

    Note over CM,API: ReplicaSet controller observes Pod count
    Note over API,ETCD: API server persists every status change

Every line in that diagram is a real RPC. The end-to-end latency is the sum of every component’s contribution. Production benchmarking of “how long from kubectl apply to Pod Running” includes API server admission, etcd commit, scheduler cycle, binding commit, kubelet watch delivery, image pull time, CNI ADD, container start.

Failure domains of each component

The control plane’s failure domain is the whole cluster; the worker’s failure domain is one node. This asymmetry explains why production clusters spend so much effort on control-plane HA and so much less on per-worker resilience.

Control-plane HA topologies

Production control-plane HA looks like:

flowchart TB
    LB[Load balancer / VIP] --> A1[API server 1]
    LB --> A2[API server 2]
    LB --> A3[API server 3]
    A1 --> E1[etcd member 1]
    A2 --> E2[etcd member 2]
    A3 --> E3[etcd member 3]
    S1[Scheduler replica 1<br/>leader] --> API
    S2[Scheduler replica 2] --> API
    C1[Controller manager<br/>leader] --> API
    C2[Controller manager<br/>replica] --> API

• API server: stateless behind a load balancer; run 2-3 replicas.
• etcd: 3 or 5 members with quorum; odd-numbered; spread across failure domains.
• Scheduler: leader-elected (via lease); run 2 replicas so failover is instant.
• Controller manager: leader-elected; run 2 replicas.
• Cloud controller manager: same.

The kubelet connects to the API server via the load balancer; if the load balancer fails, kubelets cannot renew leases, and the cluster will eventually mark every node NotReady. The load balancer is therefore a critical dependency.

How to inspect the architecture of a running cluster

kubectl get nodes -o wide
kubectl get pods -n kube-system -o wide

kubectl cluster-info

# See the API server endpoints
kubectl get endpoints kubernetes -n default

# On a control-plane node, see the running components.
# There is no systemctl step here: kubeadm runs the API server,
# controller-manager, scheduler and etcd as static Pods, so
# `systemctl status kube-apiserver` returns
# "Unit kube-apiserver.service could not be found".
ls /etc/kubernetes/manifests/
crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'

# On a worker, see kubelet and runtime
systemctl status kubelet containerd
journalctl -u kubelet --since "5 min ago" | head -50

Cross-course references

• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives the runtime uses; the architecture on top is what this lesson teaches.
• The Linux course part XIX-Linux-NetFoundations covers the network primitives the CNI composes.
• The Observability course part V-Observability-PromArchitecture covers the monitoring model that observes each component’s health.
• The Linux course part XXIV-Linux-Time covers chrony — every control-plane component depends on clocks being within tolerance.

Quiz

Production discipline

• Treat the control plane as the cluster’s central nervous system: HA, monitoring, and backup apply first to it.
• Distinguish control-plane HA (multi-replica API servers, etcd quorum, scheduler/controller leader election) from worker resilience (kubelet restart, node replacement, Pod rescheduling). The architectures are different.
• Monitor every component’s health independently. A scheduler that is reachable but not Ready has different operational impact than one that is not reachable.
• Document the component dependencies: API server needs etcd; kubelet needs API server; CNI needs API server; CSI needs API server. A failure in one cascades to the dependent.