etcd as the Kubernetes database — what it stores, who writes, who reads

etcd is the only component in a Kubernetes cluster that holds persistent state. Every other piece of the control plane — the API server, the scheduler, the controller manager, the kubelet — is derived from what etcd has recorded. If a Deployment exists, it is because etcd recorded it; if a node is Ready, it is because etcd recorded the heartbeat; if a Secret is encrypted at rest, etcd recorded the ciphertext. This lesson establishes what etcd is, where it sits in the cluster, and the consequences of that placement.

What etcd stores

etcd is the source of truth for the Kubernetes API object graph. That graph includes:

• Every API object (Pod, Deployment, StatefulSet, DaemonSet, Job, CronJob, Service, Ingress, ConfigMap, Secret, ServiceAccount, Role, ClusterRole, RoleBinding, ClusterRoleBinding, NetworkPolicy, ResourceQuota, LimitRange, PersistentVolume, PersistentVolumeClaim, StorageClass, CustomResourceDefinition, every CRD instance, and so on).
• Cluster-scoped state: Node objects, leases, events (subject to garbage collection), the discovery Endpoints object in kube-public, and the kubernetes namespace.
• API server bookkeeping: lease objects (kube-node-lease, leader election leases), events, controllerrevision history, endpointslice, apiservices.

etcd is not the source of truth for:

• The contents of Secret volumes mounted into Pods (those live in the kubelet’s data directory on the node’s file system until the Pod terminates — etcd only holds the canonical Secret object).
• Application state in PVCs (etcd has the PVC/PV objects, not the data).
• The kubelet’s container runtime cache (only the desired Pod spec is in etcd; the running container state is in containerd).
• Workload identity tokens (those are projected into Pods; only the SA reference is in etcd).

flowchart LR
    A["kubectl get all -A"] -->|API call| B[API server]
    B -->|WatchCache / LIST| C[etcd]
    C -->|/registry/<kind>/...| B
    B -->|JSON / Table| A

Where etcd sits in the cluster

etcd runs as a small number of peer processes. In a kubeadm-built cluster, etcd is a Static Pod on every control-plane node; in a stacked topology (default for kubeadm) the etcd members and the API server co-exist on the same hosts. In an external etcd topology, the etcd cluster lives on dedicated hosts that the API server reaches over the network.

flowchart TB
    subgraph CP1["Control-plane node 1"]
        AS1[API server]
        E1["etcd member<br/>(static pod)"]
    end
    subgraph CP2["Control-plane node 2"]
        AS2[API server]
        E2["etcd member"]
    end
    subgraph CP3["Control-plane node 3"]
        AS3[API server]
        E3["etcd member"]
    end
    AS1 <--> E1
    AS2 <--> E2
    AS3 <--> E3
    E1 <-->|peer TLS| E2
    E2 <-->|peer TLS| E3
    E3 <-->|peer TLS| E1
    LB[Load balancer] --> AS1
    LB --> AS2
    LB --> AS3

The peer network uses mTLS with the certificates in /etc/kubernetes/pki/etcd/. The client network (API server to etcd) uses the same CA but separate server and client certificates. The two URLs are separate: peer traffic on 2380 and client traffic on 2379.

The write path

A typical write — kubectl apply -f deployment.yaml — goes through:

1. Authentication at the API server (the kubectl context’s client cert, token, or OIDC).
2. Authorisation (RBAC evaluation; apply requires create/update/patch on the resource).
3. Admission (mutating webhooks first, then validating webhooks, then built-in defaults like PodSecurityStandards).
4. Schema validation in the API server (openapi check).
5. Optimistic concurrency check (resourceVersion comparison).
6. Encryption at rest (if configured, applied here).
7. etcd write through the storage interface: a Txn to the etcd cluster.
8. etcd commit (Raft log replicated to a majority).
9. Acknowledgement back to the API server.
10. Watcher fan-out (informer caches across the control plane and kubelets pick up the new object via watch).

The operator-visible latency of step 7–8 is the commit latency that etcd exposes as a metric (etcd_disk_backend_commit_duration_seconds). Anything that slows commit (disk pressure, fsync latency, GC pause, network between members) slows every apply in the cluster.

$ etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table

+---------------------------+------------------+---------+---------+-----------+------------+----------+------------+--------------------+--------+
|         ENDPOINT          |        ID        | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DB SIZE | LEADER ID |
+---------------------------+------------------+---------+---------+-----------+------------+----------+------------+--------------------+--------+
| https://10.0.1.10:2379    |  c5e9a1b2...     |    true  |    false  |    17     |  41289312  |   41289312 |     0     |   1.4 GB    | c5e9a1b2... |
| https://10.0.1.11:2379    |  d7a1c8f3...     |   false  |    false  |    17     |  41289312  |   41289312 |     0     |   1.4 GB    | c5e9a1b2... |
| https://10.0.1.12:2379    |  e8b4d2a5...     |   false  |    false  |    17     |  41289312  |   41289312 |     0     |   1.6 GB    | c5e9a1b2... |
+---------------------------+------------------+---------+---------+-----------+------------+----------+------------+--------------------+--------+

DB SIZE is the member’s physical file size, not the cluster’s logical data. Defragmentation runs per member, so the member at 10.0.1.12 carries more free pages than its peers while holding exactly the same keys at the same revision.

The read path

A read (kubectl get) can be served from three places:

1. WatchCache in the API server (a coherent in-memory snapshot updated via watch) — the default for LIST.
2. etcd directly — used when the watch cache is cold or when consistency with etcd’s revision is required.
3. etcd linearised read — used when a resourceVersion is supplied and the API server must reflect a write that happened after that revision.

`

$ kubectl get pods -A

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-7d5c5b9b9d-abcde   1/1     Running   0          5d
kube-system   kube-apiserver-cp-1        1/1     Running   0          30d
kube-system   kube-proxy-xyz             1/1     Running   0          30d

The watch cache is why kubectl get is fast even on a cluster with millions of objects — the LIST is served from memory without touching etcd. Writes still pay the etcd commit latency.

Why etcd does not live on the workers

Two reasons, both operational:

• Latency isolation. etcd’s commit latency is bound by disk fsync. A worker node running many Pods has noisy disk, noisy CPU, and noisy GC. etcd next to Pod work inherits every GC pause and every disk spike as observability noise, and as commit-latency amplification on the cluster.
• Failure-domain separation. Cordon and drain are ordinary worker operations. If etcd ran on workers, every drain would risk removing a quorum member. Keeping etcd on dedicated control-plane hosts means worker maintenance cannot touch etcd at all.

The data model

etcd’s data model is a key-value store with three properties operators feel directly:

• Key ordering. Keys are sorted lexicographically and ranges are returned in order. Kubernetes uses this for efficient namespace-scoped LIST: /registry/pods/<ns>/ is a prefix range that yields every Pod in a namespace.
• Versioning. Every key carries a ModRevision (a global monotonic counter). Kubernetes exposes this as resourceVersion on every object.
• Transactions. A Txn is a list of comparisons and success/failure lists. The API server uses Txn for “create only if absent”, “delete if resourceVersion matches”, and “set with previous-key CAS” — optimistic concurrency in one round trip.

flowchart LR
    K[key: /registry/pods/prod/web] -->|ModRevision| R[41289312]
    K -->|Value| V[JSON: pod spec + status]
    K -->|Lease| L[object lease, optional]
    K -->|Version| V2[incremented per write]

The failure domain

What does “failure domain” mean for etcd? In production:

• Members are independent processes. Run them on different hosts; in kubeadm with stacked topology, each member is a static pod on its own control-plane host.
• Hosts are independent failure domains. Spread control planes across racks and AZs.
• Disk is the biggest single point of failure for an individual member. Use SSDs (low fsync latency), RAID, and full-disk encryption.
• Network between members must be low-latency (< 10 ms typical target) and reliable. Partition triggers leader election; sustained partition triggers quorum loss.

The operator’s mental model is that etcd is a stateful service that must be operated like one — capacity planned, snapshotted, monitored, restored. Everything else in the control plane is derived state that can be reconstructed by re-running controllers against etcd.

Quiz

Production discipline

• Treat etcd as a stateful service. Capacity-plan it for objects and write rate, not just disk. Snapshot on a schedule. Monitor fsync latency. Run a daily restore drill on a separate host.
• Pin etcd’s CPU and memory at the kernel level. A noisy neighbour on the etcd host turns into commit-latency amplification for the entire cluster. Pin or isolate.
• Place etcd on dedicated failure domains. Different hosts, ideally different racks and AZs. Never co-locate with workloads.
• Use dedicated storage. NVMe or SSD with low fsync latency. RAID for disk failure. Watch df on the data directory — the 8 GB warning exists for a reason.
• Back up etcd, separately from PVCs. PVC bytes are application state; etcd is control-plane state. Backups run on different cadences (etcd hourly; application per-database) to different stores.

etcd is the database that the rest of the control plane treats as ground truth. Operating it well is the foundation that every other Kubernetes operation depends on.