etcd — the cluster's source of truth

etcd is the database that backs every Kubernetes cluster. Every Pod, Service, Deployment, ConfigMap, Secret, RBAC binding, lease — the entire API object graph — lives in etcd. This lesson covers what etcd is, how it stays consistent, what its operational limits are, and how to back up and restore it.

etcd in one sentence

etcd is a consistent, distributed key-value store built on the Raft consensus algorithm. Every write is committed to a majority of members before being acknowledged; every read sees the most recent committed write.

flowchart LR
    A[API server] -->|PUT /registry/pods/prod/web| L[Leader]
    L -->|replicate| F1[Follower]
    L -->|replicate| F2[Follower]
    L -->|commit when majority ack| C[Committed]

The API server is etcd’s only client in a Kubernetes cluster. Every other component goes through the API server, which in turn talks to etcd. This makes etcd’s availability the cluster’s availability floor.

Raft consensus

etcd uses the Raft algorithm. Key properties:

• One leader at a time. All writes go through the leader.
• Quorum writes. A write is committed when a majority of members (1 + floor(N/2)) acknowledge it.
• Followers replicate. Followers apply the leader’s log entries in order.
• Election on leader loss. If the leader misses heartbeats, followers hold an election after a randomised timeout.

A 3-member cluster requires 2 acknowledgements for a commit. A 5-member cluster requires 3.

Production runs 3 or 5. 5 gives more failure tolerance but increases write latency (one more round trip per commit).

On-disk layout

Each etcd member stores:

/var/lib/etcd/
├── snap/                  # snapshots
│   └── 00000000000000ff-0000000000a0b1c2.snap
├── wal/                   # write-ahead log
│   └── 0000000000000000-0000000000000000.wal
└── config                 # cluster membership

The write-ahead log (WAL) captures every committed write in order. The snapshot is a periodic point-in-time dump of the database state. On restart, etcd replays the WAL since the last snapshot to recover the state.

flowchart LR
    W[Write] --> WAL[Append to WAL]
    WAL --> MEM[In-memory store]
    MEM -->|every N commits| SNAP[Snapshot]
    MEM -->|read| R[Response]

Snapshots bound the WAL replay time on restart. Without snapshots, the WAL grows unboundedly and recovery takes hours. With regular snapshots, restart is bounded to the time since the last snapshot.

Performance characteristics

etcd has documented performance limits. The defaults that matter:

• Storage size: <= 8 GB recommended. Above this, snapshot creation and defragmentation get slow. Production runs defragmentation in a maintenance window.
• Write rate: a few hundred writes/second sustained is comfortable; thousands of writes/second will increase commit latency.
• Object count: hundreds of thousands of objects is normal; millions require careful sizing.
• Commit latency: sub-100ms p99 on healthy members; the API server’s write latency floor.

etcd writes are limited by the slowest member. A member with slow disk, a slow network, or GC pressure will extend commit latency for the entire cluster.

# Substitute your own member endpoint before running:
MEMBER=https://192.0.2.10:2379

etcdctl --endpoints="$MEMBER" endpoint status --write-out=table

+---------------------------+------------------+---------+
|         ENDPOINT          |        ID        | IS S?  |
+---------------------------+------------------+---------+
| https://10.0.1.10:2379    | c5e9...          | true   |
| https://10.0.1.11:2379    | d7a1...          | true   |
| https://10.0.1.12:2379    | e8b4...          | true   |
+---------------------------+------------------+---------+

+---------------------------+------------------+---------+
| ... (db size, leader, raft index, raft term, etc.) ...
+---------------------------+------------------+---------+

Snapshot and restore

etcd’s snapshot is the only reliable backup of cluster state. Production takes snapshots:

• Periodically (every few hours; stored off-cluster)
• Before any change that touches cluster state (upgrade, RBAC, CRD installation)
• After a known-good state (post-upgrade validation)

etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot-$(date +%Y%m%d-%H%M).db

A snapshot is a self-contained file. It can be restored:

# 1. On every control-plane node, park the API server and etcd
#    manifests so that nothing can write to etcd during the restore.
#    This is how you stop them: kubeadm runs both as static Pods, and
#    the kubelet keeps a Pod running for every file it finds here.
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/kube-apiserver.yaml \
   /etc/kubernetes/manifests/etcd.yaml \
   /root/manifests-parked/

# 2. Confirm both containers are gone. kubectl is down at this point,
#    so ask the container runtime on the node instead.
crictl ps | grep -E 'kube-apiserver|etcd'
# Expected: no rows, within one kubelet fileCheckFrequency (20s default)

# 3. Restore into a fresh directory, on every control-plane node, from
#    the same snapshot file. --name must match the member name in that
#    node's /etc/kubernetes/manifests/etcd.yaml, which kubeadm sets to
#    the node name.
etcdutl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=cp-1 \
  --initial-cluster=cp-1=https://10.0.1.10:2380,cp-2=https://10.0.1.11:2380,cp-3=https://10.0.1.12:2380 \
  --initial-advertise-peer-urls=https://10.0.1.10:2380

# 4. Swap the data directory. Keep the old one until the cluster is
#    verified — it is the only rollback you have.
mv /var/lib/etcd /var/lib/etcd.broken
mv /var/lib/etcd-restore /var/lib/etcd

# 5. Put etcd back first and let the members form a quorum, then the
#    API server. The kubelet recreates each static Pod within one
#    fileCheckFrequency interval.
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/
crictl ps | grep etcd

mv /root/manifests-parked/kube-apiserver.yaml /etc/kubernetes/manifests/
kubectl get --raw /healthz

A restore rewinds the entire cluster state to the snapshot moment. Any state written after the snapshot is lost. This is why snapshots are taken frequently and stored off-cluster.

Defragmentation

etcd stores data in a key-value engine (boltdb historically; the production-default in 3.5 is still boltdb on disk, with optional bbolt variants). When keys are deleted, the storage does not shrink — it is “fragmented”. Long-running clusters without defragmentation accumulate wasted space.

etcd exposes two metrics:

• etcd_debugger_mvcc_db_total_size_in_bytes — actual disk consumption
• etcd_debugger_mvcc_db_total_used_in_size_in_bytes — used storage

When total_size grows much larger than total_used, defragmentation helps:

# Substitute your own member endpoint before running:
MEMBER=https://192.0.2.10:2379

etcdctl --endpoints="$MEMBER" defrag

This is a maintenance window operation on each member sequentially (not all at once — defragmentation rewrites the DB file, which is expensive).

Why etcd does not run on the workers

etcd is a control-plane component, run on dedicated hosts (or on managed infrastructure). Running etcd on worker nodes is an anti-pattern because:

• Worker nodes have noisy neighbours (the workloads). etcd’s latency is dominated by disk and GC; co-locating it with Pods is a performance and reliability hazard.
• Worker nodes can be cordoned and drained for maintenance; the cluster loses etcd members on every worker drain.
• etcd’s failure domain should be independent of the work the cluster is doing.

In production, etcd members typically run on dedicated control- plane nodes (3 control-plane nodes is the minimum HA topology), or on separate small VMs in cloud-managed Kubernetes.

What the API server expects from etcd

The API server writes to etcd on every state change. The writes are:

• Serialised. Every write is a single Raft commit.
• Versioned. Each object has a resourceVersion (the etcd revision).
• Optimistically concurrent. Writes must include the expected resourceVersion; mismatches return 409 Conflict.

The API server’s write latency is dominated by:

1. Authentication + authorisation (cheap)
2. Admission (depends on webhook count)
3. Schema validation (cheap)
4. etcd commit (network to leader + replication + commit)

Production tuning focuses on step 4. A healthy etcd commits in < 50 ms; a slow etcd commits in 500 ms+, which propagates to every kubectl apply.

etcd monitoring

Production clusters monitor etcd with:

• etcd_server_has_leader — is there a leader? Must be 1.
• etcd_server_leader_changes_seen_total — leader changes should be rare; spikes correlate with network instability.
• etcd_disk_wal_fsync_duration_seconds — disk fsync latency directly bounds commit latency.
• etcd_disk_backend_commit_duration_seconds — backend commit latency.
• etcd_debugger_mvcc_db_total_size_in_bytes — DB size on disk.
• etcd_server_proposals_applied_total / ..._failed_total — applied vs failed proposals; sustained failures are a red flag.

# Example Prometheus alert
- alert: EtcdNoLeader
  expr: etcd_server_has_leader == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "etcd member {{ $labels.instance }} has no leader"

Failure modes and recovery

Production discipline: every failure mode has a recovery procedure, every procedure is rehearsed, every snapshot is stored off-cluster with integrity verified.

Cross-course references

• The Linux course part XIII-Linux-Disks covers the disk performance characteristics etcd depends on; etcd’s commit latency is bounded by disk fsync latency.
• The Linux course part XXIV-Linux-Time covers chrony — etcd elections depend on clocks being within tolerance.
• The Observability course part V-Observability-PromArchitecture covers the metrics surface etcd exposes and how to alert on it.
• The Docker course part XXXVII-Docker-Registries covers offsite storage patterns that map onto off-cluster etcd snapshot storage (object storage with versioning).

Quiz

Production discipline

• Run 3 or 5 etcd members in production; odd-numbered; spread across failure domains.
• Take snapshots regularly, store off-cluster, verify integrity with etcdutl snapshot status, rehearse the restore.
• Monitor fsync duration, leader changes, DB size, and applied/failed proposals. These predict outages.
• Pin etcd’s CPU and memory at the kernel level; etcd’s latency is dominated by disk and GC, both of which noisy neighbours disturb.
• Rehearse the restore procedure before the first incident, not during. A restore is a multi-step operation; the operator who has never done it will not do it correctly at 03:00.