etcd persistence — encoding, watch, and the cluster's write path

The API server persists every cluster object in etcd. This lesson covers how that persistence works — the encoding, the key layout, the watch stream, and the performance characteristics — and what each one means for production operations.

The write path

sequenceDiagram
    autonumber
    participant API as API server
    participant EN as Encoder
    participant ETCD as etcd
    participant WL as WAL

    API->>EN: encode object (protobuf)
    EN-->>API: bytes
    API->>ETCD: PUT /registry/pods/prod/web-7c8 (key, value, lease)
    ETCD->>WL: append to WAL
    ETCD->>ETCD: replicate to followers (Raft)
    ETCD-->>API: committed (revision = N)
    API-->>API: object.resourceVersion = N

A write to etcd involves:

1. Encoding — the object is serialised (protobuf by default; JSON is also accepted for reads)
2. Key construction — the etcd key is built from the resource’s API group, kind, namespace, and name
3. Write to leader — the API server’s request to etcd goes to the leader
4. Replication — the leader replicates the entry to followers
5. Commit — once a majority acknowledges, the leader commits and replies
6. Revision assignment — the committed entry has a global revision; this becomes the object’s resourceVersion

The end-to-end latency of a single write is bounded by: network round trips × 2 (leader → followers → leader), plus disk fsync on each member.

Encoding: protobuf vs JSON

The API server supports two encodings:

• protobuf (default for writes) — compact, fast, type-safe
• JSON — verbose, slower, used for human-readable kubectl output

Every API object has both. The protobuf form is what etcd stores; the JSON form is what kubectl returns. The API server translates between them via the conversion layer (see kubernetes-iii-02).

A kubectl request:

kubectl get pod web -o yaml

returns YAML (or JSON). Internally, the API server:

1. Reads the protobuf-encoded object from etcd
2. Decodes protobuf to the internal Go struct
3. Encodes to JSON (or YAML) for the response

A kubectl apply -f request:

1. Parses the YAML to a Go struct
2. Encodes to protobuf
3. Writes the protobuf to etcd

The protobuf encoding is smaller on disk and faster to decode, which is why it is the default for the storage path.

Key layout

etcd keys follow a hierarchical layout:

/registry/<group>/<version>/<kind>/<namespace>/<name>

Cluster-scoped resources omit the namespace:

/registry/<group>/<version>/<kind>/<name>

Examples:

/registry/core/v1/pods/prod/web-7c8
/registry/core/v1/services/prod/web
/registry/apps/v1/deployments/prod/web
/registry/core/v1/namespaces/prod
/registry/core/v1/nodes/worker-04

The key layout is documented in the registry package of the API server. Knowing the layout is what makes backup and restore tractable: an etcd snapshot captures the entire /registry/ subtree.

Reading etcd directly

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/core/v1/pods/prod/web-7c8 --print-value-only

The --print-value-only flag shows the encoded value (protobuf in storage; rendered as something readable by etcdctl when possible).

# List all keys under /registry/core/v1/namespaces
ETCDCTL_API=3 etcdctl ... get /registry/core/v1/namespaces --prefix --keys-only

This is how operators diagnose “is this object really in etcd?” — useful when the API server returns 404 but the manifest was just applied.

Watch semantics

The API server maintains a watch stream on etcd. Every change is forwarded to the watch subscribers (controllers, kubectl, monitoring tools):

sequenceDiagram
    autonumber
    participant E as etcd
    participant A as API server
    participant C1 as Controller 1
    participant C2 as Controller 2

    E-->>A: watch event (revision=N, type=PUT, key=...)
    A-->>C1: watch event (object, resourceVersion=N)
    A-->>C2: watch event (object, resourceVersion=N)
    Note over A: API server's cache is updated

The watch protocol:

• The client opens a watch with a starting resourceVersion
• The API server streams events as they happen
• Each event includes the new resourceVersion (and the previous one for MODIFIED/DELETED)
• On connection break, the client reconnects with the last resourceVersion; the API server replays missed events
• If the requested resourceVersion is too old (older than the watch cache retention), the API server returns 410 Gone and the client must re-list

The API server’s watch cache:

• Keeps events for a configurable duration (default ~5 minutes)
• Bounds memory by --watch-cache-sizes (per-resource limit)
• Is in-memory; lost on API server restart (clients must re-list)

resourceVersion

resourceVersion is the etcd revision at which the object was last written. It is included on every API response and on every watch event.

metadata:
  resourceVersion: "128034"
  uid: "7c8f2d8e-..."

resourceVersion enables:

• Optimistic concurrency — a write must include the expected resourceVersion; mismatch returns 409 Conflict
• Watch consistency — clients track their position in the event stream
• Snapshot reads — ?resourceVersion=N reads return objects as of revision N (eventually consistent)

A common pattern: a controller reads with ?resourceVersion=N, then watches events starting at N. The events fill the gap between the snapshot read and the current state.

Performance characteristics

The API server’s write latency is dominated by etcd:

• Healthy etcd commits in < 50 ms (sub-ms on local SSD)
• Network-bound commit (3 members across AZs): 5-15 ms
• Disk-bound commit (slow disk, GC pressure): 100+ ms

Production API server write latency:

kubectl get --raw /metrics | grep apiserver_request_duration_seconds

apiserver_request_duration_seconds_bucket{verb="POST",resource="pods",le="0.05"} 1245
apiserver_request_duration_seconds_bucket{verb="POST",resource="pods",le="0.1"}  1280
apiserver_request_duration_seconds_bucket{verb="POST",resource="pods",le="0.5"}  1287
apiserver_request_duration_seconds_bucket{verb="POST",resource="pods",le="1"}    1287

A p99 write latency > 500 ms is an indicator of etcd problems.

Watch latency

Watch latency is dominated by:

• API server’s cache update latency
• etcd’s notification latency
• Network between API server and etcd

A healthy cluster’s watch latency is < 100 ms p99. Production monitors:

# From kube-state-metrics or Prometheus
apiserver_watch_events_total
apiserver_watch_duration_seconds

Defragmentation

etcd’s storage does not shrink on delete. Over time, the on-disk size grows even if the number of objects is stable. The defrag operation compacts the storage.

# Defrag one member at a time; substitute its client URL:
ETCD_MEMBER=https://192.0.2.11:2379

etcdctl --endpoints="$ETCD_MEMBER" defrag

Production: schedule defrag during maintenance windows on each member sequentially. Run during low-write periods.

Compaction

etcd’s MVCC keeps every revision of every key (until compaction). The API server’s compaction runs automatically:

# /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - --auto-compaction-mode=periodic
    - --auto-compaction-retention-duration=5m

This compacts etcd’s history every 5 minutes, keeping only the last 5 minutes of revisions. Production tuning balances “history available for watch” vs “DB size”.

Backup and restore (recap)

The etcd snapshot is the only reliable cluster backup. See kubernetes-ii-03 for the full procedure.

etcdctl snapshot save /backup/etcd-snapshot.db
etcdutl snapshot status /backup/etcd-snapshot.db

A snapshot is a self-contained etcd data file; restore rewinds the cluster state.

Cross-course references

• The Linux course part XIII-Linux-Disks covers the disk performance characteristics etcd depends on.
• The Linux course part XXIV-Linux-Time covers chrony — etcd’s election timing depends on clock accuracy.
• The Observability course part V-Observability-PromArchitecture covers the metrics the API server exposes about etcd performance.
• The Docker course part XXXVII-Docker-Registries covers object storage patterns that map onto off-cluster etcd snapshot storage.

Quiz

Production discipline

• Monitor resourceVersion lag in controller logs. A controller whose resourceVersion falls too far behind cannot catch up efficiently.
• Schedule defragmentation during maintenance windows on each etcd member sequentially.
• Set auto-compaction (--auto-compaction-mode=periodic) to bound the revision history.
• Size etcd DB capacity with headroom: 8 GB is the recommended ceiling; investigate if approaching.
• Back up etcdsnapshot before any change that affects cluster state (RBAC, admission, CRD installation).