critical riskcluster affecting~30 min

Runbook: Investigate etcd Health

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture etcd endpoint health: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=<ca> --cert=<cert> --key=<key> endpoint health -w table
· Capture etcd endpoint status: etcdctl ... endpoint status -w table
· Capture etcd alarms: etcdctl ... alarm list
· Capture the etcd metrics endpoint: curl -k https://127.0.0.1:2379/metrics | grep -E "etcd_|disk_" | head
· Capture the etcd data directory size: du -sh /var/lib/etcd
· Confirm a recent snapshot exists: ls -lh /var/backups/etcd/

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Run endpoint health on every member: etcdctl ... endpoint health --cluster -w table
2Run endpoint status to read leader, term, revision, DB size: etcdctl ... endpoint status --cluster -w table
3Check alarm list: etcdctl ... alarm list (an active alarm blocks writes)
4Compare DB sizes across members; they should be within 5% (defragmentation may be needed)
5Read the disk write latency in the metrics: etcd_disk_wal_fsync_duration_seconds
6Identify the failing member (if any) and the failure mode: slow disk, network partition, clock skew
7Apply the smallest fix: defragment, restart a slow member, repair the network, restore from snapshot (only if quorum is lost)
8Verify: endpoint health reports every member healthy and a leader is elected

4 · Verification

Confirm the procedure actually fixed the problem.

✓etcdctl ... endpoint health --cluster -w table reports every member healthy
✓etcdctl ... endpoint status --cluster -w table shows exactly one leader and the others as followers
✓etcdctl ... alarm list returns empty
✓etcdctl ... endpoint status shows raft.term is reasonable (not increasing every minute)
✓etcdctl ... endpoint status shows DB sizes within 5% across members
✓etcd_disk_wal_fsync_duration_seconds p99 < 10ms
✓Kubernetes API server responds: kubectl get nodes returns Ready

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If restarting a member made things worse, leave it stopped and use the remaining quorum
↶If defragmentation made things worse, do not re-run; the DB size will recover on the next compaction
↶If a restore was made in error, restore the original data directory from the snapshot taken before the bad restore
↶Capture the etcd logs and the snapshot before any destructive action
↶For quorum loss, see kubernetes-rb-recover-failed-control-plane and consider a full restore from snapshot

6 · Escalation

When the runbook isn't enough, contact:

· Quorum lost (more than one member down for a 3-node cluster): escalate to the disaster-recovery runbook before any action
· DB size approaching the quota (default 8 GiB): defragment or compact; escalate to platform ownership
· Leader election storm: a member is unstable; capture its logs and escalate
· Clock skew across members > 30 seconds: stop and fix NTP; etcd refuses to elect a leader with skewed clocks
· Disk latency > 50ms: the disk is failing; replace the node before it loses quorum

etcd is the cluster’s source of truth. A failing etcd stops every Kubernetes API operation. The runbook reads etcd’s view of itself first, before any client-side diagnosis.

1. Confirm the etcd endpoint

Read-only / SafeConfirm the etcd endpoint


ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health -w table

etcdctl ... endpoint status --cluster -w table
# Columns: ENDPOINT, ID, VERSION, DB SIZE, IS LEADER, IS LEARNER, RAFT TERM, RAFT INDEX

A healthy cluster reports:

Every member health: true
Exactly one leader
DB sizes within 5%
Reasonable raft.term (single digits to low thousands, not growing)

2. Alarms

Read-only / SafeAlarms


# Returns:
#   memberID:X alarm:NOSPACE
#   memberID:Y alarm:NOSPACE
# Or empty (no alarm)

A NOSPACE alarm blocks writes. The fix is to compact and defragment, then disarm the alarm.

3. Performance metrics

Read-only / SafePerformance metrics


etcd_disk_(wal|backend)_fsync_duration_seconds|etcd_disk_wal_fsync_duration_seconds_count|etcd_server_proposal_duration_seconds|etcd_server_slow_apply_total" | head -20

# p99 disk latency:
# etcd_disk_wal_fsync_duration_seconds{p99} should be < 10ms
# If > 50ms, the disk is the bottleneck

Metric	Healthy	Investigate
`etcd_disk_wal_fsync_duration_seconds` p99	< 10ms	> 50ms — disk issue
`etcd_server_proposal_duration_seconds` p99	< 50ms	> 200ms — election storms or slow apply
`etcd_server_slow_apply_total`	0	> 0 — etcd missed the heartbeat timeout
`etcd_mvcc_db_total_size_in_bytes`	< 4 GiB	> 6 GiB — compaction needed

4. DB size and compaction

Read-only / SafeDB size and compaction


DB SIZE|raft.term"

# Count of compaction keys (this is the rev after the last compaction)
curl -k https://127.0.0.1:2379/metrics | grep etcd_mvcc_compact

etcd stores all historical keys until compacted. Compaction is automatic (auto-compaction in the etcd config) but can be triggered manually:

Read-only / SafeDB size and compaction


REV=$(etcdctl ... endpoint status --cluster -w json | jq -r ".[0].Status.raftIndex')
etcdctl ... compact "$REV"

# Defragment (one member at a time to keep quorum)
etcdctl --endpoints=https://<member-1>:2379 ... defrag
etcdctl --endpoints=https://<member-2>:2379 ... defrag
etcdctl --endpoints=https://<member-3>:2379 ... defrag

# Disarm any NOSPACE alarm
etcdctl ... alarm disarm

5. Restart a stuck member

Read-only / SafeRestart a stuck member


ssh <cp-failed> -- sudo crictl ps -a | grep etcd
ssh <cp-failed> -- sudo crictl stop <etcd-container-id>

# Wait for kubelet to recreate the static pod
sleep 30
ssh <cp-failed> -- sudo crictl ps -a | grep etcd

# Confirm the member rejoined
etcdctl ... endpoint health --cluster -w table

A restarted member rejoins the cluster by replaying its WAL against the leader. This takes minutes for a healthy cluster; longer if the DB is large.

6. Quorum loss

See kubernetes-rb-recover-failed-control-plane and kubernetes-rb-restore-etcd for the full sequence.

7. Verify

Read-only / SafeVerify


etcdctl ... endpoint status --cluster -w table
etcdctl ... alarm list
curl -k https://127.0.0.1:2379/health

# Kubernetes side
kubectl get nodes -o wide
kubectl get pods -A | wc -l
kubectl create -f - <<'YAML' || echo "expected to fail if exists"
apiVersion: v1
kind: ConfigMap
metadata:
name: etcd-health-test
namespace: default
data:
test: ok
YAML
kubectl delete cm etcd-health-test --wait=false

Common pitfalls

Symptom	Cause	Action
DB size rapidly growing	A controller or client in a write loop	Identify and stop the writer
`NOSPACE` alarm active	DB size exceeded quota	Compact and defragment
Leader election storm	A member unstable	Restart the member; investigate the host
`endpoint health` reports one member as unhealthy	Member cannot reach quorum or has lost its WAL	Restart the member; restore from snapshot if WAL is corrupted
Slow writes, normal disk	Many small writes from a controller	Reduce the controller’s write rate

A failing etcd is rarely a Kubernetes problem. It is a disk, a network, or a clock. The runbook distinguishes these by reading metrics before restarting anything.