etcd alarm and no-space — the cluster's disk exhaustion

When etcd raises a NOSPACE alarm it stops accepting writes and the cluster goes read-only: kubectl get still works, kubectl apply does not, and no controller can record a decision. The alarm does not clear itself once space is freed — it has to be disarmed explicitly, after the store has been compacted and defragmented, which is where most recoveries stall. This lesson covers that sequence and the difference between reclaiming revisions and reclaiming disk.

The etcd disk

The etcd’s disk is the cluster’s state. The disk is structured as a WAL (write-ahead log) and a snapshot. The WAL is the cluster’s recent writes; the snapshot is the cluster’s compacted state.

flowchart LR
    A[WAL] --> B[etcd]
    B --> C[Snapshot]
    C --> D[Disk]

The etcd’s disk is the cluster’s state.

The no-space alarm

The no-space alarm is etcd’s signal that the disk is too full. The alarm is set when the WAL or the snapshot exceeds the disk’s quota.

# Check the alarms
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  alarm list

A real alarm list:

memberID:8489529684342625758 alarm:NOSPACE

The alarm is the etcd’s signal.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
ETCD_NODE=etcd-01.example.com

# 1. Check the alarms
ETCDCTL_API=3 etcdctl alarm list

# 2. Check the etcd's status
ETCDCTL_API=3 etcdctl endpoint status

# 3. Check the disk usage
ssh "$ETCD_NODE" "df -h /var/lib/etcd"

# 4. Check the etcd's logs
ssh "$ETCD_NODE" "journalctl -u etcd -n 200"

# 5. Check the cluster's state
kubectl get nodes

The diagnostic is the alarm, the status, the disk usage, and the logs.

Common failures

• Disk full. The disk is full. The remediation is to clean up the disk.
• No defrag. The etcd’s storage is fragmented. The remediation is to defrag.
• No compaction. The etcd’s WAL is not being compacted. The remediation is to compact.
• Quota exceeded. The etcd’s quota is exceeded. The remediation is to increase the quota.

flowchart TD
    A[etcd alarm] --> B{Disk full?}
    B -->|Yes| C[Clean up the disk]
    B -->|No| D{Fragmented?}
    D -->|Yes| E[Defrag the etcd]
    D---|No| F{Compacted?}
    F -->|No| G[Compact the etcd]
    F---|No| H{Quota exceeded?}
    H -->|Yes| I[Increase the quota]
    H -->|No| J[Unknown]

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
ETCD_NODE=etcd-01.example.com
# Revision to compact to, from `etcdctl endpoint status` (see below):
REV=48219003

# Option 1: Clean up the disk
ssh "$ETCD_NODE" "rm -rf /var/lib/etcd/member/wal/*.tmp"

# Option 2: Compact the etcd
ETCDCTL_API=3 etcdctl compaction "$REV"

# Option 3: Defrag the etcd
ETCDCTL_API=3 etcdctl defrag

# Option 4: Disarm the alarm
ETCDCTL_API=3 etcdctl alarm disarm

The remediation is the etcd recovery.

The compaction

The compaction removes the old revisions. The etcd’s default is to auto-compact every 5 minutes. The manual compaction is the emergency recovery.

# Compact the etcd
ETCDCTL_API=3 etcdctl compaction $(ETCDCTL_API=3 etcdctl endpoint status --write-out=json | jq -r '.[0].Status.header.revision')

The compaction is the etcd’s cleanup.

Production discipline

An etcd alarm is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the etcd, identify the cause, apply the remediation. The etcd is the cluster’s state; the remediation is the etcd recovery.

• Check the alarm. The alarm is the etcd’s signal.
• Check the disk. The disk is the etcd’s storage.
• Compact, defrag, and disarm. The recovery is the etcd’s cleanup.

Quiz