Performance incident on etcd — the cluster's slowness

Every kubectl call, every controller reconcile, and every scheduler decision commits through etcd. A disk whose fsync takes 12ms instead of 2ms therefore shows up as cluster-wide latency with no component reporting an error anywhere, which is why it is usually diagnosed late. This lesson covers measuring the four quantities that actually bound etcd — fsync latency, disk throughput, network latency between members, and database size — and which of them an operator can change.

The etcd’s performance

The etcd’s performance is governed by:

• fsync latency. The disk’s fsync latency. The default recommendation is < 10ms.
• Disk throughput. The disk’s read/write throughput.
• Network latency. The network between the etcd members.
• Object size. The size of the objects stored in etcd.

flowchart LR
    A[fsync latency] --> B[etcd performance]
    C[Disk throughput] --> B
    D[Network latency] --> B
    E[Object size] --> B

The performance is the etcd’s bottleneck.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
ETCD_NODE=cp-01.example.com

# 1. Check the etcd's endpoint status
ETCDCTL_API=3 etcdctl endpoint status --write-out=json | jq -r '.[] | {Endpoint: .Endpoint, Latency: .Status.leader, DbSize: .Status.dbSize}'

# 2. Check the fsync latency
ssh "$ETCD_NODE" "dd if=/dev/zero of=/var/lib/etcd/test bs=4k count=1000 oflag=direct"

# 3. Check the disk I/O
ssh "$ETCD_NODE" "iostat -x 1 5"

# 4. Check the etcd's logs
ssh "$ETCD_NODE" "journalctl -u etcd -n 200"

# 5. Check the cluster's state
kubectl get nodes

The diagnostic is the latency, the fsync, the disk, and the logs.

The fsync latency

The fsync latency is the etcd’s bottleneck. The default recommendation is < 10ms. The fsync latency is the disk’s commit time.

# Substitute your own value before running:
ETCD_NODE=cp-01.example.com

# Test the fsync latency
ssh "$ETCD_NODE" "dd if=/dev/zero of=/var/lib/etcd/test bs=4k count=1000 oflag=direct"

A real fsync latency:

1000+0 records in
1000+0 records out
4096000 bytes (4.1 MB) copied, 0.012 s, 341 MB/s

The 0.012s is 12ms, which is above the 10ms threshold. The etcds are slow.

The remediation

The remediation depends on the cause:

# Option 1: Use a faster disk
# Replace the disk with NVMe SSD

# Option 2: Reduce the etcd's database size
# Compact, defrag, and prune

# Option 3: Reduce the cluster's load
# Reduce the number of objects stored in etcd

# Option 4: Increase the etcd's resources
# Increase the CPU and memory of the etcd members

The remediation is the etcd tuning.

The disk I/O

The disk I/O is the etcd’s bottleneck. The disk’s read/write throughput is the etcd’s performance.

# Substitute your own value before running:
ETCD_NODE=cp-01.example.com

# Check the disk I/O
ssh "$ETCD_NODE" "iostat -x 1 5"

A real disk I/O:

Device  r/s   w/s   rkB/s  wkB/s  await  svctm  %util
sda     12    450   120    1800   25.4   2.2    99.0

The 99% utilization is the disk’s bottleneck.

The cluster’s load

The cluster’s load is the etcd’s input. The number of objects stored in etcd is the cluster’s state.

# Check the etcd's database size
ETCDCTL_API=3 etcdctl endpoint status --write-out=json | jq -r '.[0].Status.dbSize'

A real database size:

8589934592

The 8GB is the cluster’s state. The recommended maximum is 8GB.

Production discipline

An etcd performance incident is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the etcd, identify the cause, apply the remediation. The etcd is the cluster’s state; the remediation is the etcd tuning.

• Check the fsync latency. The fsync latency is the etcd’s bottleneck.
• Check the disk I/O. The disk I/O is the etcd’s performance.
• Check the cluster’s load. The cluster’s load is the etcd’s input.

Quiz