Performance limits — disk fsync, write rate, latency budgets

etcd’s performance is bounded by a small set of physical limits: the disk fsync latency, the round-trip time between members, the bbolt page-cache working set, and the fsync budget of the OS. Operators who know which limit will hit first for their workload reason about capacity before saturation; operators who don’t know learn the limit during an incident. This lesson walks the limits, the metrics that expose each one, and the ceilings that drive planning.

The three ceilings

flowchart LR
    R[request] -->|write| WAL[wal fsync]
    R -->|read| BB[bbolt page lookup]
    WAL -->|quorum round-trip| PEER[peer ack]
    PEER -->|response| R
    BB -->|RAM pressure| FS[file system cache]

• WAL fsync latency — the time to durably persist one entry. Determines per-write commit latency floor.
• Quorum round-trip — the time to replicate an entry to a majority of members. Determines the network-bound ceiling on commit.
• bbolt working set — the size of the hot pages in memory. Determines read latency and the cliff at which the cluster starts swapping.

The three overlap. In a healthy cluster, disk fsync is the binding ceiling; in a heavily multi-AZ cluster, the quorum round-trip dominates; in a large cluster (many objects), the working set is the cliff.

Disk fsync — what to measure

# Watch the live metric
etcd_disk_wal_fsync_duration_seconds{quantile="0.99"}

The metric is per member, per call, so a slow disk on one member shows as that member’s p99 being out of line with the others. A common cause:

• The disk is rotational and the contention is somewhere in the IO scheduler queue.
• The disk is virtualised (an AWS gp2 over a noisy neighbour; a cinder volume on shared storage).
• The disk’s NVRAM cache (BBWC) is dead or disabled.
• The OS is writing to swap because real memory is full.

Quorum round-trip — the network ceiling

For a 3-member cluster in one AZ, peer round-trip is sub-millisecond. For a 3-member cluster across AZs in a cloud region, peer round-trip is 1-3 ms typical, 5-20 ms p99. For a WAN-distributed etcd, peer round-trip can exceed 50 ms.

The metric to track:

etcd_disk_backend_commit_duration_seconds{quantile="0.99"}

The backend commit duration is end-to-end server-side: WAL append + Raft replication + commitIndex advance. In a healthy cluster, it tracks WAL fsync + peer RTT. If it diverges from WAL fsync alone, peer RTT is the reason.

Object count and working set

The etcd bbolt database mmaps the file. Reads that hit the OS page cache are essentially memory reads; reads that miss the page cache are disk reads. The working set is the set of pages touched by recent activity.

Three object-count ceilings matter:

The bbolt working set is bounded by host memory. Each page is 4 KiB. A 2 GiB DB has 524,288 pages. Practical operating point: at least 25% of DB size should fit in memory comfortably.

$ kubectl get pods -A --no-headers | wc -l

184

Write rate and quorum round-trips

The cluster’s aggregate write rate is bounded by the slowest member. If the slowest member’s WAL fsync averages 15 ms, the cluster’s commit latency floor is ~15 ms plus peer round-trip.

In a healthy production cluster:

• Per-write commit latency budget: < 50 ms p99.
• Per-write read latency budget: < 25 ms p99 from the watch cache; < 100 ms p99 linearizable through etcd.
• Apply latency budget (Raft applied vs committed): < 1 second p99.

flowchart LR
    W["write attempt 1"] -->|inflight| LEADER[leader]
    LEADER -->|fsync 12 ms| WAL[wal]
    LEADER -->|replicate 5 ms| F1[follower 1]
    LEADER -->|replicate 4 ms| F2[follower 2]
    F1 -->|ack| LEADER
    F2 -->|ack| LEADER
    LEADER -->|commit| W

A 12 ms WAL fsync plus 5 ms peer replication means commit takes ≥ 17 ms (plus a couple of milliseconds for the proposal). Operator-visible: every apply is ≥ 17 ms.

Request rate — the linear ceiling

etcd’s per-request rate ceiling is bounded by the CPU and the peer network. Common production thresholds:

Kubernetes’s actual steady-state write rate is dominated by:

• Controller updates (Deployment, ReplicaSet, EndpointSlice updates from each kubelet).
• Operator updates (cert-manager, ArgoCD, ExternalSecrets).
• Application-level controller updates (HPA, VPA).

In a cluster running 1,000 Pods, expect 5-50 writes/sec of control-plane traffic; that rate is healthy. Above 100 writes/sec sustained, investigate what is writing.

The key-size limit

A single key or value in etcd has a 1.5 MiB limit (the HTTP body limit). Kubernetes objects normally sit well under that:

• Pods: a few KiB.
• ConfigMaps: 1 MiB is the documented practical limit.
• Secrets: 1 MiB is the documented practical limit.
• Custom Resources: depends; CRDs can include spec objects with arbitrary size.

A multi-MiB object slows defragmentation and increments the page cache working set disproportionately. The operator’s rule of thumb: no single object over ~1 MiB.

The capacity planning answer

For a cluster sized for production (3-member etcd, ≥ 4 vCPU, ≥ 8 GiB memory per control plane):

These are targets, not enforced limits. An operator who hits them has headroom; an operator approaching them should investigate growth; an operator exceeding them should be on a path to scale out or reduce churn.

Performance anti-patterns

The alerting set

A small, high-signal Prometheus alert set:

- alert: EtcdFsyncLatencyHigh
  expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.025
  for: 5m
  labels:
    severity: warning

- alert: EtcdBackendCommitHigh
  expr: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) > 0.100
  for: 5m
  labels:
    severity: warning

- alert: EtcdDBSizeWarning
  expr: etcd_debugger_mvcc_db_total_size_in_bytes > 8 * 1024 * 1024 * 1024
  for: 1h
  labels:
    severity: warning

- alert: EtcdMemberDown
  expr: up{job="etcd"} == 0
  for: 1m
  labels:
    severity: critical

- alert: EtcdNoLeader
  expr: etcd_server_has_leader == 0
  for: 1m
  labels:
    severity: critical

- alert: EtcdApplySlow
  expr: rate(etcd_server_slow_apply_total[5m]) > 1
  for: 5m
  labels:
    severity: warning

The rule of thumb: page on the metric that maps to the ceiling closest to the boundary. fsync above 25 ms p99; apply latency above 500 ms p99; DB size above 8 GiB; no leader for 1 minute.

Quiz

Production discipline

• fsync is the floor. A 25 ms p99 fsync is a 25 ms p99 commit latency. Watch the histogram, alert at the threshold.
• The slowest member is the cluster. Capacity planning starts with the slowest disk in the slowest member; no single member can be a slow outlier.
• Watch proposal rate by source. The cluster’s proposals_applied_total is the noisy-controller detector. Attribute it.
• 2 GiB DB and 100 writes/sec are design budgets. A cluster running below these has headroom; a cluster running at them needs watching; a cluster exceeding them needs growth mitigation.
• Alert on the ceiling that maps to the next failure. fsync above 25 ms; apply latency above 500 ms; DB size above 8 GiB.

Performance limits are the floor under the operational discipline. Knowing them is what turns “feels slow” into “the disk fsync on member B is the bound”.