Snapshots, compaction, defragmentation — 8 GB warning, 2 GB recommended

etcd is a key-value store with three operational levers that act on its storage size: snapshots bound replay time, compaction removes historical revisions, and defragmentation reclaims space bbolt reserved for deleted keys. Confusing the three is one of the most common etcd operational mistakes. This lesson walks each one, the order they happen in, and the limits that drive the 8 GB warning and the 2 GB recommendation.

The three operations, distinct

flowchart LR
    SN[snapshot save] -->|file| DISK[off-cluster]
    C[compaction] -->|release| REVS[old revisions]
    D[defragmentation] -->|rewrite| BB[bbolt file]

A new operator frequently assumes these are the same operation or that one supersedes another. They are not. Snapshot is the backup, compaction is the cleanup of revisions, defragmentation is the release of disk space that has been freed by compaction.

Compaction — releasing revisions

etcd’s MVCC model keeps every revision of every key. The revision is what allows the read-history and watch-events that the API server’s watch cache and controllers depend on. But old revisions pile up, and Kubernetes never intentionally looks at revisions older than a few seconds.

Compaction removes revisions older than a specified revision or older than a specified TTL. The default auto-compaction mode is periodic with a default retention of 5 minutes. The API server’s watchers are guaranteed to have caught up by the time compaction completes (this is why the --watch-progress-notify-interval is shorter than the compaction period; controllers periodically re-watch to avoid getting stuck on a deleted revision).

# Manual compaction to a specific revision
etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=... --cert=... --key=... \
  compaction 41289300

# Compact with a TTL (the same as auto-compaction)
etcdctl compaction --compact-time="$(date -u --date='-5 minutes' +%FT%TZ)"

Defragmentation — reclaiming space

After compaction discards old revisions, the disk space is not immediately reclaimed by the bbolt file. bbolt mmaps pages into address space; pages with deleted keys remain until the file is rewritten. The space between the actual in-use data and the on-disk file size is called fragmentation.

flowchart LR
    B[bbolt file 8 GB] -->|In-use 1.4 GB| USED[live keys]
    B -->|Free 6.6 GB| FRAG[deleted keys/old revisions]
    D[defragmentation] -->|rewrites file| NEW[bbolt file ~1.4 GB]

The operator sees this through the two key metrics:

• etcd_debugger_mvcc_db_total_size_in_bytes — file on disk.
• etcd_debugger_mvcc_db_total_used_in_size_in_bytes — live data.

When total_size is much larger than total_used, the DB is fragmented. Defrag recovers it.

# Address of the etcd member to defrag:
MEMBER_IP=192.0.2.11

# Defrag a single member (one at a time; sequentially)
etcdctl --endpoints="https://$MEMBER_IP:2379" \
  --cacert=... --cert=... --key=... \
  defrag

The defrag operation rewrites the bbolt file in place. It is heavy disk I/O and heavy CPU. The etcd leaders’ defrag should be scheduled carefully because writing the file takes the member off-line briefly and big defrags can exceed the cluster’s --election-timeout budget if they are slow.

Snapshots — backups that bound replay

A snapshot is a checkpoint of the bbolt file at a known log index. The member creates snapshots on its own schedule (--snapshot-count controls frequency; the default is 100,000 entries). The snapshot is the member’s restart accelerator.

Two separate things are commonly called snapshots:

1. Internal snapshots (the snap/ directory): used by the member for its own restart acceleration and for catch-up of new followers.
2. etcdctl snapshot save files: operator-initiated backups intended for off-cluster storage.

# Internal snapshots are at /var/lib/etcd/snap/*.snap
ls -la /var/lib/etcd/snap/
# total 8
# -rw------- 1 etcd etcd 41289312 Aug 16 10:00 00000000000000ff-0000000000a0b1c2.snap

# Operator-initiated backup
etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M).db
# "saved snapshot to /backup/etcd-..."

A snapshot file is self-contained and consistent; it is the canonical restore primitive (Part LXIX).

The 8 GB warning and 2 GB recommended

The published limits:

Why these numbers. The numbers are empirical. etcd’s maintained tests under load show:

• At DB size ≤ 2 GiB, defrag completes in seconds with no measurable commit-latency impact.
• At DB size 8 GiB, defrag takes minutes per member; the transient commit-latency spike during the defrag can trigger leader elections.
• Above 8 GiB, the bbolt page cache churns the file’s working set; commit latency grows; snapshot creation becomes a noticeable event.

Maintenance cadence

A healthy production cadence:

gantt
    title Production etcd maintenance cadence
    dateFormat HH:mm
    axisFormat %H:%M
    section Hourly
    Snapshot to off-cluster :h1, 00:00, 5m
    section Daily
    Snapshot integrity verify :d1, after h1, 5m
    section Weekly
    Compact (if not auto) :w1, 00:00, 1m
    section Monthly
    Defrag one member :m1, 02:00, 30m
    section Quarterly
    Restore drill on a fresh host :q1, 03:00, 60m

Each operation is captured as a runbook step with pre/post metrics recorded automatically. The cadence is not a suggestion; it is the discipline that keeps etcd in the 2 GiB region.

What the operator inspects before defrag

# Confirm we are talking to one member
etcdctl endpoint status --endpoints=https://10.0.1.10:2379,... \
  --write-out=table | grep -E 'ID|DB SIZE|RAFT'

$ etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=json | jq '.[] | {Endpoint, "DB SIZE": .Status.dbSize, "DB SIZE IN USE": .Status.dbSizeInUse, "RAFT INDEX": .Status.raftIndex, "IS LEADER": .Status.leader == .Status.id}'

{
"Endpoint": "https://10.0.1.10:2379",
"DB SIZE": 1500000000,
"DB SIZE IN USE": 880000000,
"RAFT INDEX": 41289312,
"IS LEADER": true
}

The size of the file versus the size of the live data is the deflection. A 1.5 GiB file with 880 MB live means ~620 MB of compaction-released space; defrag will reclaim it.

What goes wrong

UnderTheHood title=“Why fragmenting doesn’t shrink the file”> bbolt allocates pages on demand and frees them in place via free-list pages. The free-list is in the file; the OS file size does not shrink when pages are freed, because bbolt does not truncate(2) the file. Only a rewrite (defragmentation) compacts the free list back to the active range. The 8 GiB warning arises because the larger the file, the more pages the defragmentation traversal touches.

Quiz

Production discipline

• Run auto-compaction with a retention of 5 minutes or shorter. Older revisions pile up; the API server’s controllers and watchers never read revisions older than seconds anyway.
• Defrag on schedule. Weekly during low traffic, one member at a time. The maintenance window is the cost; the alternative is a DB near 8 GiB.
• Hold the line at 2 GiB. Investigate any growth above 2 GiB in steady state. The cluster is leaking somewhere: large CRD instances, retained events, misconfigured audit logs.
• Snapshot hourly to off-cluster storage. The snapshot is the recovery primitive; if you can recover it, you can recover the cluster.
• Rehearse the restore quarterly. A snapshot that has never been restored is a snapshot you don’t know how to restore.

The 8 GiB warning is the floor of the maintenance problem; the 2 GiB recommendation is the floor of the operational discipline.