Write-ahead log, bbolt, and the on-disk format

Every write etcd accepts eventually lands on disk. Two file types hold that write before it is queryable: a write-ahead log (WAL) entry that captures the raw log record, and a bbolt page that holds the new state of the key-value store. A third file type — a snapshot — is created on a schedule to bound how much WAL has to be replayed on restart. This lesson walks the on-disk layout, the fsync boundary, and how a member reconstructs its state on startup.

The directory layout

A single etcd member’s data directory looks like:

/var/lib/etcd/
├── snap/
│   ├── 00000000000000ff-0000000000a0b1c2.snap
│   └── 000000000000012a-0000000000c2b1d3.snap
├── wal/
│   ├── 0000000000000000-0000000000000000.wal
│   ├── 0000000000000001-0000000000000037.wal
│   └── 0000000000000038-000000000000009b.wal
└── member/
    ├── snap /
    │   └── db          # bbolt file (key-value store)
    └── auth /          # auth tokens (if RBAC etcd auth is enabled)

The directory is owned by the etcd user; only that user must read it. kubeadm configures this on a fresh install via the etcd static pod’s volumeMounts.

The write-ahead log

The WAL is the durability boundary. Every committed entry the leader produces is appended to the WAL before it is acknowledged to the client. WAL files are 8 MiB (default --wal-segment-bytes) and renamed as they roll over.

flowchart LR
    W[committed entry] -->|append| WAL[wal/...wal]
    WAL -->|fsync| FS[disk]
    W -->|apply to| BB[bbolt: mmap db]
    BB -->|on rotation| F[flush dirty pages]

The WAL append is synchronous. An fsync follows. The behaviour we observe as etcd operators:

• A write completes when fsync returns. The metric etcd_disk_wal_fsync_duration_seconds is the per-member-per-call latency of that fsync.
• If the leader dies between fsync and ack to the client, the entry is durable but the client sees a timeout. On restart, the new leader will either have the entry (if it replicated it before the original leader died) or not (if the original leader was partitioned from the cluster). Either way, no inconsistency.
• WAL files are never edited in place. Truncation is on snapshot boundaries.

The bbolt store

The key-value store that backs kvstore is bbolt (formerly BoltDB). bbolt is a single-file B+tree database that maps a single root bucket and supports nested buckets within. Kubernetes-embedded etcd uses bucket paths like /registry/pods/<ns>/<name> directly:

flowchart LR
    B[bbolt root] --> R[registry]
    R --> P[pods]
    R --> D[deployments]
    R --> S[secrets]
    P --> NS1[prod]
    P --> NS2[kube-system]
    NS1 --> K1[web -> JSON value]
    NS1 --> K2[worker -> JSON value]

# Show the on-disk size of the store
ls -lh /var/lib/etcd/member/snap/db
# -rw------- 1 etcd etcd 1.4G Aug 16 06:00 db

The bbolt page size (default 4 KiB) and the PageSize govern how the file is laid out. Pages are mmapped, so reads are essentially memory reads; writes happen in two phases — apply to the in-memory page (COW), then write a new page to disk.

Why two stores

etcd uses a WAL and a bbolt DB because the access patterns differ:

• The WAL is append-only and linear. It is the per-write durability record.
• The bbolt DB is random-access and searchable. It is the per-read path.

Without the WAL, bbolt’s tree would need to be rewritten with every write; without bbolt, the WAL would need to be replayed from day one for every read. The split is what makes etcd both fast and durable.

$ etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table

+---------------------------+------------------+----------+-----------+-----------+------------+----------+
|         ENDPOINT          |        ID        | RAFT TERM | RAFT INDEX | RAFT APPLIED | DB SIZE   | DB SIZE IN USE |
+---------------------------+------------------+----------+-----------+-----------+------------+----------+
| https://10.0.1.10:2379    | c5e9a1b2...       |    17    | 41289312 |  41289312 | 1.50 GB    |   880 MB  |
+---------------------------+------------------+----------+-----------+-----------+------------+----------+

The gap between DB SIZE (the bbolt file) and DB SIZE IN USE (the actual live data) is fragmentation. That gap is what etcdctl defrag recovers.

Snapshots and the snapshot store

A snapshot is the bbolt store written out as a compressed key-value tree at a known log index. Snapshots serve two purposes:

1. Restart speed. On restart, the member replays WAL only since the last snapshot, not since the beginning of time.
2. Catch-up for slow followers. A follower that has fallen so far behind that the leader no longer has the matching log entries is sent a snapshot; the follower installs the snapshot and truncates its log.

flowchart TB
    L[Leader] --> SN[Create snapshot at index=41200000]
    SN --> F1[Send snapshot to slow follower]
    SN --> LOCAL[Persist snapshot to snap/]
    F1 -->|install| F1DB[Follower's bbolt truncates log to 41200000]

Snapshots are stored in snap/. Snapshot files carry the first and last log index in the filename; the default snapshot count retained by the WAL is --snapshot-count (default 100,000).

The fsync boundary

etcd’s per-write commit latency is dominated by the WAL fsync. The bbolt write is buffered; it does not fsync per operation. The frequency of bbolt fsyncs is tunable but defaults to a low rate because bbolt’s recovery on disk corruption is to truncate to the last good checkpoint.

The restart sequence

When an etcd member starts:

sequenceDiagram
    participant D as disk
    participant W as WAL replay
    participant B as bbolt open
    participant R as Raft catch-up
    D->>W: latest WAL file(s)
    W->>B: replay entries into bbolt
    D->>B: open bbolt file (mmap)
    W->>R: appliedIndex advanced
    R->>R: catch-up: replicate to leader commit
    R->>R: serve requests

If the member is the leader (e.g. it was the only member left running), the catch-up phase is the leader waiting for followers that aren’t there; once peers reconnect, entries can flow. If the member is a follower, the catch-up phase is replication from the leader up to the leader’s current commitIndex.

What the operator inspects

The four checks that turn “something is slow” into “the WAL fsync on member X is the cause”:

# 1. Disk fsync latency p99
etcd_disk_wal_fsync_duration_seconds_p99

# 2. Backend commit duration p99
etcd_disk_backend_commit_duration_seconds_p99

# 3. DB size and fragmentation
etcd_debugger_mvcc_db_total_size_in_bytes
etcd_debugger_mvcc_db_total_used_in_size_in_bytes

# 4. Apply vs committed index lag
etcd_server_slow_apply_total

Quiz

Production discipline

• Treat /var/lib/etcd as transactional. Don’t copy it while running. Back up only via etcdctl snapshot save.
• Watch WAL fsync latency p99. The metric is the per-write latency floor. Anything above 25 ms deserves a page.
• Plan the data directory capacity. Healthy DB size is < 2 GiB; warning is 8 GiB. Defrag and compaction belong in maintenance windows.
• Bind WAL and bbolt to separate physical disks when possible. A short-stroked NVMe for WAL; capacity for bbolt. The WAL never sees a sustained write above ~10 MB/s.
• Test the restart path. A member that cannot restart is a member the cluster cannot tolerate as a quorum peer. Build a member, kill it cleanly, restart; verify it joins without operator intervention.

The on-disk layout is the foundation of etcd’s reliability. Operating that disk well is what every other Kubernetes control plane benefit inherits.