Raft consensus — leader election, terms, log replication

Raft is the algorithm etcd uses to keep multiple replicas of a key-value store in lockstep. It is a deliberate, choreographed version of “the majority wins”: every change is recorded by a leader, replicated to followers, and acknowledged by a majority before it is considered committed. This lesson walks the pieces — terms, leaders, elections, log entries, commit — and what each one buys and what each one costs.

Raft in one sentence

A replicated log: one leader at a time receives every write; followers replicate the leader’s log entries; a write is committed when a majority of members have acknowledged it. Cluster survives floor(N/2) member losses; it does not survive loss of majority.

Terms

A term is a monotonically increasing integer. Terms form logical epochs:

sequenceDiagram
    participant L1 as Leader (term 5)
    participant F1 as Follower
    participant F2 as Follower
    participant L2 as Leader (term 6)
    Note over L1: term 5 active
    L1->>F1: AppendEntries (term 5)
    L1->>F2: AppendEntries (term 5)
    L1-->>L1: L1 fails / network partition
    F1->>F2: RequestVote (term 6)
    F2->>F1: vote granted (term 6)
    L2->>F1: AppendEntries (term 6)
    L2->>F2: AppendEntries (term 6)

A new term begins whenever a follower concludes the leader is gone and becomes a candidate. Two leaders can exist temporarily during an election; only one wins (the candidate that secures a majority of votes for the new term). A leader that discovers a higher term immediately steps down to follower.

Leader election

flowchart LR
    F[Follower] -->|election timeout| C[Candidate]
    C -->|RequestVote majority granted| L[Leader]
    C -->|higher term seen| F
    L -->|AppendEntries beat heart| F
    L -->|higher term seen| F

Key parameters:

• Heartbeat interval (--heartbeat-interval): how often the leader sends empty AppendEntries to followers. The default for etcd is 100 ms.
• Election timeout (--election-timeout): how long a follower waits without a heartbeat before voting. The default for etcd is 1000 ms.
• Quorum (floor(N/2) + 1): the number of votes (or acknowledgements) needed.

The election timeout is randomised across followers so that two followers do not both become candidates at the same instant; that randomisation is the only thing standing between a healthy cluster and a livelock.

The replicated log

Every write becomes a log entry. The log entry has:

• term: the term in which the entry was created.
• index: the entry’s position in the log (monotonic).
• data: the operation (a Put, a Txn, a DeleteRange, etc.).

The leader appends to its log first; then replicates to followers; then waits for acknowledgements. An entry is committed when a quorum of members has persisted it to their logs; the leader’s commitIndex advances; the followers learn of the advance through subsequent heartbeats.

sequenceDiagram
    autonumber
    participant Client
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    Client->>L: txn If(...){Put(k,v)}
    L->>L: append to log (term N, index 41289312)
    L->>F1: AppendEntries (term N)
    L->>F2: AppendEntries (term N)
    F1->>L: ack index 41289312
    F2->>L: ack index 41289312
    Note over L: quorum ack -> commit
    L->>F1: AppendEntries (commitIndex=41289312)
    L->>F2: AppendEntries (commitIndex=41289312)
    L-->>Client: txn OK

If the leader fails after persisting but before acknowledging, a new leader’s election starts from the last committed index, and the uncommitted entries on the old leader are discarded. That is why cluster state never regresses: a new leader cannot claim to have committed entries that a prior leader only replicated.

What Raft guarantees

etcd’s API documentation defines two model guarantees (both from Raft):

• Linearizable reads. A read served by the cluster reflects at least the most recent write at the time the read was issued. Reads go through a “read-index” protocol on the leader or are routed through the leader to ensure they don’t return stale data.
• Linearizable writes. A write that returns success has been committed to a quorum. Any subsequent read is guaranteed to see that write.

Both guarantees depend on the leader being authoritative. A partitioned “leader” that cannot reach a quorum cannot commit new writes; that is the central safety property that prevents split-brain.

What Raft does not guarantee

flowchart LR
    W[write attempt] -->|leader=| LO[append & replicate]
    LO -->|ack quorum| C[commit]
    LO -->|lost quorum| BLOCK[block: no commit]
    W2[write attempt] -->|leader unreachable| FAIL[no leader: write rejected]

• Not every write succeeds. If the cluster has lost quorum, writes fail; the leader does not fabricate acknowledgements it does not have.
• Not every read succeeds. Reads that require linearizable semantics fail if the leader cannot confirm its leadership to a quorum (the read-index protocol times out).
• Not every member is up-to-date. Followers can lag. Reads from a stale follower return stale data unless the client asks for a linearizable read that goes through the leader.
• Not every committed entry is applied immediately. The Applied Index can lag the Committed Index if a member is busy. A leader with a high raft index but a low raft applied index is a leader that is committing but not keeping up.

$ etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=json 2>/dev/null | jq '.[0]'

{
"Endpoint": "https://10.0.1.10:2379",
"ID": "c5e9a1b2...",
"Status": {
  "leader": "c5e9a1b2...",
  "term": 17,
  "raftIndex": 41289312,
  "raftAppliedIndex": 41289312,
  "raftTerm": 17,
  "dbSize": 1500000000,
  "dbSizeInUse": 880000000
}
}

Why the leader matters

Every write goes through the leader. If a follower receives a write request from a client, it forwards to the leader; if no leader exists, the write fails. The read-index protocol ensures linearizable reads by forcing them through the leader:

1. The follower confirms the leader is still the leader (heartbeat to a quorum).
2. The follower reads its state at the leader’s commitIndex + 1.
3. The follower returns the read to the client.

This protocol is what makes a follower safe to read from; without it, a partitioned “leader” on a minority side could serve stale reads.

Failure modes Raft handles

What Raft asks of the operator

• Time. The election timeout must be larger than the longest expected network round trip; a tight WAN election timeout with packet loss will trigger spurious elections.
• Quorum. The cluster size determines fault tolerance. Three members: tolerate one. Five members: tolerate two.
• Network. Both low-latency (heartbeats) and reliable (the partition model is unforgiving).
• Disk. fsync latency bounds commit latency directly. A member with a slow disk slows every write to the cluster.

UnderTheHood title=“Snapshot install during catch-up”> When a follower has been offline long enough that its log no longer matches the leader’s, the leader sends a snapshot instead of log entries. The snapshot is the compressed state at a known log index; the follower truncates its log up to that index and installs the snapshot. This is what lets a freshly built member join the cluster without streaming the entire log from day one.

Quiz

Production discipline

• Know your quorum arithmetic. 3 members tolerate 1 loss; 5 tolerate 2. Past that, writes halt.
• Tune election and heartbeat for the network. Tight timeouts on a wide-area network will trigger spurious elections; loosen to the largest expected round trip.
• Watch leader changes. Each change is an election. Sustained leader changes correlate with network instability.
• Disk before network. A slow fsync on a single member is the most common etcd regression; check it before chasing the network.
• Linearizable reads on demand. kubectl get does not need them; controllers watching for the most recent state do. Use the right semantics for the right consumer.

Raft is the algorithm etcd uses; understanding it is what turns “etcd is slow” into “the disk fsync on member B is slow”.