Split-brain prevention — how Raft refuses to fabricate leadership

“Split brain” is the failure mode where two systems each believe they are authoritative for the same resource. In etcd the resource is the state of the cluster; split-brain would mean two members both accept writes that the other cannot see, producing divergent histories. Raft’s safety property makes this impossible: a leader that cannot reach a quorum cannot commit, and a partitioned “leader” is no leader once a new term is observed.

The safety property Raft guarantees

If a leader has committed a log entry at a given index, no other leader at any future term will commit a different entry at the same index.

This is the state machine safety of Raft. The implementation that produces it:

flowchart LR
    ENTRY[log entry at index N, term T] -->|commit quorum| COMMIT[committed in term T]
    COMMIT -->|new term T+| NEW[any leader in T+ honours the commit]
    NEW -->|attempt to overwrite| REJECT[rejected: term mismatch in voters' logs]

Every voter (follower that voted in this term) tracks the last log index it has seen and the term of that entry. A candidate can only be elected if its log is at least as up-to-date as a majority’s. The constraint:

Candidate.log.lastTerm ≥ follower.log.lastTerm AND Candidate.log.lastIndex ≥ follower.log.lastIndex

A candidate whose log is stale cannot get enough votes to reach quorum. This is the structural guarantee against split-brain.

Why a partitioned “leader” cannot commit

The leader’s authority comes from confirming it has quorum on every commit. Once partitioned away from quorum:

sequenceDiagram
    autonumber
    participant L as OldLeader
    participant F1 as Follower1
    participant F2 as Follower2
    L->>L: receive Put
    L->>L: append to log
    L-->>F1: AppendEntries term5
    F1-->>L: ack index N
    Note over L: only 1 ack quorum 3 unmet
    L-->>L: cannot commit
    F2->>F1: RequestVote term6
    F1-->>F2: vote granted term6
    F2->>F1: AppendEntries term6
    F2-->>F1: leader term6 quorum2
    F2-->>L: heartbeat refused
    L-->>L: steps down to follower

The old leader’s Put(k,v) is rejected because it could not get a quorum acknowledgement. No commit, no split.

What Raft does not prevent

Two things Raft does not prevent are sometimes confused with split-brain:

1. Follower lag. A follower that is slow to apply AppendEntries will serve stale reads if a client asks directly. The API server’s watch cache abstracts this for the cluster, but etcdctl get against a follower can return stale data. The fix for clients that need consistency is the read-index protocol (linearizable reads through the leader).
2. Network partitions with no quorum on either side. A 4-member cluster partitioned 2-2 has no side with quorum. Neither side can commit. The cluster is stuck; restore from snapshot is the recovery. This is not split-brain either; this is quorum loss.

flowchart TB
    A[partitioned 2-2] -->|neither side commits| B[writes reject]
    A -->|partition heals| C[election re-elects one leader]
    A -->|partition persists| D[snapshot restore]

Read-index and consistency

Clients that need “the latest state” must ask the leader. The read-index protocol:

sequenceDiagram
    autonumber
    participant C as Client
    participant F as Follower (any)
    participant L as Leader
    participant Q as Quorum
    C->>F: Read with linearizable
    F->>L: heartbeat read-index
    L->>Q: confirm leadership via heartbeat
    Q-->>L: ack
    L-->>F: read-index = current commitIndex
    F-->>C: state at read-index

A linearizable read forces the follower to confirm the leader’s authority before serving. Without this protocol, a follower could serve stale reads while the cluster has moved on.

Term expiry on partitioned leaders

A leader’s term number is its authority. A leader that receives a message with a higher term immediately steps down. The mechanism:

flowchart LR
    APPEND[AppendEntries with term T+1] -->|compare| CURRENT[leader's current term T]
    CURRENT -->|T+1 > T| STEP[step down to follower]
    STEP -->|persist new term| FOLLOWER[join as follower]

Term expiry is the recovery mechanism: if a partitioned “leader” comes back online after the partition heals, the first message it sends (or receives) with the higher term will demote it. The cluster converges to a single authoritative leader.

How Kubernetes interacts with split-brain

The API server connects to etcd via the client URL. If the leader is unreachable from the API server’s perspective (peer partition, leader crash), the API server times out writes. The Kubernetes control plane does not have its own split-brain risk because the API server is the only writer to etcd (and writes are serialised through the leader):

flowchart LR
    AS1[API server cp-1] -->|writes| E[leader]
    AS2[API server cp-2] -->|writes| E
    AS3[API server cp-3] -->|writes| E
    E -->|acks| AS1
    E -->|acks| AS2
    E -->|acks| AS3

If the API server’s view of the leader becomes stale, the API server times out and errors; the next request retries against the new leader. Kubernetes applications see a briefly-failing API; they do not see divergent writes.

What an operator looks for

The signals that Raft is doing its safety work correctly:

• Steady raft term. The leader’s term does not change for weeks or months.
• No “unstarted” members. A healthy 3-member cluster has all 3 members started.
• Consistent RAFT INDEX across members. All members reach the same applied index within a reasonable lag.
• No “etcdserver: term too old” in API server logs. A healthy client never sees this; it indicates the API server is talking to a stale leader.

The signals that Raft’s safety is being tested:

• leader_changes_seen_total rises. Each change is an election; many elections means instability.
• raft term is incrementing in the metrics. The cluster is going through more than one term per day.
• One member’s raft applied index lags the others. The lagging member is replicating slowly; investigate the disk on that member.

Common operator misconceptions

Quiz

Production discipline

• Steady raft term is healthy. Each term change is an election; each election is a chance for the cluster to have committed nothing for a heartbeat interval.
• Lagging members are not split-brain. They are slow; investigate their disk, network, or runtime.
• Two leaders cannot both commit in Raft. The model is strict: a leader’s authority is from quorum; absent quorum, it cannot commit.
• Linearizable reads go through the leader. Default reads (kubectl get) do not need them; controllers with strong-consistency needs may.
• Operators do not “fix” split-brain. Raft does.

Split-brain prevention is the reason Raft is the consensus algorithm etcd uses; the operator’s job is to recognise the symptoms of fault tolerance being tested, not to attempt manual fixes.