Distributed systems fundamentals every Kubernetes operator must internalise

Kubernetes is not a clever monolith. It is a set of cooperating distributed components — API server, scheduler, controller manager, kubelet, etcd, CNI, CoreDNS — each of which can fail independently, each of which communicates over an unreliable network, and each of which depends on clocks being approximately right. This lesson covers the four distributed-systems concepts that explain the design of every part of Kubernetes.

CAP and PACELC

The CAP theorem says a distributed store can give you at most two of:

• Consistency — every read sees the most recent write
• Availability — every request receives a response (success or failure), even if some nodes are down
• Partition tolerance — the system continues to operate despite arbitrary network message loss between nodes

Network partitions happen. So P is non-negotiable, and the real choice is between C and A.

PACELC extends CAP: in the absence of partitions (Else), the system still trades Latency against Consistency. That is, even on a healthy network, a distributed system must choose between responding quickly or being consistent.

flowchart LR
    AP[AP system] --> A1[Always responds]
    AP --> C1[May return stale data]
    CP[CP system] --> C2[Always consistent]
    CP --> A2[May reject writes during partition]

    E1["etcd: CP + PC (consistency over latency)"]
    E2["DNS resolvers in CoreDNS: AP + PA (latency over consistency)"]
    E3["Event recording in kubelet: AP (always emit; eventually consistent)"]

Kubernetes uses etcd as the source of truth and etcd is CP/PC: during a partition, etcd rejects writes to preserve consistency; on a healthy network, etcd pays the consensus cost (latency) to give every read a consistent view. The Kubernetes control plane accepts the latency tax in exchange for a coherent cluster state.

CoreDNS is the opposite: it is an AP cache. When a partition separates CoreDNS from the API server, CoreDNS continues to serve the records it already has (possibly stale) rather than failing every name resolution. The trade-off is “do I refuse name resolution or do I serve potentially-stale names?” — and for DNS, serving stale is usually the right answer.

Partial failure

A cluster with N nodes fails partially every day. With Kubernetes’ default kubelet heartbeat (10s) and node-monitor- grace-period (50s), a node that misses two heartbeats is marked NotReady. With a 100-node cluster, the expected rate of node loss events is on the order of a few per week; with 1000 nodes, a few per day.

The orchestrator’s design response to partial failure is:

• Idempotent reconciliation. A controller that observes state, decides what should be true, and acts; if it acts twice, the second act is a no-op. This means a dropped network message between controller and API server is not catastrophic — the next reconciliation cycle re-runs the same logic.
• Quorum rules. etcd requires a majority of members to accept a write. A 3-member cluster tolerates 1 failure; a 5-member cluster tolerates 2. Losing quorum means the cluster cannot accept writes.
• Leases and heartbeats. The kubelet renews a lease every few seconds; the API server evicts the lease after a grace period of missed renewals. Leases are how distributed systems detect “this component is gone” without a hard connection drop.

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server
    participant N as Other node

    loop Every 10 seconds
        K->>API: Renew node lease
        API-->>K: ack
    end

    Note over K,API: kubelet process crashes
    API-->>API: 50s elapsed since last lease renew
    API->>API: Mark node NotReady
    API->>API: Pod eviction loop: reschedule Pods from NotReady node
    N->>API: New Pod created on healthy node

Consensus: why etcd uses Raft

etcd is built on the Raft consensus algorithm. Raft’s properties:

• Safety — a value committed to the log is never rolled back, as long as a majority of nodes are non-faulty.
• Availability — the cluster continues to accept writes as long as a majority of nodes (quorum) are reachable.
• Liveness — eventually, if a quorum is reachable, every proposed value is either committed or rejected.

In a Raft cluster, every member has a role at any moment: leader, follower, or candidate. The leader sequences all writes; followers replicate. If the leader is unreachable, the followers hold an election after a randomised timeout.

stateDiagram-v2
    [*] --> Follower
    Follower --> Candidate: election timeout (no heartbeat)
    Candidate --> Leader: majority votes
    Candidate --> Follower: discovered higher term
    Leader --> Follower: discovered higher term

The Kubernetes implications:

• Writes to etcd (every API server write) are committed to a majority of etcd members before the API server returns success. This is the latency floor for any kubectl apply.
• A network partition that splits the etcd members such that no side has a majority halts writes. The API server returns errors for write requests until quorum is restored.
• The leader’s election timeout is configurable in etcd (default 1s). Short timeouts give fast failover at the cost of more spurious elections on a noisy network.

Eventual consistency at the controller layer

etcd is consistent, but the controllers that watch etcd and reconcile state are eventually consistent:

• A controller’s informer lists resources and watches for changes. The watch stream can be interrupted; the informer re-lists on resume.
• A controller’s reconcile loop runs against the observed state in the API server cache, not against the etcd log directly.
• Two controllers can race on the same object; the API server serialises writes via optimistic concurrency (resourceVersion) — the last writer wins, and the loser re-reads and re-tries.

The operational consequence is that a controller’s reconcile is best-effort and re-runnable. A dropped watch event is not a disaster; the next reconcile cycle (typically within seconds) will re-evaluate the same object and converge.

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as API server
    participant Store as etcd
    participant C as Deployment controller
    participant W as Worker

    Op->>API: kubectl apply deployment.yaml
    API->>Store: persist Deployment (resourceVersion=1)
    Store-->>API: committed
    API-->>Op: 200 OK
    Note over C,Store: controller informer watches Deployments
    Store-->>C: Deployment ADDED event
    C->>Store: list Pods with owner ref
    C->>API: create ReplicaSet
    API->>Store: persist ReplicaSet
    C->>WS: schedule Pods via API
    Note over C,W: observe-diff-act loop continues

Leases and timeouts

A lease is a time-bounded claim on a piece of state. The API server’s node object has a lease (node.status.conditions plus the kubelet’s lease object); the controller manager’s leader election uses leases; every controller has a lease for leader election within its namespace in kube-system.

Tuning leases means choosing between two failure modes:

• Lease too long — a failed component appears alive for longer; failover is slower; the system tolerates network blips well.
• Lease too short — a flaky network looks like a failure; failover is fast but spurious.

Production defaults are conservative (kubelet 10s lease, 50s grace). Tune on the slow side if you have noisy networks; tune on the fast side if you have small blast radius and high sensitivity to component downtime.

Clock drift

A distributed system that uses timestamps for ordering depends on the clocks being close. Kubernetes uses timestamps for:

• Event timestamps (admission, scheduling, container start)
• Lease expiry (kubelet, controllers, leader election)
• Certificate validity (API server, kubelet, etcd peer)
• Audit log ordering
• Log correlation across components

A cluster where clocks drift by more than a few hundred milliseconds will see:

• Leases that expire before they should (or after)
• Certificate validation failures
• “TLS handshake error: x509: certificate has expired or is not yet valid” for clients with skewed clocks
• Out-of-order events in kubectl get events

chrony is the de-facto NTP client on Linux. Production clusters run chrony on every node, sync to a stratum-2 or stratum-3 source, and alert on offset > 100ms or root dispersion > 50ms.

chronyc tracking
# Reference ID    : C0A80101 (192.168.1.1)
# System time     : 0.000000234 seconds fast of NTP time
# Last offset     : +0.000012 seconds
# RMS offset      : 0.000234 seconds
# Frequency       : 8.123 ppm fast
# Residual freq   : -0.001 ppm
# Skew            : 0.012 ppm
# Root delay      : 0.001234 seconds
# Root dispersion : 0.000987 seconds
# Update interval : 64.2 seconds
# Leap status     : Normal

Failure domains and blast radius

A Kubernetes cluster inherits the failure domains of its infrastructure:

• Rack — power, network switch, top-of-rack failure
• Availability zone — data centre failure, network partition
• Region — large-scale disaster
• Cluster — control-plane failure, etcd corruption, network partition

The orchestrator’s job is to make workload survival independent of as many of these as possible, by spreading Pods across them. But the cluster itself is a single failure domain for the control plane. That is why production clusters:

• Run a 3-member etcd quorum spread across failure domains
• Run multiple API servers behind a load balancer
• Spread nodes across zones (topology.kubernetes.io/zone) where the cluster spans more than one
• Treat cluster loss as a real scenario (see Part XCVIII)

Cross-course references

• The Linux course part XXIV-Linux-Time covers chrony, NTP, and the operational impact of clock drift on distributed systems — Kubernetes depends on this being right.
• The Linux course part XIX-Linux-NetFoundations covers the network reliability model Kubernetes inherits.
• The Observability course part CIX-Observability-InvestigationWorkflows covers evidence-based diagnosis of distributed-systems failures, which is the mindset Kubernetes incidents demand.
• The Linux course part XXXI-Linux-Audit covers the audit infrastructure that becomes the cross-host timeline of a distributed-system incident.

Quiz

Production discipline

• Treat the cluster as a distributed system with routine partial failure, not as a single computer.
• Configure chrony on every node with a stratum-2/3 source and alert on offset > 100ms.
• Size etcd to tolerate the failures you can afford: 5 members for production HA, 3 only for dev/staging.
• Tune lease and timeout values against your network’s characteristics and document the rationale.
• Audit the clock-skew assumptions every time you add a new TLS-protected path; a clock that was within tolerance for the API server may not be within tolerance for a new admission-webhook.