Why distributed systems need an orchestrator

A single host running a single container is a solved problem. A production estate running hundreds of containers across tens of hosts is not a solved problem — it is a recurring series of incidents until the operator introduces an orchestrator. This lesson sets up the failure modes that drive every Kubernetes design decision that follows.

What breaks when you scale containers by hand

Running containers manually with docker run, docker compose up or systemd units on a fleet of hosts works up to about a dozen services. Past that, the operator starts paying a tax in incident time that grows faster than linearly:

flowchart LR
    A[1-5 services / 1 host] --> B[5-20 services / 2-3 hosts]
    B --> C[20-100 services / 5-20 hosts]
    C --> D[100+ services / 20+ hosts]

    A --> A1["Trivial: one host, one script"]
    B --> B1["Drift starts: SSH + ad-hoc automation"]
    C --> C1["Operator becomes the orchestrator"]
    D --> D1["Pager fatigue: humans cannot keep up"]

At each step, the symptoms are different but the causes are the same: a human is being asked to do what a control loop should.

Symptom 1: failed containers are not restarted where they should be

A container crashes on host web-04 at 03:00. Without an orchestrator, nothing restarts it. With an orchestrator, a new copy is placed on a different host because the original is unhealthy. This is not “high availability” by accident — it is the orchestrator’s explicit job to track desired replica count against observed running replicas and to act on the difference.

Symptom 2: rolling deploys become manual surgery

Updating 50 containers across 10 hosts means ssh, docker pull, docker stop, docker run. Each step is a place where the deployment can become partial — five containers on the new version, forty-five on the old, and the load balancer is sending 50% of traffic to each. Production systems need atomic-or-rolling semantics; humans do not provide them reliably.

Symptom 3: configuration and secrets drift

Two months in, web-04 has the old TLS certificate because nobody restarted the container after cping the new cert into place. Three hosts have the new DATABASE_URL; seven do not. The orchestrator’s job is to make the configuration declarative — the operator writes “the database URL is X” once, and the system converges every host to X.

Symptom 4: scaling is a project, not a knob

A marketing campaign is sending 10x the normal traffic. The operator must ssh to every host, decide which containers to start, decide where, decide which load balancer to update. With an orchestrator, the operator writes “if CPU > 70%, scale out” — the system handles the rest, including scaling back when traffic returns to normal.

Symptom 5: observability is host-shaped, not service-shaped

Without an orchestrator, “is my service healthy” is a join across every host’s logs, every host’s metrics, every load balancer’s view. With an orchestrator, the service is the unit of identity — the operator asks the system, not the hosts.

What an orchestrator does (and does not) solve

A container orchestrator takes a declarative description of the desired state (“there should be 5 replicas of web:v2.1 behind this service, with these environment variables, on nodes labelled role=app”) and a set of workers (the hosts), and runs a control loop that closes the gap between the two:

sequenceDiagram
    autonumber
    participant Op as Operator
    participant API as API server
    participant Store as State store (etcd)
    participant Ctrl as Controller
    participant Worker as Worker node

    Op->>API: Declare desired state
    API->>Store: Persist
    Ctrl->>Store: Observe (watch)
    Ctrl->>Worker: Reconcile (create/schedule)
    Worker-->>Ctrl: Report status
    Ctrl->>Store: Update observed state
    Note over Ctrl,Store: Loop continues — drift closes automatically

The orchestrator’s responsibilities are:

• Scheduling: which container runs on which host, given constraints (resources, affinity, taints, topology)
• Lifecycle: starting, replacing, restarting unhealthy containers
• Scaling: adjusting replica counts, vertically or horizontally, in response to signals or operator intent
• Configuration and secrets distribution: declarative injection of env vars, files, credentials
• Service discovery and routing: stable network identity (ClusterIP, Service) decoupled from container identity (Pod IP)
• Self-healing: replacing containers that fail their probes, rescheduling when hosts disappear
• Rolling updates and rollbacks: progressive replacement with rollback on failure

What the orchestrator does not solve:

• Application correctness. The orchestrator can guarantee “5 replicas of web:v2.1 are running and reachable”; it cannot guarantee that v2.1 actually works. That remains the operator’s job (tests, canary, observability).
• Stateful data safety. The orchestrator schedules Pods; it does not back up the database they connect to. Storage and backup are separate problems.
• Network reliability inside the cluster. The cluster network is software-defined and can fail in new ways.
• Cost. Schedulers optimise for constraints, not money.
• Disaster recovery of the orchestrator itself. Kubernetes needs its own backup (etcd), HA, and recovery strategy — which the orchestrator cannot provide for itself.

The distributed-systems tax

Every orchestrator is a distributed system. Kubernetes inherits three consequences that shape its design and operation:

1. Partial failure is normal. A 100-node cluster loses a node every few days on average; a 1000-node cluster loses nodes every few hours. The orchestrator must treat node loss as a routine event, not an exceptional one.
2. Network is unreliable. Packets are dropped, partitions happen, latencies spike. The orchestrator cannot rely on synchronous RPCs as if the network were a bus.
3. Clocks drift. Every host’s clock is wrong by some amount; certificate validation, log timestamps, lease expiry, and TLS all depend on clocks being within tolerance. NTP / chrony is a prerequisite for Kubernetes, not an optional nice-to-have.

flowchart LR
    A[Single host] --> B[Multi-host]
    B --> C[Distributed system]
    C --> D[Failures are routine]
    C --> E[Network is unreliable]
    C --> F[Clocks drift]
    D --> G[Orchestrator design]
    E --> G
    F --> G
    G --> H[Kubernetes: API + reconcilers + watch + etcd]

These three properties explain nearly every Kubernetes design choice: the API server is the single funnel for state (so it can serialize decisions), etcd uses Raft consensus (so the cluster state survives partial failure), the scheduler is a separate component (so it can retry without blocking the API), and the kubelet runs every few seconds (so a missed heartbeat does not mean a lost node).

What Kubernetes actually is

Kubernetes is:

• A declarative API for describing desired cluster state (Pods, Services, Deployments, ConfigMaps, etc.).
• A set of controllers that observe the cluster and reconcile the observed state toward the desired state.
• A scheduler that places Pods on nodes based on constraints and resources.
• A worker agent (kubelet) on every node that runs the containers and reports back.
• A storage backend (etcd) that durably persists all API objects.
• A set of conventions (label selectors, owner references, finalizers) that let controllers coordinate without a central brain.

Kubernetes is not:

• A container runtime (it delegates to containerd / CRI-O via CRI).
• A network (it delegates to a CNI plugin).
• A storage system (it delegates to a CSI driver).
• A load balancer (it delegates to a cloud controller or MetalLB).
• A configuration management system (it stores config; it does not render it).

What Kubernetes changes for the operator

Adopting Kubernetes shifts the operator’s day-to-day work:

Each row is a category of work that shrinks. None of them disappears entirely: probes still need to be designed, RollingUpdate parameters still need tuning, secrets still need rotation, HPA targets still need thought, host placement still matters for performance and HA.

Cross-course references

• The Linux course part LXXVIII-Linux-Containers covers the kernel primitives (namespaces, cgroups, OverlayFS) that the orchestrator inherits from the OS.
• The Docker course parts XXV-Docker-Foundations and XXVII-Docker-Install cover the image and runtime model that Kubernetes workloads run on top of.
• The Linux course part XXIV-Linux-Time covers chrony and NTP — a prerequisite for any distributed system that does certificate validation, log correlation, or lease expiry.
• The Observability course part I-Observability-Foundations covers the metrics/logs/traces signals a Kubernetes estate feeds into and the operational difference between monitoring and observability.

Quiz

Production discipline

A container orchestrator is the answer to a specific operational question: how do I run many containers across many hosts without becoming the bottleneck myself? Kubernetes answers that question well, but it is not the answer to every operational question.

Production discipline for Part I:

• Treat the orchestrator as a distributed system with its own failure modes, not as a magic layer that removes them.
• Recognise that Kubernetes is a planner, not an executor for application correctness — your tests, canaries, and observability remain load-bearing.
• Plan for the control plane’s own HA, backup, and recovery before the first workload lands; an etcd that is not backed up is a single point of failure.
• Document the operational differences: what was a runbook before is now a manifest; what was a script is now a controller’s job.
• Accept that adopting Kubernetes is a multi-quarter migration, not a swap.