What an orchestrator actually does — the reconciliation loop · Docker & Containers

The previous lesson listed the triggers for outgrowing single-host Compose. This one is about what you are actually adopting when you act on them — because the three orchestrators differ wildly in size and ecosystem, and are identical in the one mechanism that changes how you work.

That mechanism is a control loop. Everything operationally strange about orchestrators, and everything they cost, comes from it.

Imperative and declarative

Two different things the word 'deploy' can mean

Imperative. docker run is an instruction: create this container, now. The daemon does it and forgets why. If the container dies, nothing knows what it was supposed to be, except in the narrow sense of a restart policy that says “start this exact container again”.

Declarative. docker service create or kubectl apply submits a desired state to a store: “there should be four replicas of this image, with these resources, on nodes matching these constraints”. A controller then runs, forever:

Observe. What is actually running?
Diff. How does that differ from the desired state?
Act. Create, destroy, or move whatever closes the gap.
Go to 1.

Nobody tells the controller to do this. It is not triggered by your command; your command only changed the stored state. The loop was already running and will keep running after you log off.

The consequence people meet on day one is that imperative commands stop working. Kill a container that an orchestrator is managing and it comes back within seconds, on that node or another:

Service impact possiblefutile— Removing a managed container does not remove it.

$ docker rm -f myapp.3.qk8x1v2w9r7t

myapp.3.qk8x1v2w9r7t

$ docker ps --filter label=com.docker.swarm.service.name=myapp --format '{{.Names}}'
myapp.3.n4d7p1x8s2gv

Illustrative output

Same task, new container ID, four seconds later. This is not a bug and it is not something to fight. To stop a service you change the desired state — scale it to zero, or remove the service object — and the loop does the rest.

What the loop is good at

The loop is why an orchestrator can offer things a single host cannot:

Self-healing. A node dies, the controller observes that three of four replicas exist, and places the fourth somewhere else. No human involved.
Rolling updates. Changing the image in the desired state makes the diff non-zero for every replica; the controller works through them at whatever parallelism you configured, checking health as it goes.
Rescheduling. Draining a node for maintenance is a desired-state change (“no work on this node”), and the loop moves everything off it.
Convergence after partial failure. A deploy interrupted halfway leaves the desired state intact, so the loop finishes the job when the control plane comes back.

None of that is free.

What the loop costs

The four obligations you take on

1. You now operate a consensus store. The desired state has to survive node failure, so it lives in a replicated database using Raft — SwarmKit’s own store, or etcd for Kubernetes, or Nomad’s Raft. That store needs:

An odd number of members, and quorum arithmetic you can do under pressure.
Backups, and a tested restore. A cluster whose store is lost has no desired state, which means it has no idea what should be running.
Its own monitoring: Raft leader elections, commit latency, disk fsync latency. Consensus stores are unusually sensitive to slow disks.

2. Every operation becomes asynchronous. docker service update and kubectl apply return when the desired state was accepted, not when reality matches it. A deploy that returns 0 has told you nothing about whether the application is running. Every deploy script needs an explicit wait-and-verify step, and every “it deployed fine” needs to be treated as an unverified claim.

3. Certificates and node identity. Nodes authenticate to the control plane with certificates that expire and rotate. Swarm defaults to a 90-day node certificate validity. Kubernetes has its own set — kubelet client certs, API server certs, service account signing keys. A cluster that has been switched off for longer than the certificate lifetime may not come back without manual intervention.

4. A second system to debug. When something is not running, the cause is now either the application or the control plane, and telling them apart is a new skill. “The container is not starting” and “the scheduler cannot place the task” look identical from the outside and have nothing in common.

What happens when the scheduler cannot place a task?

Nothing. That is the problem.

You ask for four replicas with a memory reservation of 8 GB each, and the cluster has three nodes with 8 GB free. Three tasks start. The fourth sits in a pending state indefinitely. There is no error, no crash, no log line from the application — because the application never ran.

docker service ps myapp --no-trunc shows the pending task and its error; kubectl describe pod shows scheduler events explaining which predicate failed. Neither is anywhere near where a single-host operator would look, which is the container logs of a container that does not exist.

Unschedulable work is the most common orchestration incident and the one least visible to monitoring built for single hosts. Alert on the gap between desired and running replica counts, not on container health.

What an orchestrator does not do

It is worth being blunt, because this is where most disappointment comes from.

Problem	Does an orchestrator solve it?
A node dies and stateless replicas must move	Yes
Rolling out a new image without downtime	Yes
Spreading work across hosts by resource fit	Yes
Making a single-writer database highly available	No — that is the database’s replication and failover, not the scheduler’s
Data that must follow a container between nodes	No — you need shared or replicated storage, which is its own project
An application that cannot run two copies at once	No — it makes it worse, because the loop wants to run two during an update
Sticky sessions held in local memory	No
Deciding what “healthy” means	No — you still write the health check, and a wrong one causes rolling restarts of a working service

An orchestrator schedules processes. It does not make an application distributed, and pointing one at a stateful monolith usually produces a less reliable system than the single host it replaced, because there are now two things that can fail.

Imperative and declarative

What the loop is good at

What the loop costs

What an orchestrator does not do

Knowledge check