Immutable infrastructure, and why it does not delete this course

There is a serious argument that this course is solving a problem that should not exist. It goes like this: drift accumulates on hosts that live a long time. Do not have hosts that live a long time. Build an image, deploy it, and when you need a change, build a new image and replace the instances. Nothing is ever modified in place, so nothing can diverge.

The argument is good and the model works. This lesson takes it seriously, states exactly what it solves, and then states exactly what it does not — because the second list is longer than its advocates usually mention, and because the practical answer for nearly every real estate is “both”.

What replacement actually solves

Drift cannot accumulate on a host that lives six hours

This is the central claim and it is straightforwardly true. All five mechanisms from lesson 2 — the 02:00 fix, the test package, the vendor engineer, the partial rollout, the build-time divergence — need time to operate. A host replaced weekly has a bounded window in which anyone can change anything, and the change is erased at the next replacement rather than persisting for eighteen months.

Note that it does not prevent the changes. Somebody can still SSH in at 02:00 and edit a file. What it prevents is that change surviving. That is a meaningfully weaker guarantee than “nobody changes anything”, and it has a sharp edge covered below.

Build-time divergence is solved properly

This is the one convergence genuinely cannot fix, and where the immutable model is unambiguously better.

Two hosts built three months apart from “latest stable” are different in ways no declaration enumerates: transitive dependency versions, kernel point release, the exact contents of a base image. Convergence does not remove this, because it only enforces what you declared, and nobody declares the version of every transitive dependency.

Replacement from a single built artefact removes it by construction. All instances came from the same bytes. That is a stronger identity claim than any convergence run can make.

Rollback becomes a real operation

Configuration management does not give you rollback. Reverting a declaration and re-converging is not the same as restoring the previous state — it applies the old declaration to hosts that have since changed, which is a forward operation that happens to move in a backwards direction.

With images, the previous version is an artefact that still exists. Rolling back means deploying it. That is a genuine restore, and it is one of the strongest practical arguments for the model.

The state is testable before it is deployed

An image can be built, booted, tested and inspected before any production instance is touched. A convergence run is tested by running it on something that resembles production, which is a weaker statement because the thing it resembles is not the thing it will run against.

What replacement does not solve

Something still has to build the image

This is the observation that keeps this course relevant, and it is not a technicality.

An image does not appear. Something installs the packages, writes the configuration, creates the users, applies the hardening baseline and lays down the certificates. That something is configuration management, run against a build host, at build time. The tooling is the same; only the target changed.

flowchart LR
  R["Repository<br/>declared state"] --> B["Build host"]
  B --> I["Image artefact"]
  I --> D1["instance 1"]
  I --> D2["instance 2"]
  I --> D3["instance 3"]
  R -.->|"same declaration,<br/>different target"| B

The declaration does not go away in an immutable estate. It moves from running against a fleet to running against a single build host, and it is still a declaration of desired state that must be written, reviewed, maintained and owned. Every cost in lesson 5 is still paid.

What changes is the blast radius of a mistake, and it changes in both directions: a bad convergence run damages the hosts it reaches, while a bad image damages every instance built from it thereafter — but does not touch the ones already running, which is a genuinely better failure shape.

Configuration that is not in the image

Instances need things the image cannot contain: their hostname, their address, their certificates, their secrets, their membership of a cluster. These arrive at first boot, from cloud-init, a metadata service, a secrets manager, or a configuration run.

That first-boot configuration is a declaration of desired state applied to a running host. The model has not been eliminated; it has been scheduled differently.

Data

An immutable host cannot hold the data that made it valuable. State has to live somewhere that survives replacement — a database, an object store, a network filesystem, a volume that is detached and reattached.

And the thing holding the state is usually not immutable, which is the next section.

What cannot be rebuilt on demand

Four categories, and they are not edge cases. Between them they cover most of what actually breaks at 03:00.

Stateful hosts with large local data

A database server with eight terabytes of local storage is not replaceable in the sense the model requires. You can replace the operating system around the data — some teams do exactly that — but the replacement is a maintenance operation with a runbook, a window and a rollback plan, not a routine deployment.

The distinction that matters: replacement is cheap when the host holds nothing. The cost of replacement scales with the state the host holds, and past some threshold it stops being a deployment mechanism and becomes a migration.

Appliances and vendor-managed systems

A storage array, a hardware load balancer, a SAN switch, a vendor-supported database appliance. You do not build its image. You configure it through the interface the vendor provides, and its state is whatever accumulated in it since it was racked.

These are snowflakes by construction and no amount of tooling changes that. What tooling can do is record their configuration and detect when it changes, which is a convergence-shaped problem.

Network devices

Switches, routers and firewalls run vendor operating systems, are configured through their own interfaces, and cannot be replaced by booting a new instance — the physical device is the device. Ansible manages these through dedicated network modules and a different connection model, which a later part of this course covers.

Note that this is a case where the push, agentless model is a genuine advantage: you cannot install an agent on a switch.

Physical hosts and on-premises estates

Replacement assumes provisioning a new instance is cheap and fast. On bare metal it is neither. Re-imaging a physical server takes a maintenance window, sometimes a person in a data centre, and gives you one host at a time.

Estates with a fixed fleet of physical machines get very little from the replacement model and quite a lot from convergence, which is why on-premises infrastructure is where configuration management remains unambiguously the right answer.

The hybrid estate is the normal case

Almost every real estate has all four approaches running at once, and that is not a failure to standardise.

Read that table as the answer to “which approach should we use”. The question is malformed. The right question is per-tier, and the same team answers it differently for different tiers on the same day.

A test you can apply

For a given tier, ask: if this host became unhealthy right now, would the correct response be to fix it or to replace it?

The answer is usually decided by how much irreplaceable state the host holds, and that is measurable rather than a matter of opinion:

$ df -h --output=target,used,pcent -x tmpfs -x devtmpfs |
grep -vE '^(Mounted|/boot|/$)'

/var/lib/postgresql   7.9T   84%
/var/log              12G     6%

Eight terabytes under /var/lib/postgresql answers the question by itself. Replacing this host is a restore, not a deployment, and no amount of tooling maturity changes the arithmetic. A host whose equivalent output is a few hundred megabytes of logs is a different proposition entirely.

If the answer is replace, the tier wants the immutable model, and you should invest in making replacement fast and reliable rather than in making convergence comprehensive.

If the answer is fix — because replacement would mean restoring eight terabytes, or coordinating a cluster membership change, or a data centre visit — the tier wants convergence, and you should invest in being able to state and verify its intended state.

If the answer is “it depends what is wrong”, you have a hybrid tier, and the honest design is convergence for the routine and a documented replacement procedure for the rest.

Knowledge check