Configuration management and image baking - choosing the seam

Immutable infrastructure is often presented as a replacement for configuration management. It is not. It moves configuration management to a different moment - and the interesting work is deciding which moment, for each setting, rather than adopting a tool.

There are exactly three moments available.

The decision, per setting

Ask three questions in order, and stop at the first yes.

Is it identical for every instance of this role, and does it change rarely? Bake it. Packages, the application binary, systemd units, the base security configuration, the monitoring agent. Baking it makes the boot faster, removes a runtime dependency on a package repository, and means the thing you tested is the thing that runs.

Is it known at launch and different per instance or per environment? Apply it at launch. The environment name, the database endpoint, the log destination, the cluster to join. This is what keeps one image usable in staging and production - promote by tagging, not by rebuilding.

Does it have to change while the instance is alive? Only then does run-time convergence enter. And it is worth pressing on the question, because the honest answer is usually “no, but replacing the fleet takes 40 minutes and I want it now” - which is a pipeline speed problem, not a configuration model problem.

Configuration management as a build step

The playbooks and manifests that used to converge live servers are usually the fastest way to build an image, and reusing them means one definition rather than two. Packer runs them as provisioners:

build {
  sources = ["source.amazon-ebs.web"]

  provisioner "ansible" {
    playbook_file = "./site.yml"
    extra_arguments = ["--extra-vars", "build_mode=true"]
  }

  provisioner "shell" {
    inline = ["sudo ansible --version"]
  }
}

Before you assume the playbook works in that context, check what it actually does:

ansible-playbook --syntax-check site.yml
ansible-playbook --list-tasks site.yml

Then be honest about the differences, because a playbook written for convergence carries assumptions a build machine breaks:

• Facts. A playbook that branches on the target hostname, its IP address or its environment tag gets the builder’s values. The image then carries decisions made about a machine that existed for eleven minutes.
• Handlers that restart services. Harmless on a live host, meaningless in a build - and worse, a service started during the build may write state that ends up in the image.
• Secrets. A playbook that fetches a secret and writes it to disk has just baked it into the image, for every instance, forever. Secrets are a launch-time or run-time concern; see linux-cloud-identity-and-instance-roles.
• Anything that assumes network identity. Registering with a service discovery system, joining a domain, requesting a certificate. All of these belong at first boot, not in the image, because the image is not a machine.

The usual structure is a build_mode variable that skips the tasks that need a real instance, plus a first-boot role that runs only those. Two entry points, one body of code.

Convergence as a detector, not a remediator

On an immutable fleet, a convergence tool still has a job: it tells you when a machine has diverged from what its image says it should be. Run it in check mode, on a schedule, and treat any diff as a signal.

ansible-playbook --check --diff -i inventory site.yml

$ ansible-playbook --check --diff -i inventory site.yml

TASK [nginx : deploy site config] **********************************
--- before: /etc/nginx/conf.d/site.conf
+++ after: /etc/nginx/conf.d/site.conf
@@ -3,7 +3,7 @@
-    proxy_read_timeout 30s;
+    proxy_read_timeout 5s;
changed: [web-03]

PLAY RECAP *********************************************************
web-03    : ok=24   changed=1    unreachable=0    failed=0

--check predicts changes without making them and --diff shows them; together they answer “has anything on this host diverged from the definition”. On a mutable fleet you would follow that with a real run. On an immutable fleet a non-zero changed count means something wrote to a machine that should have been replaced instead, and the response is to find out what wrote it, fix the image, and replace the instance.

Puppet’s --noop and equivalents in other tools serve the same purpose. The point is the mode, not the tool.

What to do when you genuinely need a live change

Sometimes a setting has to change now and the pipeline is thirty minutes long. That is a real situation and it deserves a real answer rather than a rule that gets broken quietly:

• Prefer a mechanism designed for it. A feature flag, a configuration service the application re-reads, a value in a parameter store fetched on a timer. These change behaviour without changing the machine, so the image stays authoritative.
• If you must change the machine, treat it as a temporary measure with an expiry: record it, mark the instance, and open the change against the image immediately. That is the emergency path, and it is covered in linux-immutable-trade-offs-and-emergency-change.

The distinction that matters: changing an application’s behaviour through data it reads is not mutation of the machine. Editing a file on the machine is.

Knowledge check