The trade-offs, and the emergency change that has to exist

The preceding lessons made the case for replacing machines rather than patching them. This one is the other side of the ledger, because a model adopted without its costs understood is a model that gets abandoned loudly eighteen months later.

What it costs

Change latency. A one-character fix to a configuration file becomes a full pipeline: build, validate, publish, roll. Twenty minutes is good; an hour is common. On a mutable fleet the same fix is a file edit and a service reload. That difference is felt most during an incident, which is exactly when people reach for the escape hatch.

The mitigations are real but partial: keep configuration that changes often out of the image entirely - a feature flag, a value the application re-reads - so the pipeline is only in the path for changes that genuinely alter the machine. Layer images so an application update does not rebuild the base. Run bakes in parallel across regions.

Storage and sprawl. Every published image carries a snapshot, every snapshot bills monthly, and a pipeline running per merge produces images faster than anybody classifies them. Retention that is safe requires the reference checks from linux-cloud-images-and-image-pipelines, and building it late is much harder than building it early.

Fleet-wide rebuilds for small patches. A libc advisory now means rebuilding every image and replacing every instance. That is a well-tested procedure, which is a genuine benefit - but it is hours of pipeline and a full rolling replacement for a patch that apt-get upgrade would have applied in ninety seconds. Fleets with a high patch cadence feel this constantly, and the common compromise is a hybrid: replacement for application changes, in-place patching for urgent security fixes, followed by an image rebuild that makes the patch permanent.

Debugging changes shape. You cannot fix the machine in front of you, and the machine in front of you is going to be deleted. That forces a discipline - collect evidence, reproduce in a new instance, fix the image - which is better engineering and slower. It also costs the incidental learning that comes from repairing a system by hand, so the knowledge has to be transferred into the image and the runbook deliberately.

Where replacement is the wrong approach

Be specific about this rather than treating it as a purity question.

• Very small fleets. For three servers, a configuration management repository gives you most of the reproducibility for a fraction of the machinery. An image pipeline is fixed cost, and it amortises over instance count.
• Physical hardware and appliances. A machine that cannot be launched from an image cannot be replaced by one. The image discipline still helps - a documented, automated build - but the deployment model is different.
• Licensed software keyed to hardware or an install identifier. Replacement consumes or invalidates activations.
• Regulated and validated systems. Where a change requires formal requalification, “replace the instance” is a qualification event, not a deploy. That does not rule out images; it rules out replacing casually.
• Exploratory and development environments, where the whole value is a fast edit loop. Immutability there buys reproducibility nobody is asking for.

The emergency change

There will be a moment when the fix cannot wait for a pipeline: an active incident, a pipeline that is itself broken, a dependency that is down. An escape hatch that has not been designed will be invented at 3 a.m. by whoever is on call, and it will be invented badly.

Design it. The procedure has five steps and the last two are the ones that matter.

1. Decide whether to take the instance out of service. If the change is risky, deregister it from the load balancer first. If the outage is total, apply and watch.

2. Make the change, as narrowly as possible.

3. Record it on the instance, in a place monitoring can see. This is the step that makes the difference between a temporary fix and a permanent mystery:

sudo tee /etc/immutable-override <<EOF
date: $(date -Is)
operator: $(id -un)
incident: INC-2481
change: raised proxy_read_timeout in /etc/nginx/conf.d/site.conf to 30s
image-change: not yet opened
EOF

4. Open the change against the image immediately - before the incident is closed, while the reasoning is still in your head. Update the line to reference it.

5. Replace the instance once the new image is published, and verify the override file is gone from the fleet.

Detecting overrides across the fleet is one loop:

for HOST in $(cat fleet.txt); do
  if ssh -o BatchMode=yes -o ConnectTimeout=5 "$HOST" \
       'test -f /etc/immutable-override' 2>/dev/null; then
    echo "OVERRIDE: $HOST"
    ssh -o BatchMode=yes "$HOST" 'cat /etc/immutable-override'
  fi
done

$ bash check-overrides.sh

OVERRIDE: web-07.example.com
date: 2026-07-19T02:14:33+00:00
operator: oncall
incident: INC-2481
change: raised proxy_read_timeout in /etc/nginx/conf.d/site.conf to 30s
image-change: not yet opened

That output is a three-week-old fix that is one health check away from disappearing. Finding it on a Tuesday afternoon is enormously cheaper than finding it during the next incident.

Making the emergency rarer

The best response to a frequently used escape hatch is not a stricter policy. It is to remove the reasons it gets used:

• Measure the pipeline. If an urgent image change takes fifty minutes, the hatch will be used. Time each stage and attack the longest one; a fifteen-minute path removes most of the pressure.
• Move volatile configuration out of the image. Anything reachable through a feature flag or a re-read configuration value never needs a bake, so it never needs an override.
• Keep the previous image warm. If the fastest fix is a rollback and rollback is one command, many incidents never reach the point of wanting a hand-edit.
• Review the overrides monthly. Every one of them is evidence about which changes the pipeline is too slow for. That is the most useful backlog input an immutable platform produces.

Knowledge check