Golden images and immutable patterns - the modern approach

Golden images and immutable patterns are the modern approach to server management. Build once, deploy many, replace on change. This lesson covers the concepts and the workflow.

What golden images are

A golden image is a pre-built server image:

• Operating system.
• Application and dependencies.
• Configuration.

The image is built once, tested, and stored. New servers are deployed from the image. Changes are made by building a new image, not by modifying running servers.

Build a golden image with Packer

Packer is not in the Debian or Ubuntu archives — sudo apt install packer fails with “Unable to locate package”. Add HashiCorp’s own repository:

# The key, into its own keyring rather than the global trust store
wget -qO- https://apt.releases.hashicorp.com/gpg |
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

# The repository, scoped to that key with signed-by
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" |
  sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install packer
packer version

The signed-by scoping is the same discipline as any other third-party repository — see linux-third-party-repository-risk. On an Ubuntu derivative, lsb_release -cs returns the derivative’s codename, which HashiCorp does not publish; substitute the upstream Ubuntu codename there.

The template defines the image. Write it in HCL2 — JSON templates are legacy, deprecated since Packer 1.7, and do not support packer init:

# web.pkr.hcl

packer {
  required_plugins {
    amazon = {
      version = ">= 1.3"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

locals {
  timestamp = regex_replace(timestamp(), "[- TZ:]", "")
}

source "amazon-ebs" "web" {
  region          = "us-east-1"
  source_ami      = "ami-12345"
  instance_type   = "t3.medium"
  ssh_username    = "ubuntu"
  ami_name        = "web-${local.timestamp}"   # REQUIRED
  ami_description = "nginx golden image"
}

build {
  sources = ["source.amazon-ebs.web"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nginx postgresql-client",
      "sudo systemctl enable nginx",
    ]
  }

  provisioner "shell" {
    inline = [
      "sudo cloud-init clean --logs --seed",
      "sudo rm -f /etc/ssh/ssh_host_*",
      "sudo truncate -s 0 /etc/machine-id",
      "sudo rm -f /var/lib/dbus/machine-id",
      "sudo rm -rf /var/lib/dhcp/* /var/lib/systemd/random-seed",
      "sudo rm -f /root/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys",
      "sudo find /var/log -type f -exec truncate -s 0 {} +",
      "sudo rm -f /root/.bash_history /home/*/.bash_history",
    ]
  }
}

Build it in three steps, not one:

packer init .        # fetch the amazon plugin
packer validate .    # catch template errors locally
packer build .       # launches a real EC2 instance and costs money

The output is a new AMI with the application installed and its machine identity stripped.

Image hygiene: strip the identity before you bake

The second provisioner is not optional, and it is the step most golden-image pipelines are missing.

A Packer build boots a real machine. That machine generates a machine ID, SSH host keys, a DHCP client identifier and cloud-init state, and all of it is on disk when the image is captured. Without a de-provisioning step, every instance launched from the image is the same machine as far as the rest of the estate is concerned.

What that costs:

• SSH host keys. Every host presents the same key. Anyone who obtains the image, or compromises a single instance, holds the host key for the whole fleet and can MITM SSH to any of them. Operators are trained not to notice, because the fingerprint is “the same as always”.
• /etc/machine-id. journald cannot tell the hosts apart in remote logs, systemd-networkd derives the DHCPv6 DUID from it so instances steal each other’s leases, and any inventory or monitoring system keyed on machine ID collapses the fleet into one host.
• cloud-init state. With /var/lib/cloud populated, cloud-init believes it has already run and skips user-data on first boot.

SSH host keys are regenerated on first boot by ssh-keygen.service or by cloud-init, so removing them is safe. For libvirt or qcow2 images the whole step has a one-liner equivalent:

sudo virt-sysprep -a image.qcow2

Validate the hygiene rather than assuming it. Launch two instances from the finished image and prove they differ:

# Host key fingerprints must differ between the two instances
ssh-keyscan -t ed25519 host-a host-b

# Machine IDs must differ
ssh host-a cat /etc/machine-id
ssh host-b cat /etc/machine-id

Identical output from either command means the image was baked without de-provisioning. Rebuild it; do not fix the instances.

Immutable deployment

For immutable deployment, the running servers are replaced rather than modified:

# New image is available
NEW_AMI=ami-67890

# 1. Pin a new launch template version to the new AMI
NEW_VERSION=$(aws ec2 create-launch-template-version \
    --launch-template-name web --source-version '$Latest' \
    --launch-template-data "{\"ImageId\":\"$NEW_AMI\"}" \
    --query 'LaunchTemplateVersion.VersionNumber' --output text)

# 2. Point the ASG at it - this affects future launches only
aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name web \
    --launch-template "LaunchTemplateName=web,Version=$NEW_VERSION"

# 3. This is the step that actually replaces the running instances
aws autoscaling start-instance-refresh --auto-scaling-group-name web \
    --preferences '{"MinHealthyPercentage":90,"InstanceWarmup":300,"CheckpointPercentages":[10,50,100],"CheckpointDelay":600}'

# 4. Watch it, and keep the abort lever in reach
aws autoscaling describe-instance-refreshes --auto-scaling-group-name web \
    --query 'InstanceRefreshes[0].[Status,PercentageComplete]'
aws autoscaling cancel-instance-refresh --auto-scaling-group-name web

The checkpoints in --preferences are what make this a deploy rather than a gamble: the refresh stops at 10% and 50% and waits CheckpointDelay seconds, giving monitoring time to reject the build before the rest of the fleet takes it. MinHealthyPercentage bounds how much capacity the refresh may remove at once.

The new instances are the new image. The old instances are replaced. There is no “in-place update” to manage.

Rollback

Rollback is deploying the previous image - and it needs the same replacement step, for the same reason:

# The previous launch template version already points at the previous AMI
PREV_VERSION=3

aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name web \
    --launch-template "LaunchTemplateName=web,Version=$PREV_VERSION"

# Replace the fleet - without this, nothing rolls back
aws autoscaling start-instance-refresh --auto-scaling-group-name web \
    --preferences '{"MinHealthyPercentage":90,"InstanceWarmup":300}'

Better still, make the deploy roll itself back. With AutoRollback the refresh reverts to the previous launch template version when the ASG’s health checks or the attached CloudWatch alarms fail:

aws autoscaling start-instance-refresh --auto-scaling-group-name web \
    --preferences '{"MinHealthyPercentage":90,"InstanceWarmup":300,"AutoRollback":true}'

A rollback that only updates the launch template is worse than no rollback at all. During an incident caused by a bad image, the command succeeds, the operator declares the incident mitigated, and every instance carries on serving the broken build. Verify the rollback by reading the AMI ID off a running instance, not by reading the ASG configuration:

aws autoscaling describe-auto-scaling-instances \
    --query 'AutoScalingInstances[].InstanceId' --output text \
    | xargs aws ec2 describe-instances --instance-ids \
    --query 'Reservations[].Instances[].[InstanceId,ImageId]' --output text

New instances are launched from the previous image. Old instances are terminated. The rollback is a deploy, not a revert.

When immutable is appropriate

Immutable is appropriate for:

• Stateless services: web servers, APIs, queues.
• Containers: each container is replaced, not patched.
• Cloud-native: VMs, ASGs, and Kubernetes.

Immutable is not appropriate for:

• Stateful services: databases with persistent data (the data is preserved; the service is replaced around it).
• Custom hardware: special devices that cannot be replaced.
• Regulated workloads: where replacement requires validation.

Knowledge check