Cloud images and the image build pipeline

linux-golden-images-and-immutable-patterns covered baking an image and deploying it by replacement. This lesson is what surrounds that step: where the base image comes from, whether you can trust it, and what has to be recorded so the image is still useful six months later when somebody asks whether it contains a vulnerable library.

What a cloud image is

A distribution cloud image is not an installer. It is a disk image, already installed, published as qcow2, raw or a provider-specific format, and it differs from an ISO-installed system in ways that matter operationally:

• No interactive installer. The filesystem is already laid out, usually as a single small root partition.
• cloud-init is present and enabled. First-boot configuration comes from user-data, not from a person.
• The root filesystem grows on first boot, via the growpart and resizefs modules, to fill whatever disk the instance was given.
• The serial console is enabled in the kernel command line, because that is the only console a cloud instance has.
• The package set is minimal, and there is no desktop, no documentation, and often no man pages.
• There is no machine identity. No SSH host keys, no populated /etc/machine-id - they are generated on first boot.

That last point is the one people rediscover when they build their own image from a running machine instead. The de-provisioning discipline is in the golden images lesson, and it exists because a cloud image starts identity-free and a running system does not.

Verify the base before you build on it

Everything you bake inherits the trust of the image you started from. Publishers sign their checksum files; verify both links in the chain.

# Fetch the image, the checksum list, and the detached signature
curl -fLO https://cloud.example.com/images/debian-13-genericcloud-amd64.qcow2
curl -fLO https://cloud.example.com/images/SHA512SUMS
curl -fLO https://cloud.example.com/images/SHA512SUMS.sign

# 1. Is the checksum list signed by the publisher's key?
gpg --verify SHA512SUMS.sign SHA512SUMS

# 2. Does the image match the list?
sha512sum --ignore-missing --check SHA512SUMS

$ sha512sum --ignore-missing --check SHA512SUMS

debian-13-genericcloud-amd64.qcow2: OK

Both steps are needed and neither substitutes for the other. The checksum alone proves the file was not corrupted in transit; the signature proves the checksum list came from the publisher. Verifying the checksum against a list you downloaded over the same connection as the image proves nothing about an attacker who controlled that connection.

The key itself has to come from somewhere you already trust - the distribution’s keyring package, or a fingerprint you checked out of band. This is the same reasoning as linux-repository-trust-and-gpg, applied one layer up.

Pin the base, and bump it deliberately

An image built from “the latest Ubuntu 24.04 image” is not reproducible. Two builds a week apart start from different bytes, so a regression between them cannot be attributed, and a rebuild of last month’s release does not reproduce last month’s release.

Pin the base by its identifier - the AMI ID, the image digest, the exact filename and checksum - and record the pin in the same repository as the template.

The obvious objection is correct: a pinned base stops receiving security updates. The answer is not to unpin it, it is to make bumping it a scheduled, tested event:

• A job proposes a new base pin on a cadence, as a change to the repository.
• The pipeline builds and validates against it.
• The change is reviewed and merged like any other.

That gives you both properties - reproducible builds and current bases - and it makes “we are three months behind on the base image” a visible open change rather than an invisible default.

The stages

A pipeline that produces trustworthy images has five stages, and most homegrown ones are missing two of them.

1. Validate the template. Free, fast, and it prevents a build failure that has already launched and billed an instance:

packer fmt -check .
packer validate .

2. Build. The bake step, ending in de-provisioning. Covered in the golden images lesson.

3. Validate the image. This is the stage most pipelines skip, and it is the one that stops a broken image reaching production. Run it against an instance launched from the candidate image, and fail the pipeline on any check:

#!/bin/bash
set -euo pipefail

# Provisioning completed, not merely "the machine booted"
cloud-init status --wait --long

# No unit failed to start
systemctl is-system-running --wait || systemctl --failed --no-pager

# Unit files are well-formed
systemd-analyze verify /etc/systemd/system/*.service

# The application answers, not just the port
curl -fsS --max-time 5 http://127.0.0.1:8080/health >/dev/null

# Identity was stripped: these must be absent or empty
test ! -e /etc/ssh/ssh_host_ed25519_key
test ! -s /etc/machine-id

# No credentials were baked in
! find / -xdev -name 'credentials' -path '*/.aws/*' -print | grep -q .

systemctl is-system-running --wait blocks until boot settles and then returns non-zero if anything failed, which turns “probably fine” into a pass or a fail.

4. Record provenance. For every published image, store:

The package manifest is the one that pays for itself. Capture it during the build:

dpkg-query -W -f='${Package}\t${Version}\t${Architecture}\n' \
  | sort > /tmp/manifest.tsv
# RPM: rpm -qa --qf '%{NAME}\t%{VERSION}-%{RELEASE}\t%{ARCH}\n' | sort

When the next advisory lands, the question “which images contain the affected version” is a grep over stored manifests rather than a survey of running instances. A fuller treatment of software inventory is in linux-package-provenance-and-sbom.

5. Promote. The same image artefact moves from development to staging to production by being tagged, never rebuilt. An image rebuilt for production is a different image, and the testing you did in staging applied to something else.

Working on images without booting them

You do not always need a builder instance. libguestfs can modify an image file directly, which is faster and leaves no provisioning traces:

qemu-img info debian-13-genericcloud-amd64.qcow2
qemu-img convert -f qcow2 -O raw base.qcow2 base.raw

virt-customize -a base.qcow2 \
  --install nginx,jq \
  --run-command 'systemctl enable nginx'

virt-sysprep -a base.qcow2

virt-sysprep is the packaged equivalent of the de-provisioning provisioner: it removes SSH host keys, machine ID, logs, shell history and cloud-init state in one step. On a libvirt or on-premises pipeline it is the right default; on a cloud-builder pipeline the explicit shell provisioner is easier to review.

These tools live in the libguestfs-tools package and are not installed by default on most systems.

Retiring images

Images accumulate, each with a snapshot behind it that costs money every month, and the obvious cleanup - delete everything older than N - is a production incident waiting for a quiet week.

Knowledge check