Container image build and push — the immutable artefact

The container image is one of the two artefacts the cluster consumes. It must be content-addressed, signed, and promoted by digest, not by tag. CI’s job in this part of the pipeline is to build the image, push it to the registry, attest it, and land the digest in the rendered manifest. The GitOps controller never rebuilds; it consumes the digest CI published.

Build, push, attest

flowchart LR
    A["Source commit"] --> B["docker buildx build --push"]
    B --> C["Image digest in registry"]
    C --> D["syft or trivy SBOM"]
    D --> E["cosign attest --predicate SBOM"]
    C --> F["cosign sign --keyless"]
    C --> G["Render manifest with digest"]
    G --> H["OCI manifest bundle"]

The build step is a single invocation that produces a content-addressed image and pushes it to the registry. A typical production invocation:

docker buildx build \
  --tag "${REGISTRY}/app:${GIT_SHA}" \
  --tag "${REGISTRY}/app:latest" \
  --platform linux/amd64,linux/arm64 \
  --cache-from "type=registry,ref=${REGISTRY}/app:cache" \
  --cache-to "type=registry,ref=${REGISTRY}/app:cache,mode=max" \
  --push \
  .

The ${GIT_SHA} tag is the per-commit reference. The latest tag is a moving alias and must never be the only reference in a rendered manifest. The push produces an immutable digest in the registry; the tag is just a pointer to that digest.

After the push, two attestations are produced and pushed to the same registry:

cosign sign --yes "${REGISTRY}/app@${DIGEST}"

cosign attest --yes --predicate sbom.json --type spdxjson \
  "${REGISTRY}/app@${DIGEST}"

The image, the signature, and the SBOM all key off the same digest. The controller can verify any of them by digest.

Landing the digest in the manifest

The render step in CI must land the digest, not the tag, in the rendered YAML. For a Helm chart, a templated values file supplies the digest:

image:
  repository: registry.example.com/app
  digest: sha256:8a3f9d2c1e7b6f4a5d9c0e2b1a8f7d6c5b4a3e2d1c0b9a8f7e6d5c4b3a2f1e0d
  tag: v3.2.7

For a Kustomize tree, a patch in the overlay sets the digest:

images:
  - name: app
    digest: sha256:8a3f9d2c1e7b6f4a5d9c0e2b1a8f7d6c5b4a3e2d1c0b9a8f7e6d5c4b3a2f1e0d

Either way, the rendered manifest carries the digest. When the controller syncs, the cluster pulls exactly that image.

Why promotion is digest promotion

The same image digest can be promoted from staging to production without rebuilding. The CI job publishes once to the registry; the values file or overlay is then updated to point to that digest for the next environment. Two manifests, two environments, one immutable blob:

flowchart LR
    A["CI build at commit 8a3f9d2"] --> B["registry.example.com/app@sha256:8a3f..."]
    B --> C["overlays/staging image: digest 8a3f..."]
    B --> D["overlays/prod image: digest 8a3f..."]
    C --> E["Argo CD sync to staging"]
    D --> F["Argo CD sync to production"]

Rebuilding per environment defeats the purpose. If staging builds commit 8a3f9d2 and production builds the same commit a few minutes later, two different digests can land - even with bit-identical inputs, build timestamps and layers change. Promote the digest, not the build.

What the build step does not do

The build step produces one artefact and its attestations. It does not:

• Decide whether the image is correct. CI validation and human approval do that.
• Apply the image to the cluster. The controller does that.
• Tag for human consumption only. Tags are aliases; the digest is what matters.
• Re-render the manifest with a different digest. If the digest in the manifest must change, a new commit updates the values file or overlay.

Production discipline

1. Build once, push once, attest twice. One build produces the image; cosign signs the image and attests the SBOM. Two registry writes, one source artefact.
2. Pin the digest in the rendered manifest. Tags in the manifest are for humans; the cluster pulls by digest.
3. Promote the digest, not the build. Updating the values file or overlay to point at the existing digest moves the same artefact to the next environment.
4. Verify the signature before sync. The controller - or an admission controller - rejects unsigned or wrongly-signed images.
5. No latest in production manifests. A latest tag is a contract that resolves to whatever someone pushed most recently.

Cross-course references

• Containers for Production Sysadmins - Parts XI-XIV cover image supply chain, SBOM generation, and cosign signing.
• Kubernetes for Production Sysadmins - Parts XI-XIV cover image policy admission and signature verification.
• This course, Part LIII (ContainerSupplyChain) - lessons git-cicd-gitops-liii-01 through git-cicd-gitops-liii-06 cover the supply chain that produces this artefact.
• This course, Part XLV (ImmutableIdentity) - covers content addressing and digest promotion across environments.

Quiz