Staging and production artifacts differ

Reported symptoms

●Staging and production run the same image tag (`1.4.2`) but have different `sha256:` digests in the registry
●`docker inspect` on each image shows a different `Created` timestamp but the same tag and the same git SHA in labels
●The SBOMs attached to the two images list different transitive package versions for one critical dependency
●A bug reproduced in production was reported "fixed in staging" — the fix is present in the staging image, absent in the production image
●The CI workflow has two jobs: `build-staging` and `build-production`, each running the full build
●The two builds ran six hours apart; the upstream base image was updated between them
●The deployment manifests reference the image by tag (`1.4.2`), not by digest

Evidence

· `crane manifest --platform linux/amd64 registry.example.com/app:1.4.2-staging` and `...:1.4.2-production` (or `1.4.2` versus `1.4.2`) show different `config.digest`
· `cosign verify --key <pub> registry.example.com/app:1.4.2` succeeds for both tags, but each is signed against a different digest
· The CI log shows `build-staging` ran at `T+0h`, `build-production` ran at `T+6h`; the upstream `FROM` base image was rebuilt and pushed in between
· `docker build --no-cache --pull` from the same Dockerfile with the same context produces a third digest, different from both
· `syft registry.example.com/app:1.4.2 -o spdx-json | jq .packages[].version` differs from the same command against `...:1.4.2-prod`
· The build workflow has no `outputs:` from `build-staging` consumed by `build-production`; each job runs the full Docker build
· `docker history` on the two images shows different layer SHAs at the base layer
· No lockfile is consulted during build; the build does a `go mod download` / `npm ci` that resolves dependencies fresh each time

Diagnosis and resolutionclick to reveal

Root cause

The workflow builds the image once per environment rather than once per release. A release is what is being tested in staging and deployed to production; an environment is just where the binary happens to run. By rebuilding for each environment, the workflow introduces two new sources of non-determinism between staging and production: the wall-clock time (an upstream base image may have been rebuilt in between) and the dependency resolver (without a lockfile hash pinning the cache key, the dependency tree may differ between runs even at the same SHA). The structural failure is the absence of content addressing: the tag `1.4.2` is mutable, and the registry permits the same tag to resolve to different digests over time. Without a digest-pinned deployment, the staging and production environments drift by construction.

Remediation

Build once, tag with a content address, and promote the address. Add a single `build` job at the top of the workflow whose output is the image digest (`docker buildx build --output type=image,push=true --tag registry.example.com/app:${ github.sha }` then capture the digest from the build output, or use buildkit''s attestation). Tag that digest with a human-readable version in a separate promotion job, never rebuild: `crane cp registry.example.com/app:${ github.sha } registry.example.com/app:1.4.2-prod` (or push the same digest under a second tag). Update the production deployment manifest to reference the digest, not the tag. Disable `:latest` and other mutable tags in the registry, or at minimum require image immutability on the prod repository.

Verification

`crane manifest registry.example.com/app:1.4.2` (and any other version tag) resolves to the same digest as `crane manifest registry.example.com/app:${ github.sha }`. Staging and production manifests both reference that digest. Re-running the build for the same commit produces the same digest (because the cache key includes the lockfile hash and the workflow consumes its own prior output). An SBOM diff between staging and production images is empty.

Prevention

Build once, promote by digest, deploy by digest. The image registry is a content store, not a name store: every deployment reference should be an immutable identifier (`sha256:...` or a registry-native equivalent), with the human-readable version tag living in the manifest and being updated only by a promotion job. Enable registry immutability on production repositories so the same tag cannot be pushed twice. For Argo CD, use `ImageUpdater` with digest pinning and disable `:latest` in the manifest generator. For Flux, use the `image-automation-controller` with `digestMirror` rather than tag rewriting. Treat "rebuild for environment" as a structural antipattern and route any request to do so through a review that asks, explicitly, what non-determinism the rebuild is introducing and why.

A tag is a mutable alias; a digest is an address. Two builds of the same source at different times are not the same artifact, and a deployment that references a tag will drift between staging and production by construction. The fix is build once, promote by digest, deploy by digest — and treat rebuild-per-environment as the structural antipattern that it is.