The build versus rebuild trap — why rebuilding per environment destroys provenance

The build-versus-rebuild trap is the assumption that rebuilding an artifact per environment is equivalent to promoting the same artifact across environments. The assumption is wrong: same source does not mean same bytes. Toolchain drift, OS package versions, timestamp-bearing layers, and other sources of non-determinism make each rebuild a new artifact with a new digest. The bytes that survived staging are not the bytes that survive production; the digest that passed integration tests is not the digest that runs in production.

The trap in concrete terms

A team that rebuilds per environment typically justifies the practice with one of three arguments: ‘the environment needs different config’, ‘the environment needs a different base image’, or ‘the rebuild is faster than pulling’. Each argument assumes that the rebuild produces an equivalent artifact. The assumption is false.

flowchart LR
    A["Commit X"] --> B["Build dev — digest sha256:aaa"]
    A --> C["Build staging — digest sha256:bbb"]
    A --> D["Build prod — digest sha256:ccc"]
    B --> E["Test A passed"]
    C --> F["Test B passed"]
    D --> G["No tests ran"]

The trap is visible in the diagram: three builds, three digests, three independent identities. The tests passed against digest aaa and bbb; the production deployment runs digest ccc. The link between ‘what was tested’ and ‘what runs’ is severed, and the production artifact is the only one that no test covered.

Sources of non-determinism

A container build is not deterministic by default. Sources of non-determinism include:

• Timestamp-bearing layers. A COPY of a file with a future mtime, a RUN date in a build step, a tar archive with non-reproducible metadata.
• OS package versions. A RUN apt-get install that resolves against a snapshot mirror whose index has shifted since the previous build produces a different layer.
• Toolchain drift. A base image rebuild that updated glibc, OpenSSL, or the language runtime produces a different foundation for every dependent layer.
• Build cache state. Two builds with different cache states take different paths through the Dockerfile and produce different intermediate layers.

Each source is small; the cumulative effect is that a rebuild is almost never byte-identical to the original. The bytes that survive staging are not the bytes that survive production; the digest changes; the identity changes; the artifact is a different artifact.

How to break the trap

The fix is build-once-deploy-many. The pipeline builds the artifact once at the commit. The build produces a digest. The pipeline records the digest. Every environment — dev, staging, production — pulls the same digest. The differences between environments are injected at runtime, not at build time.

docker build --tag app:$COMMIT_SHA .
docker push registry.example.com/app:$COMMIT_SHA
DIGEST=$(crane manifest digest registry.example.com/app:$COMMIT_SHA)
echo "$DIGEST" > build/digest.txt

The environment-specific configuration — database URL, feature flags, secrets — is injected by the orchestrator as environment variables or mounted config files. The artifact does not contain the environment; the orchestrator supplies the environment to the artifact. The digest is the same in dev, staging, and production; the bytes are the same; the tests run against the same bytes that run in production.

Reproducible builds as defence in depth

Build-once-deploy-many is the operational discipline; reproducible builds are the engineering discipline that makes the discipline enforceable. A reproducible build is a build whose output bytes are determined by the inputs: the same source, the same toolchain versions, the same base image, the same build flags. Given the same inputs, the build produces the same digest.

Production discipline

1. Build once, deploy many. The pipeline produces a single artifact at the commit; every environment consumes the same digest.
2. Inject environment config at runtime, not at build time. The artifact is environment-agnostic; the orchestrator supplies the environment.
3. Record the digest at build time and reference it everywhere. The deployment manifest pins the digest; the audit trail records the digest; the rollback uses the digest.

Cross-course references

• Git, CI/CD & GitOps — Part XLV-03 (Promotion) establishes the promotion discipline this lesson depends on.
• Terraform for Production Sysadmins — Part XIV (Plan/Apply) applies the same principle to plan files: the plan bytes are not rebuilt per environment.
• Linux for Production Sysadmins — Part XXXIV (ConfigMgmt) covers the runtime-config pattern in package and service management.

Quiz