Reproducibility and immutable references — the commit hash as the unit of reproducibility

Reproducibility is the operational property that says: given the same inputs, the system produces the same output, every time. In a version- controlled infrastructure repository, the inputs are the repository contents plus the build environment. The output is the artifact that production runs. The reference that ties the input to the output is the commit hash — and the reference that ties the artifact to production is the artifact’s digest. Both must be immutable, or reproducibility is a fiction.

Mutable versus immutable references

A Git branch is a mutable reference: it is a pointer that advances whenever a commit is added to it. The name main is a label, not an identity; the same label can refer to commit a1b2c3 today and d4e5f6 tomorrow. A lightweight tag is also mutable: it points at a commit, but the tag can be deleted and recreated pointing at a different commit non-maliciously (force-push). An annotated tag is a Git object in its own right and is immutable once created — but the convention in many repositories is to delete and recreate tags, so even annotated tags are not a guaranteed immutable reference.

The only natively immutable reference in Git is the commit hash itself. The hash is a content-addressed pointer: it refers to the exact bytes of the repository at the moment of the commit, and there is no sequence of operations that can make it refer to anything else. Immutability is not retention: if no ref or reflog reaches the commit, Git may eventually prune the object during garbage collection. A hash cannot be made to identify different bytes, but it can stop resolving when the identified object is no longer retained.

flowchart LR
    A["main branch\nmutable pointer"] --> M["commit a1b2c3"]
    B["v1.2.3 tag\nshould be immutable"] --> M
    C["commit hash a1b2c3\nimmutable reference"] --> M
    M --> D["repository tree"]

# A branch reference is mutable
git rev-parse main
git checkout main
# ... new commits ...
git rev-parse main   # different hash now

# A commit hash is immutable
git rev-parse a1b2c3
# ... time passes, branch moves ...
git rev-parse a1b2c3   # still a1b2c3

Pinning as a discipline

Reproducibility is a chain of immutable references. The chain starts at the commit hash, continues through the artifact digest produced by the build, and ends in the deployment record that names the artifact digest. If any link in the chain is mutable, the chain is broken: the build that produced the artifact is reproducible, but the deployment that runs the artifact is not.

In practice, pinning looks like this in an infrastructure repository:

• Source. Every artifact reference points to a commit hash, not a branch name. CI pipelines read $GITHUB_SHA or equivalent; build manifests in the repo pin dependencies to specific versions.
• Artifact. Every container image is referenced by SHA-256 digest in the Kubernetes manifest, Helm values, or Terraform code. The human-readable tag is generated from the digest as a convenience; the production record references the digest.
• Deployment. Every deployment record (a GitOps commit, a CI run artefact, an Argo CD application) names the artifact digest and the source commit. The correlation is durable.

sequenceDiagram
    participant R as Repository
    participant C as CI build
    participant A as Artifact registry
    participant P as Production
    R->>C: build from commit a1b2c3
    C->>A: push image sha256:abc
    A->>P: deploy image sha256:abc
    Note over R,P: every link references\nan immutable ID

The cost of mutable references

A mutable reference is a deferred choice. The choice is “what does this name mean?” and the answer is “whatever it means at the moment the reference is resolved”. In a development environment, this is useful: a developer wants main to track the latest code. In production, this is dangerous: an operator wants to know exactly what artifact is running, and a mutable reference cannot answer that question.

The cost of a mutable reference in production is not just the possibility of a different artifact being deployed. It is also the loss of the ability to correlate a deployed artifact with a specific build, a specific commit, and a specific review. The chain from production back to a commit is broken at the artifact link, and the audit trail is incomplete.

Production discipline

1. Branches are for development; hashes are for production. A container manifest in production is referenced by digest. A build configuration is pinned by commit hash. A CI workflow is triggered by commit hash, not by branch, when reproducing a build.
2. Tags are advisory, not authoritative. A tag like v1.2.3 is a human-readable convenience; the immutable reference is the commit hash the tag points at. The tag can be deleted and an unreachable commit can later be garbage-collected, so protected refs, repository retention, and backups must preserve release commits intentionally.
3. Promotion is by digest. When an artifact moves from staging to production, the production record references the same digest that was tested in staging. If the staging record says sha256:abc and the production record says sha256:def, the two are not the same artifact, regardless of what the human-readable tag says.

Cross-course references

• Linux for Production Sysadmins - Part XII (RepoSecurity) discusses pinning apt/dnf to a specific snapshot of a repository, the package-management analogue of pinning by digest.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) explains why Ansible roles are pinned to commit hashes in CI.
• Terraform for Production Sysadmins - Part VI (Lockfile) describes how the dependency lockfile gives Terraform a digest-equivalent reference for the provider graph.

Quiz