Reproducibility and the Blueprint

The on-call engineer answers the page at 03:00. The audit asks: when a production configuration was re-applied six months later, does it produce the same change set? The answer is either “yes, here are the hashes” or “we do not know.” A team that does not know has a forensics problem, not a configuration problem. This lesson is about producing the answer.

The first principle

The same configuration, against the same state, with the same provider versions, produces the same plan.

Three inputs determine plan output. If any of them change, the plan can change. Reproducibility means controlling all three.

plan output = f(configuration, state, provider versions)

• Configuration is the .tf files in the working directory plus the modules they reference. The configuration is the intent.
• State is the most recent successful apply record. The state lives in a backend. The backend must be the same.
• Provider versions are the plugins Terraform uses to talk to upstream APIs. Different versions can read different attributes from the same resource and therefore produce different plans.

These three inputs are the entire reproducibility surface. If they are all controlled, the plan is reproducible. If any of them is uncontrolled, the plan is not.

The three constraints

Each input has a constraint. Each constraint has a file or flag.

1. Pin provider versions in the configuration

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 5.50.0"
    }
  }
}

The pin says: this configuration is verified against AWS provider version 5.50.0 exactly. terraform init will refuse to use a different version. The exact pin is the strongest guarantee. A pessimistic constraint (~> 5.50) admits every later 5.x release up to but excluding 6.0, and even the tighter ~> 5.50.0 admits every 5.50.x patch. Either way the resolved version can change between runs, which breaks reproducibility.

The rule:

# Reproducible — exact pin
version = "= 5.50.0"
# Acceptable for libraries — pessimistic pin
version = "~> 5.50.0"
# Not reproducible — floating version
version = ">= 5.0.0"

2. Reference modules from immutable sources

module "network" {
  source = "git::https://github.com/example/tf-modules.git//network?ref=v2.3.1"
}

The module source is pinned to a tag (v2.3.1), not a branch (main) and not a commit that moves. A branch is mutable; a tag is, by convention, immutable. A commit SHA is also immutable but harder to read.

# Reproducible — tag-pinned
source = "git::https://git.example.com/tf-modules.git//network?ref=v2.3.1"
# Reproducible — pinned commit SHA
source = "git::https://git.example.com/tf-modules.git//network?ref=3f9d2a1"
# Not reproducible — branch
source = "git::https://git.example.com/tf-modules.git//network?ref=main"

Terraform’s terraform get records the resolved commit in the lock file. That commit is what is reproducible even if the module repository moves on.

3. Commit .terraform.lock.hcl

The dependency lock file records the version and hashes of every provider and module the configuration uses. It is the ground truth for reproducibility.

# .terraform.lock.hcl
# Do not edit. Generated by `terraform init`.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.50.0"
  constraints = "= 5.50.0"
  hashes = [
    "h1:4f8z1q2+k9J3aBcDeFgHiJkLmNoPqRsTuVvWwXyZ0123=",
    "zh:1a2b3c4d5e6f70819a2b3c4d5e6f70819a2b3c4d5e6f70819a2b3c4d5e6f7081",
  ]
}

Three properties of the lock file:

• It is generated by terraform init. Engineers do not write it by hand.
• It is committed to source control alongside the configuration. It does not live in the operating filesystem.
• It enforces the install against the recorded hashes. A provider whose hash does not match is rejected.

The single strongest reproducibility control is committing .terraform.lock.hcl to source control. A team without it has no record of the providers that produced yesterday’s plan.

Reproducibility versus idempotency

Two properties that are easy to conflate.

• Idempotency. Running the same apply twice produces no additional changes. This is a property of the plan output vs. the real world.
• Reproducibility. Running the same configuration at a later date produces the same plan. This is a property of the plan function vs. inputs.

A configuration can be idempotent but not reproducible. A module referenced from main is reproducible only at the moment of commit. A year later, main has moved. The next apply produces a different plan.

A configuration can be reproducible but not idempotent. Two different terraform apply calls against a state with a manual change in between will produce different plans. That is expected; the second plan is reconciling drift.

The lesson: plan reproducibility is the property this lesson teaches; idempotency is a separate (and necessary) property covered in the workflow lesson.

The reproducibility test

The discipline is a one-line test that catches a non-reproducible configuration before production does.

# In a clean directory, with the same configuration and state:
rm -rf .terraform
terraform init
terraform plan -out=baseline.tfplan

# Save the plan summary:
terraform show -no-color baseline.tfplan > baseline.txt

A month later, the same engineer (or a different one) runs:

rm -rf .terraform
terraform init
terraform plan -out=current.tfplan
terraform show -no-color current.tfplan > current.txt

# Compare the two plans:
diff baseline.txt current.txt
# Empty diff: reproducible.
# Non-empty diff: investigate.

The test is the operational definition of reproducibility. If the diff is empty across six months, the configuration is reproducible. If the diff is not, the configuration is not — and the diff is the first place to look.

How to validate the configuration is reproducible

Three checks at PR review time.

# READ-ONLY: is the lock file committed?
git -C infra/ ls-files | grep -E '\.terraform\.lock\.hcl$'

# READ-ONLY: are provider versions pinned exactly?
grep -E 'version\s*=' infra/*.tf | head

# READ-ONLY: are module sources pinned?
grep -rE 'source\s*=' infra/*.tf | head

If any of the three returns “no”, the configuration is not fully reproducible. Each gap has a known fix; the gaps do not need to be tracked as issues.

Production failure modes

1. Provider version drift in CI. CI uses latest. The engineer’s laptop uses 5.50.0. The plan produced in CI is not the plan the apply executes. Production breaks.
2. Module source is main. The module maintainer merged a breaking change at 17:00 Friday. Monday morning’s plan proposes changes that were not in the previous plan.
3. Lock file is gitignored. Every terraform init downloads a different provider version. No hash enforcement. A compromised mirror produces a silent change.
4. Lock file changed without required_providers. The lock file moved to 5.50.1; the configuration still pins to 5.50.0. CI fails noisily, which is correct, but the team clicks through and applies locally.
5. State backend moved. State moved from S3 to Terraform Cloud without the existing configurations picking it up. The state file is empty. The next plan proposes all resources as new.

Security implications

Reproducibility is a security control as well as an operational one. The lock file is a hash-enforced allow-list for provider binaries.

• A compromised Terraform mirror cannot inject a malicious provider that satisfies the recorded hash. The init step fails before any apply runs.
• An audit can reconstruct the exact provider binaries used on any historical apply, by retrieving the lock file at that commit and re-verifying the hashes.
• A reproducible configuration is auditable. A non-reproducible configuration is a guess.

The implication: a team that does not commit .terraform.lock.hcl does not have a reproducible configuration. The configuration cannot be relied on for audit. A change to that policy is a change to the team’s compliance posture.

Performance implications

• Init cost. A clean terraform init downloads all providers and modules. For multi-provider configurations, this is tens of seconds. CI environments that throw away the working directory pay this cost every run. CI caches the .terraform directory.
• Plan cost. Plan cost scales with the resource count, not the lock file. The lock file does not affect plan speed.
• Apply cost. Apply cost scales with the number of resource changes and the parallelism of independent changes. Provider version does not affect apply speed.

If CI is slow, the cache is misconfigured. The lock file is not the cause.

What comes next

The next lesson is drift — the recognition that even a fully reproducible configuration will, over months, diverge from reality, and the discipline that catches it before the next planned change does.

Verification

Six checks at PR review time confirm reproducibility is in force.

# READ-ONLY: the lock file is in source control.
git ls-files | grep -E '\.terraform\.lock\.hcl$'

# READ-ONLY: providers are pinned exactly (no `~>` or `>=`).
grep -E 'version\s*=' *.tf

# READ-ONLY: module sources are tagged or commit-pinned.
grep -rE 'source\s*=' *.tf

# READ-ONLY: provider mirror hashes are recorded.
grep -E 'hashes\s*=' *.tf

# READ-ONLY: the reproduction test passes against a clean
# directory.
rm -rf .terraform
terraform init
terraform plan -out=baseline.tfplan > /dev/null
# A second plan against the recorded state should be empty.

# READ-ONLY: CI enforces the lock file.
# A `terraform init` failure on hash mismatch should fail the
# pipeline, not be downgraded to a warning.

A configuration that passes the six checks is reproducible within its observed inputs. A configuration that fails one check is reproducible only by accident. Fix the failing check before merging the PR.