Team-Wide Version Rollout

A team rollout is not a list of apt install commands. It is a change that touches every host an operator touches, and the failure mode of an inconsistent rollout is silent: two engineers see different plans, the CI runner is on a different version than the apply host, and the next incident is a provider-driven diff. This lesson is the operational procedure that prevents that.

The four artefacts of a rollout

A team rollout has four artefacts. If any of them is missing, the rollout is incomplete.

+-----------------------+ +-----------------------+
| Standardised install  | | Pinned version        |
| - apt repo OR binary  | | - 1.9.8 (exact)       |
| - one method, all hosts| | - same on every host  |
+-----------------------+ +-----------------------+
+-----------------------+ +-----------------------+
| ~/.terraformrc        | | Smoke test            |
| - disable checkpoint  | | - version, plan       |
| - plugin_cache_dir    | | - state reachability  |
+-----------------------+ +-----------------------+

The first two are about the binary. The third is about client defaults. The fourth is the verification that proves the rollout landed cleanly.

Standardise the package source

Pick one method and apply it across every host the team owns. The canonical choice for Ubuntu 24.04 and Debian 12 is the HashiCorp apt repository, covered in detail in the lesson on installation methods. The key points for a rollout:

• Document the apt source in the runbook. The exact URL, the exact keyring path, the exact signed-by clause. Do not assume the next person will remember.
• Verify the GPG fingerprint once and pin it. The apt.releases.hashicorp.com/gpg key fingerprint is published by HashiCorp. Record it in the runbook; an unexpected change is an integrity incident.
• Use apt-mark hold terraform on production hosts until the team has tested the next minor.
• Bake the install into the image. New CI runners and new bastion hosts come from a golden image that already has the right version. The rollout is “deploy the new image”; the install is not redone on each host.

Pin the version

The team picks one version and stays on it until the upgrade procedure is run. The pin is in three places:

• The package (apt-mark hold terraform=1.9.8* or the exact binary download).
• The configuration (required_version admits the pin).
• The CI image (the Dockerfile or Packer build pins the same version).

If the three diverge, the rollout is incomplete.

terraform {
  required_version = ">= 1.9.8, < 1.10.0"
}

The lower bound matches the pinned binary. The upper bound is the next minor, which the team will only enter after a deliberate upgrade procedure.

Configure ~/.terraformrc

~/.terraformrc (or %APPDATA%\terraform.rc on Windows) is the client configuration file. Two settings matter for production rollouts.

# ~/.terraformrc - shared client defaults
disable_checkpoint = true

provider_installation {
  filesystem_mirror {
    path    = "/usr/share/terraform/providers"
    include = ["registry.terraform.io/*/*"]
  }
  direct {
    exclude = ["registry.terraform.io/*/*"]
  }
}

disable_checkpoint = true opts the CLI out of the anonymous telemetry that pings checkpoint.hashicorp.com on every command. In a regulated environment this is the difference between a passing and failing compliance audit. It is also the right setting for an air-gapped network where the ping will fail and produce a visible warning on every command.

The provider_installation block is the file-system mirror. The team’s mirror at /usr/share/terraform/providers is populated by the platform team (Ansible, Puppet, or a Packer image) with the provider plugins the team uses. The direct { exclude = ... } line tells the CLI to fall back to the upstream registry only for providers that are not in the mirror. This is the production default for a team that wants reproducible provider resolution without a private registry.

Smoke test

The smoke test confirms three things: the binary is the right version, the CLI can plan against the dev backend, and the CLI can reach the production state backend.

1. Version check. Confirms the binary is what we think it is.

# READ-ONLY
terraform version

Terraform v1.9.8
on linux_amd64

2. Plan against the dev backend. A minimal configuration that targets a development workspace, with no real resources, just to prove the CLI works end-to-end.

# READ-ONLY
terraform init -backend=false
terraform plan

The init -backend=false skips the backend configuration; this is useful when the smoke test runs on a host that should not touch the real backend.

3. State-backend reachability. Confirms the CLI can talk to the configured backend. For S3:

# READ-ONLY
aws s3 ls s3://acme-tf-state-eu-west-1/

For the self-hosted terraform serve:

# READ-ONLY
curl -fsS -H "Authorization: Bearer $TF_TOKEN" \
  https://tf-state.internal.example.com:9080/_health

For Terraform Cloud:

# READ-ONLY
terraform login

A failed reachability check is the most common rollout failure. The host can run terraform version perfectly and still fail to apply because the IAM role, the network ACL, or the bearer token is wrong.

Rollback

A rollback is the procedure for returning every host to the previous known-good version. Two cases.

Case 1: the new version broke a behaviour we depend on.

# SERVICE-IMPACT: downgrade the package on every host
sudo apt-get install -y terraform=1.9.7-1
sudo apt-mark hold terraform

The version must be available in the configured apt repository. If the team pinned to the latest minor, the previous minor may not be in the repository; in that case, the rollback is to rebuild the image with the previous version baked in.

Case 2: the new version broke the install itself.

# DATA-LOSS-RISK: remove the new binary entirely
sudo apt-get purge -y terraform
sudo apt-get install -y terraform=1.9.7-1
sudo apt-mark hold terraform

After any rollback, re-run the smoke test. The version must match, the dev plan must succeed, and the state-backend reachability check must pass. If the rollback did not restore the previous behaviour, the rollback was not the fix.

Production failure modes

1. The team has not standardised the install method. Symptom: a new engineer runs a different install method; the host PATH has two terraform binaries; which terraform returns whichever the PATH prefers. Recovery: standardise on one method; uninstall the other on every host.

2. The required_version upper bound was bumped without an upgrade procedure. Symptom: a CI run fails because the CI image is on the new minor but the laptop fleet is on the old minor; plans diverge. Recovery: revert the upper bound; treat the minor bump as an explicit event.

3. The mirror is out of date. Symptom: terraform init fails with “provider not available in the mirror”; the CLI falls back to direct, the network blocks the registry, and the engineer is stuck. Recovery: the mirror is part of the rollout; when the team upgrades Terraform, the mirror is updated in the same change.

4. The ~/.terraformrc is per-engineer and untracked. Symptom: two engineers get different behaviour from the same configuration because their terraformrc differs. Recovery: ship the canonical terraformrc from the platform team’s dotfiles repository or from a configuration-management tool.

5. The state-backend credentials are missing on the new host. Symptom: the smoke test passes version and plan, fails on state-backend reachability. Recovery: the IAM role, the AWS profile, or the bearer token is not provisioned. Add it to the host bootstrap.

6. The previous version was not retained. Symptom: a rollback is needed; the apt repository only carries the current minor; the previous binary was deleted. Recovery: keep at least one prior version in the repository; for the binary method, keep the prior zip on a known path.

Security and performance

• disable_checkpoint is a compliance setting. Anonymous telemetry is opt-in but the default behaviour in some builds is to call out. Set disable_checkpoint = true in the team’s canonical terraformrc.
• The mirror is a supply-chain boundary. Whatever lands in the mirror is what every host resolves. Verify hashes on mirror upload; reject anything whose hash does not match the upstream registry.
• Provider cache. With the mirror enabled, init is fast on the second run because the plugin is in the local cache. The cache lives in the directory configured by plugin_cache_dir, default ~/.terraform.d/plugin-cache. On a CI runner, point this at a path that survives between jobs.

Production guidance

• One method, one version, one mirror. The team’s hosts should look the same when probed.
• The smoke test is automated. A simple shell script that runs version, plan, and reachability. The script lives in the team’s runbook repository and is run by the host bootstrap.
• The rollback is rehearsed. Quarterly. Pick a host, run the rollback procedure, confirm the smoke test passes on the prior version, then restore.
• The terraformrc is versioned. Shipped from a configuration management tool or a dotfiles repository; never per-engineer.

What comes next

The next lesson covers required_version, the configuration-side constraint that complements the binary pin and prevents an operator with the wrong version from applying at all.

Verification

Run the smoke test on a host you have just rolled out:

# READ-ONLY
terraform version
terraform -install-autocomplete
which terraform

# CONFIGURATION: install autocomplete (re-run is idempotent)
terraform -install-autocomplete

# READ-ONLY: confirm the backend is reachable
curl -fsS -H "Authorization: Bearer $TF_TOKEN" \
  https://tf-state.internal.example.com:9080/_health

A clean output of {"status":"ok"} confirms the auth path is working.