medium riskservice affecting~40 min

Runbook: Manage Module Versions and Releases

1 · Prerequisites

Confirm every item is in place before any state change.

Module Structure and Conventions
Designing a Module Interface
Module Sources and Versioning
Module Releases and Upgrades
Testing Modules with terraform test
Upgrading Internal Modules Safely
Write access to the module repository and to whatever publishes its tags, plus the ability to run a plan against at least one real consumer stack in a non-production environment.
A list of the consumers of this module and the version constraint each of them uses. Without it, the blast radius of the release is unknown.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The default branch is protected and the change is merged, not sitting in a local working tree. A tag pushed from an unmerged branch names a commit that is not in the history everyone else reads.
· The consumer list and their constraints are in hand — grep -rn "modules.git//vpc" . across the consuming repositories, or the registry usage report if there is an internal registry.
· The intended semver bump is written down before the tag is composed, with the specific interface change that justifies it named alongside.
· The CHANGELOG entry exists and includes a Migration section if anything in the release is breaking.
· terraform test passes on the merge commit, and the assertions that cover the changed behaviour have been read rather than merely counted.
· A downstream plan has been run against a real consumer stack in a non-production workspace, and the number of proposed replacements is known.
· The version number is not already published. Tags are immutable; re-tagging a published version is the one action in this runbook that has no clean recovery.
· The signing key for the tag is available to whatever creates the tag, and the tag will be created by the release pipeline rather than by hand where that pipeline exists.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Establish who picks this release up without being asked. A consumer pinned to an exact tag adopts nothing until a human edits it; a consumer on a registry constraint that admits the new version adopts it on their next terraform init -upgrade. Those two populations need different amounts of care.
2Classify the change against the module interface: inputs removed or renamed, defaults changed, types tightened, outputs removed or retyped. Every one of those is MAJOR, whatever the size of the diff that produced it.
3Classify the change a second time against the downstream plan. A change that forces resource replacement in a consumer stack is breaking even when the interface is untouched — that is the class semver alone cannot see.
4Write the CHANGELOG entry before writing the release. If the entry cannot be written without the word "and", the release is doing two things and should be two releases.
5Write the Migration section for every breaking item, in the form of the exact edit a consumer makes to their own configuration.
6Run terraform test against the module and confirm the assertions that cover the changed behaviour actually exercise it. A green suite that never touches the changed input proves nothing about this release.
7Prove it on a real consumer: point a non-production copy of an actual consumer stack at the release commit, terraform init -upgrade, and read the plan. Count the replacements. Zero is the expected answer for a MINOR or PATCH release.
8Open the release pull request with the version bump, the CHANGELOG entry and the downstream plan output attached, and have it reviewed by someone who did not write the change.
9Tag the merge commit with a signed, immutable semver tag and push the tag. The tag is created from the merged commit on the default branch, never from a local working tree.
10Announce the release to the consumers you listed in step 1, naming the bump, the migration if there is one, and the replacements the downstream plan showed if there were any.
11Watch the first consumer adoption. The first stack to upgrade is the pilot and it should be one you own — not the team that finds out from a plan on a Friday afternoon.

4 · Verification

Confirm the procedure actually fixed the problem.

✓git tag --list "v*" | sort -V shows the new tag in the expected position in the sequence, with no gap and no re-used number.
✓git tag -v v4.2.0 verifies the signature, so a consumer can establish that the release came from the team and not from someone with push access.
✓git log --oneline v4.1.0..v4.2.0 lists exactly the commits the CHANGELOG entry describes. A commit in that range that is not in the CHANGELOG is an unannounced change.
✓A fresh consumer checkout resolves the new version: change the pin, rm -rf .terraform/modules, terraform init -upgrade, and confirm the resolved module is the tag you expect rather than a cached copy of the previous one.
✓The downstream terraform plan against a real consumer stack proposes zero replacements for a MINOR or PATCH release. Any replacement means the bump is wrong and the release should not have shipped as one.
✓The CHANGELOG entry renders correctly wherever consumers actually read it — the repository front page or the registry listing, not only the raw file.
✓The announcement has reached the consumer list from step 1, and the pilot consumer has upgraded and applied successfully.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶A published tag is never deleted and never moved. The rollback for a bad release is a new release that reverts the change: git revert the offending commit, bump the patch or major appropriately, and tag again.
↶POINT OF NO RETURN: git push --force on a tag that any consumer has already resolved. Terraform caches modules under .terraform/modules, so a moved tag leaves one consumer running the old code and another running the new one, both pinned to the same version string, and the difference only surfaces on a fresh init in whichever environment initialises next.
↶Deleting a tag is worse than moving it. Every consumer pinned to that ref fails their next terraform init with a fetch error, which converts your release mistake into their outage.
↶If the bad release was a MAJOR, the revert release is also a MAJOR. Consumers who already migrated to the broken interface must not be broken a second time by the fix.
↶If a consumer has already applied the bad release, the rollback for that stack is a consumer-side change: pin back to the previous tag, terraform init -upgrade, read the plan, and check for replacements before applying. Reverting the module does not revert their infrastructure.
↶A release that has not yet been tagged rolls back for free — close the pull request. That is why the tag is the last step and not the first.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the module owner and the consuming teams before publishing any MAJOR release. A major bump is a piece of work you are scheduling in someone else's backlog, and it needs their agreement rather than their surprise.
· Escalate to the consuming team immediately if the downstream plan proposes replacements you did not expect. Do not tag first and investigate afterwards; the tag is the part that cannot be withdrawn.
· Escalate to the security team if a tag was moved or force-pushed, whatever the intent. Consumers can no longer prove that the code they resolved is the code that was reviewed, and that is a supply-chain question rather than a versioning one.
· Escalate to the platform owner if consumers are pinned to a moving branch. That is not a release problem to solve one tag at a time; those consumers are adopting every commit, released or not.
· Escalate to the incident commander if a bad release has already been applied in production by a consumer. The release repository is no longer the incident — their infrastructure is.

Publishing a module version takes two commands and about four seconds. Everything difficult about it happens before those four seconds, because afterwards the version is not yours any more.

Two facts set the shape of this runbook. The first is that a tag, once any consumer has resolved it, is permanent in practice — the recovery for a bad release is another release, never a correction of the one you published. The second is that the blast radius of a release is not visible in the module’s own diff. It is visible in the plan a consumer gets, and the two can disagree completely: a three-line change to a resource block can be a no-op in the module repository and a proposal to replace forty production resources in somebody else’s workspace.

So the work is classification, then proof, then the four seconds.

Who adopts this without being asked

Before anything else, know which of your consumers gets this release whether or not they read the announcement.

How the consumer references the module	Who adopts the new version
`?ref=v3.4.0` on a Git source	Nobody, until a person edits the ref
A commit SHA on a Git source	Nobody. Immutable and unambiguous, at the cost of meaning nothing to a reader in a year
Registry source with `version = "3.4.0"`	Nobody, until a person edits the constraint
Registry source with `version = "~> 3.4"`	Everyone on the next `terraform init -upgrade`, for any 3.4.x you publish
Registry source with no `version` at all	Everyone, on any release, including a major
`?ref=main` or any branch	Everyone, on every commit, released or not

The bottom three rows are where a mis-classified release does its damage. The consumers there are not upgrading; they are being upgraded, and they will discover the release in a plan they ran for an unrelated reason.

Read-only / Safebuild the consumer list before you release

# Across the checked-out consuming repositories:
grep -rn --include='*.tf' 'terraform-modules.git//vpc' /srv/repos/ \
| sed 's/.*ref=//' | sort | uniq -c | sort -rn

# Anything pinned to a branch rather than a tag is an immediate finding.
grep -rn --include='*.tf' 'ref=main\|ref=master' /srv/repos/ || echo 'none'

Step 1: classify against the interface

The interface is the set of promises a consumer’s configuration depends on: the inputs it passes, the defaults it relies on not changing, and the outputs it reads.

Change	Bump	Why
Renamed an input	MAJOR	Every consumer setting it breaks at plan time
Removed an input	MAJOR	Same, and there is no forwarding address
Changed a default a consumer relies on	MAJOR	The consumer’s configuration is unchanged and their infrastructure moves anyway
Tightened an input type or added a validation	MAJOR	Configurations that were valid yesterday are rejected today
Removed or retyped an output	MAJOR	Breaks the consumer expression that reads it
Added an optional input with a default	MINOR	Existing configurations are unaffected
Added an output	MINOR	Nothing that worked stops working
Internal refactor, tags, documentation	PATCH	The interface and the resulting infrastructure are both unchanged

The question that settles every row: does a consumer who changes nothing get a different outcome? If yes, it is MAJOR, no matter how small the diff.

Step 2: classify against the downstream plan

The interface classification catches breaks Terraform reports at plan time. It misses the more expensive class entirely.

Step 3: prove it, twice

First against the module’s own tests. terraform test runs .tftest.hcl files, each run block executing a plan or an apply against the module and checking assert conditions.

Read-only / Safethe module's own contract

cd /srv/repos/terraform-modules/vpc
terraform test

# Read which assertions actually cover the change you made. A suite that
# never sets the input you changed is green for reasons unrelated to
# this release.
grep -rn 'assert' tests/ | head -20

Then against reality, which is the step people skip.

Read-only / Safea real consumer, a real plan, in a non-production workspace

# A copy of an actual consumer stack, pointed at the release commit.
cd /srv/repos/consumer-platform-staging
terraform workspace show

# Edit the module ref to the release commit or the release branch, then:
rm -rf .terraform/modules
terraform init -upgrade
terraform plan -input=false -no-color -out=upgrade.tfplan

# The number that decides whether this release ships.
terraform show -json upgrade.tfplan \
| jq '[.resource_changes[]
       | select(.change.actions | index("delete"))] | length'

Zero is the expected answer for a MINOR or a PATCH. Anything else means the release is breaking regardless of what the interface says, and the choices are to make it a MAJOR, to add the moved blocks that keep the addresses stable, or not to ship it.

The rm -rf .terraform/modules is not superstition. Terraform caches resolved modules there, and a stale cache is exactly how a release gets verified against the previous version’s code.

Step 4: write the CHANGELOG before the release

Configuration changeCHANGELOG.md

## [4.2.0] - 2026-08-19

### Added
- Optional `tags` input, merged into every resource the module creates.
- Output `vpc_arn`.

### Changed
- Default for `enable_flow_logs` is now `true`.

### Migration
- Consumers relying on flow logs being off must set
`enable_flow_logs = false` explicitly before upgrading.
- Downstream plan on a representative stack: 3 in-place updates,
0 replacements.

Writing the entry first is a design review disguised as documentation. If the entry needs the word “and” to describe the release, the release is doing two things, and two things released together cannot be reverted separately.

The Migration section is written as the exact edit a consumer makes to their own file. “Update your configuration accordingly” is not a migration; it is the author asking every consumer to redo the analysis the author already did.

Step 5: the four seconds

Configuration changetag the merged commit, signed

# On the merged commit on the default branch — never a local branch.
git checkout main
git pull --ff-only
git log --oneline -3

git tag -s v4.2.0 -m "Release v4.2.0"
git push origin v4.2.0

git tag -v v4.2.0

The signature is what lets a consumer establish that the code they resolved is the code your team reviewed. Where a release pipeline exists, the pipeline creates the tag: a tag made by hand is a release that skipped whatever the pipeline checks.

Then announce it, to the list built at the start — the bump, the migration if there is one, and the downstream plan result. A release nobody was told about is adopted by accident.

Step 6: rolling back is rolling forward

Configuration changethe only supported rollback

# Substitute the commit that shipped the bad release:
BAD_COMMIT=a1b2c3d4

git switch -c release/v4.2.1 main
git revert --no-edit "$BAD_COMMIT"

# Add the 4.2.1 CHANGELOG entry naming exactly what was reverted, then:
git add CHANGELOG.md
git commit -m "changelog: v4.2.1 reverts the v4.2.0 default change"

# Review and merge as normal, then tag the merge commit:
git tag -s v4.2.1 -m "Release v4.2.1 (reverts v4.2.0)"
git push origin v4.2.1

The bad tag stays where it is. It is the record that the release happened, which is what a consumer debugging their own plan next month needs to find.

One asymmetry worth stating: reverting the module does not revert anybody’s infrastructure. A consumer who already applied the bad release has real resources in the shape the bad release produced. Their rollback is their own change — pin back, init -upgrade, read the plan, and check for replacements before applying.

Common patterns

Symptom	Cause	Response
Consumers report unexpected replacements after a patch bump	Classified against the interface only	Publish a corrected release; add a downstream plan to the release gate
A consumer resolves a version different from a colleague’s, same pin	A tag was moved	Treat as supply chain: re-tag as a new version, tell every consumer to clear `.terraform/modules`
`terraform init` fails for consumers after a release	A tag was deleted or the repository was renamed	Restore the reference. Never solve a release problem by removing a ref
Consumers never adopt new releases	No announcement, or a MAJOR nobody scheduled	Announce with the migration; agree major bumps with consuming teams before publishing
The release breaks one consumer and not the others	The proving stack was not representative	Pick the pilot for coverage, not convenience
Version numbers keep skipping	Tags created by hand, out of the pipeline	Move tag creation into the release pipeline
A plan differs between two consumers on the same tag	A cached module directory	`rm -rf .terraform/modules` then `terraform init -upgrade`, and compare again

Why this matters

Module versioning looks like bookkeeping and behaves like an API. The number you choose is the only signal a consumer’s constraint can act on, and a mis-chosen number does not produce a wrong document — it produces an upgrade nobody approved, on a machine nobody is watching, at a time nobody chose.

Two habits carry most of the weight. Classify twice: once against the interface, which tells you what a consumer must edit, and once against a real downstream plan, which tells you what their infrastructure will do. Those two answers disagree far more often than they should, and the second one is where the outages live.

And treat the tag as the last thing that happens, not the first. Every step before it is free to undo; the moment it is pushed, the only direction left is forward.