Skip to main content
RunBook Academy

← All runbooks in Ansible

medium riskservice affecting~75 min

Runbook: Release a role

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · The list of consumers of this role is known - which repositories, which controllers, which pipelines pin it
  • · The changes since the previous release have been read, not just counted
  • · Whether any change is breaking has been decided against the roles documented interface, not against intent
  • · The full test matrix passes on the commit being released, not on a later one
  • · The version number is agreed and follows the scheme the estate uses
  • · The previous release version is recorded so consumers have something to pin back to

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Read the diff since the last release and classify every interface change
  2. 2Decide the version number from that classification, not from how large the change felt
  3. 3Write the changelog entry, including an explicit upgrade note for every breaking change
  4. 4Freeze: create the release branch or tag candidate from the exact tested commit
  5. 5Run the full test matrix on the release candidate, across every platform the role claims to support
  6. 6Build the artefact and install it from the build output into a clean environment
  7. 7Verify a consumer playbook works against the installed artefact, not against the source tree
  8. 8Tag and publish to the internal Galaxy server or artefact store
  9. 9Verify the published artefact installs by version from a clean controller
  10. 10Notify consumers with the version, the changelog and the upgrade note
  11. 11Do not update any consumer pin as part of this runbook - that is each consumers own change

4 · Verification

Confirm the procedure actually fixed the problem.

  • The tag points at the commit that the test matrix passed on
  • ansible-galaxy collection install of the published version succeeds on a clean controller with an empty collections path
  • The installed artefact contains the role, and ansible-doc or the argument spec reflects the documented interface
  • A consumer playbook runs in check mode against the installed artefact with no resolution errors
  • The changelog names every breaking change and how to migrate
  • The previous version is still installable - a release that breaks the rollback path is not releasable

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • Before publishing, rollback is deleting the tag and the candidate branch
  • After publishing, do NOT delete or overwrite the published version - a version that changed after publication breaks every consumer who cached it
  • Publish a corrected higher version instead, and mark the bad one deprecated if the store supports it
  • Consumers roll back by changing their pin to the previous version and reinstalling - that is why the previous version must remain installable
  • If the bad version was already installed on controllers, reinstall the previous version explicitly with --force; ansible-galaxy will not downgrade otherwise

6 · Escalation

When the runbook isn't enough, contact:

  • · Escalate to consumers before releasing any breaking change - a major version they were not expecting is a change to their estate, made by you
  • · Escalate if the test matrix cannot be run on a platform the role claims to support; either fix the matrix or narrow the claim, do not release on assumption
  • · Escalate to the platform owner if the artefact store requires a version to be overwritten - that is a store problem, not a release problem
  • · Escalate if a published version must be withdrawn; consumers need telling directly, not through a deleted tag they will never notice

Releasing a role is the moment it stops being yours. After publication, other teams install it by version, pin it, and run it against hosts you do not know about. Every assumption you left implicit becomes their problem, and the version number is the only thing telling them how much to worry.

This runbook is deliberately conservative about two things: what counts as a breaking change, and never republishing a version that already exists.

When to use this runbook

  • Cutting a release of an internal collection.
  • Publishing a shared role that other repositories consume.
  • Releasing a fix that consumers need to pick up deliberately.
  • Promoting a role from one team’s repository into shared ownership.

Blast radius

Publishing itself changes nothing - no consumer picks up a new version until they change their pin. That is the design, and it is why pinning matters.

The blast radius is delayed and indirect: every consumer that upgrades, across every host they manage. You will not see it, which is exactly why the changelog and the version number carry the weight here.

Inputs

  • The consumer list.
  • The previous released version and its tag.
  • The commit to be released.
  • The test matrix definition and its last full result.

Step 1: Read the diff and classify it

Read-only / Safewhat changed
git fetch --tags
git log --oneline 1.4.2..HEAD
git diff 1.4.2..HEAD -- roles/nginx_frontend/defaults roles/nginx_frontend/meta

The second command is the one that decides the version. defaults/ and meta/argument_specs.yml are the role’s public interface; changes elsewhere are implementation.

Classify every change in that diff:

ChangeVersion impact
A variable in defaults/ was renamed or removedBreaking
A default value changed in a way that alters host stateBreaking - consumers who relied on the default get a different outcome
A previously optional variable became requiredBreaking
A new variable was added with a safe defaultMinor
A new capability was added, off by defaultMinor
A bug was fixed with no interface changePatch
Internals refactored, same interface, same outcomePatch

Step 2: Decide the version

Derive it from the table, not from how big the work felt. A three-line change that renames a variable is a major release. A two-thousand-line refactor with an identical interface is a patch.

If the estate does not use semantic versioning, use whatever it does use - but the classification above still decides which slot moves, and consumers still need to be able to read the number and know whether to be careful.

Step 3: Write the changelog

Configuration changechangelog entry
## 2.0.0 - 2026-08-11

### Breaking changes
- nginx_frontend_workers renamed to nginx_frontend_worker_processes.
Rename the variable in your group_vars. No behaviour change.
- Default listen port changed from 80 to 8080. If you relied on the
default, set nginx_frontend_listen_port: 80 explicitly BEFORE
upgrading, or the next converge will move the listener.

### Added
- nginx_frontend_upstreams accepts a list of backend addresses.

### Fixed
- Config template no longer emits a duplicate server_name directive
when nginx_frontend_site contains a dot.

Every breaking entry needs the migration, in the entry, in the imperative. “Renamed X to Y” tells a reader what happened. “Rename the variable in your group_vars” tells them what to do, which is what they are reading it for at 9am on an upgrade day.

Step 4: Freeze

Configuration changerelease candidate
git checkout -b release/2.0.0
git log -1 --format='%H %ci %s'

Record that hash. Everything from here tests this commit. A fix merged to the main branch during the release does not go into this release; it goes into the next one.

The failure this prevents is the tag that points somewhere the test matrix never ran. It happens quietly: a merge lands while the matrix is running, the tag is cut from the branch head afterwards, and the released artefact contains code that was never tested.

Step 5: Full test matrix on the candidate

Configuration changemolecule test --all
cd roles/nginx_frontend
molecule test --all 2>&1 | tee "release-2.0.0-matrix.log"
grep -E 'idempotence|verify|FAILED' "release-2.0.0-matrix.log" | tail -20

Every platform the role’s metadata claims to support gets a scenario. If a claimed platform has no scenario, you have two honest options: write the scenario, or remove the claim. Releasing with an untested platform in meta/main.yml is telling consumers you tested something you did not.

The idempotence step is not optional here. A role that converges twice with changes is a role that will produce a service restart on every run across every consumer’s estate.

Step 6: Build, then install what you built

Configuration changeansible-galaxy collection build
cd /srv/automation/collections/example/infra
ansible-galaxy collection build --output-path /tmp/release
ls -l /tmp/release/
Read-only / Safeinstall into a clean path
rm -rf /tmp/verify-collections
ansible-galaxy collection install \
/tmp/release/example-infra-2.0.0.tar.gz \
-p /tmp/verify-collections

ANSIBLE_COLLECTIONS_PATH=/tmp/verify-collections ansible-galaxy collection list

Step 7: Run a consumer playbook against the artefact

Read-only / Safeconsumer check
cd /tmp/consumer-test
ANSIBLE_COLLECTIONS_PATH=/tmp/verify-collections \
ansible-playbook consumer.yml --syntax-check

ANSIBLE_COLLECTIONS_PATH=/tmp/verify-collections \
ansible-playbook consumer.yml --limit staging-web01.example.com --check --diff

Use a real consumer playbook if you have one, referring to the role by its fully qualified name. That is the call path consumers use, and it is where a namespace or naming mistake surfaces.

Step 8: Tag and publish

Configuration changetag
git tag -a 2.0.0 -m 'Release 2.0.0 - see CHANGELOG.md'
git push origin 2.0.0
Configuration changepublish
ansible-galaxy collection publish \
/tmp/release/example-infra-2.0.0.tar.gz \
--server internal_galaxy

Configure the server and its token in ansible.cfg under a [galaxy_server.internal_galaxy] section, or supply the token from an environment variable in CI. Do not paste a token on the command line - it lands in shell history and in the process table.

Step 9: Verify the published version installs

Read-only / Safeinstall from the server
rm -rf /tmp/published-check
ansible-galaxy collection install 'example.infra:==2.0.0' \
-p /tmp/published-check -s internal_galaxy
ANSIBLE_COLLECTIONS_PATH=/tmp/published-check ansible-galaxy collection list

Then confirm the rollback path still works, because that is what consumers will reach for:

Read-only / Safeprevious version still installable
rm -rf /tmp/previous-check
ansible-galaxy collection install 'example.infra:==1.4.2' \
-p /tmp/previous-check -s internal_galaxy

A release that removed the previous version from the store is a release with no rollback, and consumers find that out at the worst time.

Step 10: Notify consumers

Send the version, the changelog, and - for a major release - the migration steps, directly to the consumer list. Do not rely on them noticing a tag.

Then stop. Updating a consumer’s pin is not part of this runbook. Each consumer upgrades on their own schedule, through their own change process, with their own canary. A release that arrives already applied to someone else’s estate is not a release, it is a change you made to their infrastructure.

Rollback

StageRollback
Candidate branch, matrix runningDelete the branch.
Tagged, not publishedgit tag -d 2.0.0 and delete the remote tag.
PublishedDo not withdraw. Publish a higher corrected version; notify consumers.
Bad version installed on controllersReinstall the previous version explicitly.
Configuration changedowngrade a controller
ansible-galaxy collection install 'example.infra:==1.4.2' \
-p ./collections --force
ansible-galaxy collection list | grep example.infra

--force is required. Without it ansible-galaxy sees a version already installed and leaves it alone, so the “downgrade” reports success and changes nothing - and the next play runs the broken version. Check the collection list output afterwards; do not trust the install message.

Common patterns

SymptomLikely causeResolution
Works in the repo, fails after installA file was excluded from the build or never committedInstall the tarball into an empty path and retest
Consumers on the same pin behave differentlyA version was republishedPublish a new version; never overwrite
Downgrade appears to succeed but changes nothingansible-galaxy will not replace an installed version without --forceAdd --force; verify with collection list
Consumers broken by an upgrade that was called a patchA default value changedReclassify; a changed default is breaking
Tag points at untested codeThe tag was cut from the branch head after the matrix ranTag the tested commit; freeze before testing
A claimed platform fails at a consumer siteIt was in meta/main.yml with no scenarioAdd the scenario or drop the claim

Escalation

Escalate when:

  • The release contains a breaking change. Consumers need telling before, not after.
  • A supported platform cannot be tested.
  • The artefact store wants you to overwrite an existing version.
  • A published version must be withdrawn.

References

  1. Distributing collections
  2. Collection structure
  3. Collection changelogs
  4. Semantic Versioning