Promoting a change from development to production

“We promoted it to production” is one of those phrases that sounds like a process and is usually a description of a feeling. Ask three people on the same team what it meant and you may get three answers: that the change was merged, that somebody ran the playbook against production, or that a pipeline stage turned green.

This lesson gives it a definition you can check.

A change has been promoted when the run against production used the same automation commit and the same resolved dependencies as the run that was tested, differing only in the inventory it was pointed at.

Three things identical, one thing different. Every failure mode in this lesson is one of those three quietly not being identical.

The three that must not differ

The commit. Not “the same change” — the same commit hash. A change that was rebased, squashed, or amended between the staging run and the production run is a different body of code, and the difference is exactly the kind nobody re-reads because “it was just a rebase”.

The dependencies. requirements.yml resolved to specific collection and role versions when staging ran. If those versions are not pinned, the production run resolves them again, later, possibly to different code. Lesson 2 covers the pinning syntax; here the point is that the pins are part of what is promoted, not infrastructure around it.

The controller. ansible-core itself, and the Python it runs on. A staging run on 2.21.3 and a production run on 2.22 are not the same run, and minor releases do change behaviour — the porting guide exists for that reason. Part III’s reproducible-controller lesson covers pinning it in requirements.txt; the promotion consequence is that a controller upgrade is a change that needs its own promotion, not a maintenance detail.

The one thing that differs is the inventory path. That is the whole design: the same code, aimed at different machines.

Recording what actually ran

A promotion you cannot reconstruct afterwards is a promotion you cannot audit, and reconstruction is precisely what you need three weeks later when something is wrong and the question is what changed.

Ansible can tell you most of it from inside the run, using only read-only facts:

# playbooks/_provenance.yml — imported first by every entry point
- name: Record run provenance
  hosts: all
  gather_facts: false
  run_once: true
  tasks:
    - name: Assemble the run manifest
      ansible.builtin.set_fact:
        run_manifest:
          inventory_sources: "{{ ansible_inventory_sources }}"
          play_hosts: "{{ ansible_play_hosts_all | length }}"
          ansible_version: "{{ ansible_version.full }}"
          config_file: "{{ ansible_config_file | default('none') }}"
          release: "{{ release_identifier | default('UNTAGGED') }}"

    - name: Publish it
      ansible.builtin.debug:
        var: run_manifest

$ ansible-playbook -i inventories/production playbooks/_provenance.yml

TASK [Publish it] **************************************************************
ok: [web-prod-01.example.com] => {
  "run_manifest": {
      "ansible_version": "2.21.3",
      "config_file": "/srv/estate/ansible.cfg",
      "inventory_sources": [
          "/srv/estate/inventories/production"
      ],
      "play_hosts": 2,
      "release": "2026.08.11-1"
  }
}

Four of those five fields come free from special variables, and each one closes a specific hole:

• inventory_sources is the resolved path, so it distinguishes inventories/production from inventories — the mistake from lesson 1 that changes everything and reports success.
• play_hosts is the host count. A number that differs from the expected one for this environment is the single best early warning available.
• ansible_version catches the controller drift nobody thinks to check.
• config_file catches the run that picked up a different ansible.cfg, or none, because it started from the wrong directory.

release_identifier is the one you have to supply — from the CI job, as -e release_identifier=$RELEASE or an environment variable. It is worth the extra step, because it is the field that ties the run to the tag.

The promotion checklist

Run this before the production job, every time. It is short because everything expensive has already happened.

1. The commit is the one that was tested - the same hash, not the same change. A rebase, a squash or an amend since the staging run means the staging evidence describes different code, and the change goes back through staging.
2. Every entry in requirements.yml carries an explicit version, and none of them was bumped since the staging run. A bumped pin means staging did not test this dependency set.
3. ansible --version on the machine that will run production equals the version that ran staging. A controller upgrade is its own change with its own promotion.
4. Staging evidence exists and is recent: a --check --diff output or a real staging run, attached to the pull request, showing what the change does. Part XXXVIII makes this a merge gate.
5. The inventory is confirmed by host count. Run ansible-playbook -i inventories/production --list-hosts and compare the number with the count the README records for this environment. This is where the parent-directory mistake surfaces.
6. The vault id is the production one. A run with the wrong id fails at variable load before contacting any host, which is the good outcome - confirm it deliberately rather than discovering it.
7. Blast radius is stated and bounded: how many hosts, in what batches, with what serial. If the answer is all of them at once, say so out loud and decide whether that is acceptable for this change.
8. Rollback is identified before the run - the previous tag, and what rerunning it would and would not undo. Name what is not reversible: a migrated database, a rotated credential, a deleted file with no backup.
9. Someone is on call, by name, and knows this is happening. A change nobody is watching is a change discovered by customers.

What a green pipeline does not prove

Worth stating plainly, because a promotion model can create false confidence as easily as it creates safety.

• It does not prove the change is correct. It proves the change did in staging what the author expected, on staging’s hosts, with staging’s data.
• It does not prove production’s state resembles staging’s. Drift is Part XXXVI’s subject and it is orthogonal to promotion: identical automation against divergent hosts produces divergent results.
• It does not prove the change is reversible. Rollback is a separate design question, and “revert the tag and rerun” undoes configuration but not migrations, rotations or deletions.
• It does not prove the timing is safe. A perfectly promoted change applied during peak traffic is still a change applied during peak traffic.

The model’s actual claim is narrow and worth having: whatever was observed in staging was observed of this exact code. Everything else has to be argued separately.

Knowledge check