Staging that is worth having

Staging is the rung where testing stops being cheap. A container matrix costs seconds and a few gigabytes of images; a staging fleet costs machines, addresses, certificates, a copy of the data, and somebody’s attention when it breaks.

Because it is expensive, it is usually built once, at the smallest size that could be described as staging, and then left alone. What that produces is not a smaller production — it is a different system that shares production’s name, and a green run against it is a claim about that different system.

The purpose of this lesson is to make the trade explicit, so that the staging you pay for buys the coverage you think it does.

The dimensions, ranked by what they buy

Not all fidelity is worth the same. Ranked by how often the gap actually causes an incident:

The first three are close to free and are the ones most often wrong. The bottom three are genuinely expensive and are where honest documentation substitutes for coverage.

Host count changes behaviour, not just scale

This is the fidelity gap people most consistently underestimate, because it feels like a difference of degree.

A play with serial: 25% behaves differently depending on how many hosts are in the group. Executed against ansible-core 2.21.3, with an inventory of three hosts and then one of ten:

$ ansible-playbook -i inv3.ini serial.yml

PLAY [Batching demonstration] **************************************************

PLAY RECAP *********************************************************************
web01.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

$ ansible-playbook -i inv10.ini serial.yml

PLAY [Batching demonstration] **************************************************

PLAY RECAP *********************************************************************
web01.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
web02.example.com          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Three hosts gives a batch of one; ten gives two; two hundred would give fifty. A rollout that was verified on a three-host staging fleet was verified with a batch size of one — which is to say, it was never a rolling update at all. Nothing about restarting fifty hosts at once was exercised, and neither was the load balancer’s reaction to losing a quarter of the pool.

The same logic applies to max_fail_percentage, to forks and connection concurrency, and to anything where a small number rounds to something qualitatively different.

Staging must be built by the automation it tests

This is the cheapest high-value property on the list, and the most commonly missing.

If staging hosts were built by hand, or built once from an image that has since diverged, then a role that succeeds against them has been tested against a machine your automation did not create. Every implicit assumption the role makes about pre-existing state — a user, a directory, a repository, a mount — is satisfied by history rather than by code, and none of it will be true on a freshly provisioned production host.

The failure has a signature: the role works in staging, works on existing production hosts, and fails on the first new production host built after the change. By then the change is weeks old and nobody connects the two.

The fix is structural. Staging is provisioned by the same automation as production, from the same roles, differing only in inventory and variables — which is the repository-architecture question the later part covers. A staging environment you can destroy and rebuild is one whose fidelity you can check; one you are afraid to rebuild has already drifted.

ansible -i inventories/staging web \
-m ansible.builtin.setup \
-a 'filter=ansible_distribution*,ansible_service_mgr,ansible_pkg_mgr,ansible_kernel'

ansible -i inventories/production web \
-m ansible.builtin.setup \
-a 'filter=ansible_distribution*,ansible_service_mgr,ansible_pkg_mgr,ansible_kernel'

Running that pair on a schedule and diffing the output catches the slow divergence — a point release here, a kernel there — before it is the explanation for an incident.

The trap of the shared staging environment

One staging fleet, several teams, no coordination.

The symptom is a class of test failure that is real, unreproducible, and somebody else’s: your role fails because another team’s deployment is mid-flight, a service you depend on is being restarted, or the host you were testing against was rebuilt underneath you.

The damage is not the failed run. It is that the team learns to re-run the job when it fails, and that habit is indistinguishable from the habit of ignoring genuine failures. A staging environment that produces unexplained failures trains people to disbelieve staging.

Two mitigations, both cheap relative to a second fleet: give each team its own inventory group and its own hosts within the shared environment, and make the environment’s occupancy visible so that a failure can be attributed rather than guessed at.

Write down what staging does not cover

The deliverable from this lesson is a paragraph, kept next to the inventory:

# Staging fidelity statement

Last reviewed: 2026-08-11

Matches production:
- Ubuntu 24.04, same point release, same kernel series
- systemd, same unit files, built by the same roles
- Same reverse proxy and TLS termination
- Same filesystem layout and mount options

Does NOT match production:
- 3 web hosts against production's 40. Percentage-based serial
  resolves to a batch of 1 here and 10 there; rollout behaviour is
  NOT exercised. Run staging rollouts with serial: 1 explicitly.
- Database is a 2 GB anonymised extract; production is 400 GB.
  Migration timings from staging are meaningless.
- No production traffic. Restart-during-request behaviour is
  untested here.
- Single availability zone; production spans three.

Consequence: a green staging run supports "the change applies cleanly
and the service comes back". It does not support "the rollout is safe
at production scale" or "the migration completes inside the window".

The last paragraph is the point. It converts staging from a ritual into a claim with stated bounds, and it gives the person deciding whether to proceed something to reason about.

Knowledge check