What you cannot test before production · Ansible

Every layer in this part has been honest about a gap. This lesson is about what remains when you have built all of them properly, because that residue does not go away with more effort — it is structural.

The conclusion the part has been building to: testing is not something you complete. There is no configuration of containers, VMs and staging that reduces the residue to nothing, so the last rung is not a formality performed after testing finished. It is the final test, and treating it as one changes how it is run.

The classes that only exist in production

Load. Not “many requests” — the specific shape of your traffic. A configuration change that adds a few milliseconds per request is invisible at ten requests per second and is an outage at ten thousand, because queues are not linear. Synthetic load is a useful approximation and it is generated by a program that does not behave like your users.

Real data volume and shape. A migration that takes four seconds against a 2 GB extract takes forty minutes against 400 GB, and takes four hours if the production table has an index distribution your extract flattened. Query plans change with statistics; statistics come from real data.

Real hardware. Firmware revisions, NIC offload settings, disk controllers, NUMA topology, and the two machines in the fleet that were bought a year later and have a different CPU stepping. A role that tunes anything hardware-adjacent is tuning for hardware it has not seen.

Real neighbours. Production hosts share things: a hypervisor, a storage array, a switch, a NFS server, a DNS resolver, a rate-limited external API. Behaviour that is fine in isolation can be an incident when forty hosts do it simultaneously — and “simultaneously” is precisely what automation makes possible.

Real concurrent operators. Someone else is deploying. Someone is mid-incident. A scheduled job starts. Your change is not the only thing happening, and the interaction between two safe changes is not guaranteed to be safe.

Time. Certificates expire, log rotation happens at 03:00, the monthly cron collides with your daily one, and the leap in disk usage appears on day nine. A test that runs for ninety seconds cannot observe anything with a period longer than ninety seconds.

The canary as a test

A canary is not “deploy to one host first and hope”. It becomes a test when it has the four things any test has: a hypothesis, a defined observation, a pass condition, and a defined response to failure.

Choose the host deliberately

The convenient canary is the first host in the group, or the one whose name sorts first, or the one somebody knows is idle. All three are selections for irrelevance, and an irrelevant canary is a test with a hypothesis nobody wanted to test.

A representative canary takes real traffic, has real data, and sits on hardware the rest of the fleet resembles. If your fleet is genuinely heterogeneous, the canary is one host per class rather than one host.

Service impact possiblethe canary run— Changes one production host. SERVICE-IMPACT because the play restarts the service on it. Check the host list first; --limit is the only thing standing between this and the fleet.

# 1. Confirm exactly which hosts this would touch. READ-ONLY.
ansible-playbook -i inventories/production site.yml \
--limit web07.example.com --list-hosts

# 2. Predict the change without making it.
ansible-playbook -i inventories/production site.yml \
--limit web07.example.com --check --diff

# 3. Make it, on one host.
ansible-playbook -i inventories/production site.yml \
--limit web07.example.com --diff

Step 1 is not ceremony. --limit takes a pattern, and a pattern that matches nothing runs against nothing while a mistyped pattern can match more than intended — --list-hosts is how you find out which of those you have written, before rather than after.

Define the window before you start

The most common canary failure is not a bad change. It is a canary that was observed for ninety seconds, declared fine, and rolled out — when the defect it was there to catch appears on a memory leak’s timescale, or at the next log rotation, or when the connection pool cycles.

The window is a judgement about the failure modes you are worried about, and it is worth writing down alongside the change:

Concern	Minimum useful window
the service fails to start	seconds
the config is wrong in a way requests reveal	minutes of real traffic
a resource leak	hours, and a graph
log volume or disk fill	one rotation cycle
a scheduled job interaction	one period of that job
certificate or token expiry	not observable; check the expiry directly

The last row matters. Some concerns are not observable within any sensible window and must be checked rather than watched.

Decide the abort condition in advance

Written before the change, not derived afterwards from whatever the graphs happen to show.

Error rate on this host above X for Y minutes.
Latency at the 99th percentile above Z.
Any entry matching a specific pattern in the service log.
Memory above a threshold at the end of the window.
Anything at all in the log that nobody can explain.

That last one is the one people leave out and the one that catches genuine surprises. “We do not know what that message is” is an abort condition, not a curiosity to investigate later.

Closing the loop

The final habit is what turns this ladder into something that improves rather than something you maintain.

Every defect that reaches production is evidence about your pipeline. For each one, the question from lesson 1 applies: which is the lowest rung that could have caught this, and why did it not run there?

What escaped	Where it belongs afterwards
a mistyped module option	lint rung, with the `args` rule enabled, or the disposable integration rung
a role that was not idempotent	the second-run gate — it was not running, or the task was excluded
a service that never started	outcome verification in the scenario, and a systemd-capable environment
an assumption about a distribution	the platform matrix, plus that distribution
a rollout that took out too much at once	the staging fidelity statement, and an absolute `serial`
a migration that overran the window	not testable pre-production; the fix is a measurement on real data, not a test

The last row is the honest one. Some findings do not become tests. They become a documented limitation and a different kind of control — a measurement, a monitor, a runbook step — and pretending otherwise produces a test that reassures without covering.

Under the hood

There is a structural reason the residue cannot be eliminated, and it is worth stating precisely because it stops the argument going in circles.

Every pre-production environment is a model. A model is useful exactly because it omits something; a model that omitted nothing would be the system itself, at which point it is production and carries production’s risk. So the fidelity of a test environment and the safety of testing in it are the same quantity traded in opposite directions.

That is why “make staging identical to production” is not a goal you converge on. Taken to its conclusion it means duplicating production including its data, its traffic and its neighbours — which is a second production, with a second production’s cost and a second production’s ability to cause a real incident when you break it deliberately.

The canary resolves the trade differently. Instead of building a high-fidelity model, it uses production itself and limits the blast radius to one host. Perfect fidelity, minimal exposure, and a real consequence if it goes wrong. That combination is only available in production, which is the whole reason the ladder ends there.

What happens if the canary passes and the fleet rollout fails?

It happens, and it is informative rather than embarrassing.

The usual causes are the ones a single host structurally cannot show. Something shared saturated — a package mirror, an API, a storage array, a DNS resolver — under forty simultaneous requests. The load balancer reacted to losing a batch. Two hosts that were fine individually contended for the same lock. The canary was less representative than believed.

The response is the same loop. Each of those maps to a control: rate limits and serial for the shared-resource cases, a health check that gates the next batch for the load balancer case, and a better canary selection for the last. None of them makes the canary redundant; they make the step after the canary less of a leap.

The mistake to avoid is concluding that testing does not work. What the failure shows is that the fleet rollout is its own rung, with its own defect class, and it needs its own controls — which is what the rolling-update part is about.