Blast radius as a design constraint · Ansible

This course has used the phrase “blast radius” since the welcome page. Here is the definition it has been using, stated once, precisely, so that the rest of the part can build on it:

Blast radius is the number and criticality of the systems a mistake can affect.

Two nouns, and the second is the one people drop. A change that touches five hundred development laptops and a change that touches two database primaries have wildly different host counts and comparable consequences — arguably the second is worse, because the development boxes can be rebuilt this afternoon and the database primaries cannot.

Keeping the two axes separate is what makes the idea usable. Collapse them into “how many hosts” and you get a rule that says a two-host change is always safe, which is how production outages happen on Tuesdays.

Two axes, not one

Read-only / Safethe mental model this part refers back to

        high  |  restart 2 db primaries      patch all prod servers
            |  rotate the CA key           change prod firewall rules
criticality |
            |  install a package on        reconfigure logging on
      low   |  3 dev boxes                 400 dev boxes
            +----------------------------------------------------
               few hosts                        many hosts

The top-left quadrant is the one that gets skipped. It looks small — two hosts, a single command — and it carries the highest consequence per host in the estate. Every control in this course applies there too, and the fact that --limit db-primary01.example.com is short does not make the change safe.

The bottom-right quadrant is where people over-invest in ceremony and under-invest in the thing that actually helps: knowing the number. Four hundred development boxes is fine to change at once, provided the number really is four hundred development boxes and not four hundred hosts whose composition nobody checked.

Criticality is a property of the host, so record it on the host

If criticality only exists in people’s heads, it cannot be part of a guardrail. Put it in the inventory, where a play can read it:

db01.example.com:
  criticality: tier1
  service_owner: payments
  change_window: 'sun 02:00-04:00'

A pre-flight check can then refuse a run that touches a tier1 host without an explicit acknowledgement variable, which is a control that scales with the estate instead of depending on whoever is on shift recognising the hostname.

The four questions

Before any run that changes anything, four questions. They are short, they are answerable from the command line, and the discipline is that you answer all four rather than the one that comes to mind.

1. What changes?

Not “what does the playbook do” — what will this invocation do. Tags, when: conditions and variables all narrow it.

Read-only / Safethe tasks this run would execute— Real output from ansible-core 2.21.3. --list-tasks resolves the playbook, including includes and roles, and prints the tasks per play with their tags. It connects to nothing.

$ ansible-playbook -i inventory/ site.yml --list-tasks

playbook: site.yml

play #1 (web): Configure the web tier	TAGS: []
  tasks:
    Placeholder	TAGS: []

Then --check --diff, which answers the question properly by reporting what would change on each host rather than what tasks exist. Its limits — modules without check-mode support, and a command task that skips — are covered in the check-mode lesson, and they are the reason --check is evidence rather than proof.

2. On which hosts?

This is the central skill of this part and it has its own lesson. The short version: ansible-playbook <playbook> --limit <pattern> --list-hosts, and read the count.

Read-only / Safethe number that is the blast radius— Real output from ansible-core 2.21.3. The count on the hosts (N): line is the first axis, stated as an integer, before anything happens.

$ ansible-playbook -i inventory/ site.yml --limit us_east --list-hosts

playbook: site.yml

play #1 (web): Configure the web tier	TAGS: []
  pattern: ['web']
  hosts (3):
    web04.example.com
    web03.example.com
    web02.example.com

3. In what order?

A change that reaches every host simultaneously has no failure point at which you can stop. A change that reaches one host, then five, then the rest, has two.

Ordering is the difference between “we broke one host and stopped” and “we broke all of them”. serial, order: and canary groups are how it is expressed, and Parts XXXI and XXXII own them; what belongs here is that order is part of blast radius, not a separate concern. The same change to the same hosts has a different radius depending on whether it lands all at once.

4. How do I stop it?

The question nobody rehearses.

Interrupting ansible-playbook with Ctrl-C stops the controller from issuing further tasks. It does not undo what has run, and it does not recall a task already dispatched to a host. Under serial, hosts in later batches are simply never reached — which is the useful property, and is why batching is a stopping mechanism as much as a pacing one.

Knowing the answer means knowing three things before you start: how you will notice something is wrong, how you will halt the run, and what the recovery is for the hosts already changed. If the answer to the third is “restore from backup”, that is a legitimate answer and it changes how you would have sized the batch.

What changes: --list-tasks for scope, then --check --diff for content.
Which hosts: --list-hosts with the same --limit you will use for real, and read the count out loud.
How many are critical: how many of those hosts are tier1, and does the count match the change ticket.
In what order: is this one batch or several, and what is the first batch.
How do I stop: what will show me it is going wrong, what halts it, and what recovers the hosts already changed.

The count is the control, and it has two failure directions

Both directions are silent, and this is the property that makes the habit necessary rather than merely sensible.

Too few. A pattern with a typo matches nothing. The run exits 0. A nightly job reports success for three weeks having done nothing.

Too many. A pattern that resolves to three hundred hosts when you meant thirty also exits 0, also reports success, and does change three hundred machines. There is no warning for “more than you expected”, because Ansible has no idea what you expected.

Under the hood

There is one more axis, and it does not fit on the diagram: whether the change is reversible.

Ansible’s execution model gives you no rollback. There is no transaction log, no undo, no previous state recorded anywhere unless a module chose to keep one. What exists is narrower and worth knowing precisely:

Modules like copy, template and lineinfile support backup: true, which writes a timestamped copy of the file before changing it. That is a file-level undo you have to have asked for in advance.
serial limits how many hosts reach the irreversible state before you can stop, which is mitigation rather than reversal.
A block with rescue: runs remediation you wrote. It is not a rollback; it is a second set of tasks that also has to work.

Everything else — a package upgrade, a schema migration, a deleted file, a restarted service that did not come back — is forward-only.

So the practical form of the fourth question is not “how do I roll back”, because usually you cannot. It is “how many hosts can I afford to be wrong on, given that the ones I have already changed will stay changed”. That reframing is what makes batch sizing a blast-radius decision rather than a performance one.

What happens if the host count is right and the change is still wrong?

Then targeting did its job and the failure was elsewhere, and it is worth being precise about that, because “verify the host list” is not a claim that verifying it makes a change safe.

The host list bounds how far a mistake spreads. It says nothing about whether the change itself is correct. A perfectly targeted run of a playbook containing a bad template renders the bad template on exactly the hosts you intended, which is a complete success by every measure this part teaches.

The controls that address that half are different ones: check mode and diff for seeing the content before it lands, idempotence for making a second run a no-op, staged rollout for making the first batch small enough that a bad template is discovered on three hosts rather than three hundred.

The two halves compose, and neither substitutes for the other. Knowing the host list is what turns “the change was wrong” into “the change was wrong on the canary group”, which is the difference between an incident and a Tuesday.

Knowledge check

Why this matters

Most Ansible mistakes are not exotic. The playbook was fine, the module was fine, the change was one somebody had made a hundred times. What went wrong was the set of machines it landed on.

That failure has no signature in the tool. There is no error, no warning, no non-zero exit — just a run that succeeded against more hosts than anybody intended. It is invisible in the logs because from Ansible’s point of view nothing unusual happened.

Which puts the whole burden on a habit: form an expectation, print the host list, compare the count, and only then run. The rest of this part is the mechanics of doing that quickly and reliably — the pattern syntax that decides what a target expression means, the --limit that narrows it, the commands that prove the result, and the guardrails that refuse when the number is wrong.