Anti-pattern: no limit, no canary · Ansible

Service impact possiblethe anti-pattern, in full— SERVICE-IMPACT: this play changes configuration and restarts a service on every host in the inventory, simultaneously, with the default forks.

- name: Configure the application
hosts: all
become: true
roles:
  - app

Invoked as ansible-playbook site.yml. No --limit, no serial, no canary group, no --check first.

There is nothing syntactically wrong with it, it is what every tutorial shows, and it is how most repositories start.

What it looks like when it fires

The template has a defect — a variable that is undefined on some hosts, a value with the wrong unit, a directive removed in the newer package version. The role renders it, writes it, and notifies a handler.

With the default forks = 5 and a fast play, three hundred hosts take a couple of minutes. With forks raised for a large fleet, as it usually is, ninety seconds is realistic.

Read-only / Safewhat the operator could have known first— Real output from ansible-core 2.21.3 against a 16-host inventory, trimmed. This command is read-only and takes under a second. In the incident above, nobody ran it.

$ ansible-playbook -i inventory/hosts.yml site.yml --list-hosts

playbook: site.yml

play #1 (all): Configure the application	TAGS: []
  pattern: ['all']
  hosts (16):
    web10.example.com
    web01.example.com
    db03.example.com
    ...

The failure is not the outage

An outage is bad and recoverable. The specific damage here is that simultaneity removes the two things you need in order to recover.

1. There is no healthy population to diagnose against

Diagnosis is comparison. “This host is broken, that host is fine, what differs” is how these are solved, and it takes minutes.

When all three hundred hosts received the change at the same time, every host is broken. There is nothing to compare against. The investigation becomes reading the diff and reasoning from first principles, at 03:00, with the service down — and the change is now the prime suspect and the only evidence.

2. There is nothing to roll back from

Rollback needs a known-good state to return to. Part XLVIII establishes that Ansible has no universal rollback: the previous state was not captured, because nothing captured it.

In a staged rollout, the un-updated hosts are the known-good state. They are serving traffic, they hold the previous configuration, and you can read the old file off one of them. In a simultaneous rollout there is no such host, and the recovery is reconstructing the previous configuration from Git — assuming the previous version was committed, and assuming the change is reversible at all, which package upgrades and schema migrations frequently are not.

The corrected form

Three additions, none of them clever.

Service impact possiblestaged, with an explicit canary— SERVICE-IMPACT: still changes hosts, but one first, then ten percent, then the rest - with a health gate between batches.

- name: Configure the application
hosts: app
become: true
serial:
  - 1
  - 10%
  - 100%
max_fail_percentage: 0
roles:
  - app
post_tasks:
  - name: The service must answer after the change
    ansible.builtin.uri:
      url: 'http://{{ ansible_host }}:8080/healthz'
      status_code: 200
    retries: 5
    delay: 6
    register: health
    until: health.status == 200
    delegate_to: localhost

serial as a list is a ramp: batch one is a single host, then ten percent, then everything remaining. max_fail_percentage: 0 stops the play if any host in a batch fails, so a bad first batch does not proceed to the second. The health check makes “the batch succeeded” mean the service answered rather than that the tasks exited zero — Part XXVI’s distinction between verifying outcomes and verifying tasks.

Read-only / Safethe invocation habit— READ-ONLY: both commands only read. Run them in this order, every time, and read the count out loud before the third.

# 1. Who does this hit?
ansible-playbook site.yml --limit app_prod --list-hosts

# 2. What would change?
ansible-playbook site.yml --limit app_prod --check --diff

# 3. The canary only.
ansible-playbook site.yml --limit canary

And the guardrail that does not depend on remembering, from Part XXX:

Read-only / Safea pre-flight play that refuses an oversized target— READ-ONLY: assert connects to nothing and changes nothing. As its own play with no serial, it evaluates once for the whole run.

- name: Pre-flight
hosts: all
gather_facts: false
vars:
  max_hosts: 25
tasks:
  - name: Refuse a target set larger than the declared ceiling
    run_once: true
    ansible.builtin.assert:
      that: ansible_play_hosts_all | length <= max_hosts | int
      fail_msg: >-
        Refusing: {{ ansible_play_hosts_all | length }} hosts targeted,
        ceiling is {{ max_hosts }}. Narrow with --limit or raise it
        deliberately in the playbook vars.
      success_msg: '{{ ansible_play_hosts_all | length }} hosts, within the ceiling'

“No rollback plan” appears in this lesson rather than its own because it is the same gap seen from the other end: staging exists so that rollback is possible, and a rollback plan is what makes staging worth doing.

A rollback plan is not the sentence “we would revert the commit”. To be one, it must answer:

What is the previous state, concretely? A package version, a config file in Git at a known commit, a database schema revision. If nobody can name it, there is no plan.
Is the change reversible at all? A package downgrade may not restore the previous behaviour; a schema migration usually cannot be reversed; a deleted file is gone. Say so explicitly rather than discovering it during the rollback.
How is the previous state applied? A playbook run with a pinned version, a restore from a backup, a switch back to the previous artefact. It is a procedure, and it should have been tested.
How long does it take, and does that fit the outage budget? A rollback slower than the fix is not a rollback.
What is the decision point? "If the canary is not healthy within ten minutes, we roll back" is a plan. "We will see how it goes" is how a fifteen-minute incident becomes three hours.

Part XLVIII covers the absence of universal rollback in full. The operational summary: your rollback is whatever you explicitly built, and a staged rollout is the cheapest form of it because the un-updated hosts are the previous state, still running.

Under the hood

Why does hosts: all feel safe enough to write?

Two mechanisms, and they reinforce each other.

Inventory growth is invisible from the playbook. hosts: all was written when the inventory held four hosts, and it was correct then. The inventory now holds three hundred, having grown by a handful at a time, each addition reviewed on its own merits. No commit ever changed the playbook, so no review ever re-examined the target. The blast radius grew by a hundredfold without a single diff showing it.

--limit is opt-in, and its absence is silent. There is no warning for a wide run, no confirmation prompt, and no difference in output between a deliberate fleet-wide change and a forgotten flag. The safe form requires an action; the dangerous form requires none. Every other control in this course is arranged the same way, which is why the guardrail play exists — it inverts the default so that the wide run is the one requiring a deliberate act.

There is a third factor that is cultural rather than technical: the anti-pattern is invisible until it fires. A repository that has run hosts: all unguarded for two years has two years of evidence that it is fine, and that evidence is entirely valid — right up to the first defective template. The absence of an incident is not the presence of a control, and it is genuinely hard to argue for a guardrail against something that has never happened.

The counter-argument worth having ready: the guardrail is one play and one number, it prints a reassuring line on every normal run, and the thing it prevents is the only failure mode in this course with no recovery path that does not involve reconstructing state from memory.

What happens if a change genuinely must reach every host quickly?

Some do: a credential rotation after a leak, an emergency patch for an actively exploited vulnerability, a firewall rule blocking a live attack. Speed can genuinely be the right priority.

That is a decision, and treating it as one is what separates it from the anti-pattern.

Test on the canary anyway. Even under pressure, one host first costs sixty seconds and catches the class of error that would otherwise take the fleet down. An emergency patch that breaks every host is worse than the vulnerability for most vulnerabilities.

Use a fast ramp rather than none. serial: [1, 25%, 100%] reaches the whole fleet in three batches. That is not much slower than simultaneous, and it preserves the stop point.

Raise the guardrail explicitly. -e max_hosts=400 on the command line, or better, a playbook whose own vars declare the higher ceiling. Part XXX’s argument applies: an override in a committed file is reviewable; an override in shell history is not.

Say what you gave up. “We ran fleet-wide without staging because the exposure window justified it” belongs in the change record. It makes the decision reviewable afterwards and stops the exception quietly becoming the norm — which is the actual risk, because the second emergency is always easier to justify than the first.

Keep the health gate even when you drop the staging. Knowing the fleet is unhealthy ninety seconds in is worth having even when you have chosen not to stop.

Knowledge check

Why this matters

The corrected form is one line of serial and a habit of running --list-hosts first. It is the cheapest fix in this entire part, and the failure it prevents is the most expensive one.

That asymmetry is why the anti-pattern persists. Nothing about hosts: all reads as dangerous, every tutorial writes it, and a repository can run it unguarded for years with no signal at all. The danger is entirely latent, realised only when combined with a defect somewhere else — which means it never appears in a post-incident review of its own.

What it costs when it does fire is not the outage. Outages are survivable and this course spends whole parts on recovering from them. It is that simultaneity takes away the comparison you diagnose with and the healthy population you recover from, at the exact moment you need both, leaving you reading a diff at 03:00 with the entire estate down and the change as both the suspect and the only evidence.

One host first. Then ten percent. Then the rest, with a health check between. Sixty seconds of patience buys the ability to stop.