Safe CM rollout - blast radius, check mode, and the control node

linux-why-manual-doesnt-scale makes the case for configuration management: one declaration, applied consistently, to any number of hosts. That is the leverage.

The bill for it arrives the first time the declaration is wrong. A typo in a sshd_config template is a mistake either way, but by hand it locks you out of one host and by CM it locks you out of four hundred, in under three minutes, from a control node that has now also lost its way in. The mechanism that makes CM valuable is exactly the mechanism that makes a mistake catastrophic, and no amount of care about writing playbooks changes that.

What changes it is procedure: a ladder of cheap checks before an expensive one, and a habit of never running against the whole fleet in the first attempt.

This lesson is about the change. Detecting divergence that appears afterwards is the subject of linux-config-drift-detection and of the drift part of this course; the boundary is that config management is the tool and its model, and drift is what happens despite it.

The pre-flight ladder

Each rung costs more than the one above it and catches a different class of error. Climb all of them, in order.

$ # 1. Does it parse? Seconds, no hosts contacted.
ansible-playbook -i inventory/prod site.yml --syntax-check

# 2. Does it follow the rules? Catches deprecated modules,
#    shell where a module exists, missing become, unnamed tasks.
ansible-lint site.yml

# 3. WHAT will it target? The most under-used flag in Ansible.
ansible-playbook -i inventory/prod site.yml --limit webservers --list-hosts

# 4. WHICH tasks will run, given the tags you passed?
ansible-playbook -i inventory/prod site.yml --tags nginx --list-tasks

playbook: site.yml

play #1 (webservers): webservers	TAGS: []
  pattern: ['webservers']
  hosts (3):
    web01.example.com
    web02.example.com
    web03.example.com

Then the expensive rungs:

# 5. Dry run on ONE host, with the diff
ansible-playbook -i inventory/prod site.yml --limit web01 --check --diff

# 6. For real, on that one host
ansible-playbook -i inventory/prod site.yml --limit web01

# 7. Verify from outside the playbook, then widen
ansible-playbook -i inventory/prod site.yml --limit 'webservers:!web01'

Step 7 is deliberately not “run it everywhere”. Widening in groups keeps a second canary boundary, and excluding the host you already did keeps the run honest about what it changed.

What --check actually proves

Check mode is the most valuable and the most over-trusted tool in this list. It is a hypothesis, not a plan, and there are three specific reasons it can be wrong.

1. command and shell tasks do not run. They are skipped in check mode, because Ansible cannot know whether an arbitrary command is safe. Anything downstream that consumed their registered output therefore sees an empty or missing result:

TASK [a command task] ****************************
skipping: [localhost]

TASK [show it] ***********************************
ok: [localhost] => {
    "msg": "got "
}

In a real run that message would carry the command output. In check mode it is blank, and a when: condition built on it evaluates the other way — so check mode can skip a task that would run, or run one that would be skipped.

2. Sequential dependencies break. A task that templates a config file into a directory created by an earlier task reports a failure in check mode, because the earlier task did not actually create the directory. That is not a defect in your playbook, and treating it as one leads people to add ignore_errors and lose the check.

3. Tasks can be marked to run for real. check_mode: false on a task means it executes during --check. It exists for good reasons — a read-only fact-gathering command whose output later tasks need — and it is a genuine hazard when it appears on a task that changes something.

None of that makes check mode useless. It is excellent at what it is for: showing you which files a run would rewrite and what the new content would be. Read the --diff output as the question “am I surprised by any of this?” and treat a surprise as a stop condition.

Bounding the blast radius

Four controls, each answering a different question.

--forks is the underrated one during a risky change. The default is five, and on a hundred-host run that means twenty waves in a couple of minutes — too fast to interrupt when the first host reveals the problem. -f 1 turns the same run into something you can watch and stop with Ctrl-C after the second host.

--start-at-task and --step are the recovery tools. After a run fails part-way, --start-at-task 'Configure nginx' resumes without repeating what already succeeded, and --step prompts before each task so you can walk a suspect playbook through by hand.

The control node is fleet-root

The control node holds SSH private keys, or the credentials to obtain them, for every managed host. Anyone who can run ansible-playbook on it can run any command as root on the entire estate. Anyone who can merge to the playbook repository can do the same, on the next scheduled run, with no interactive session at all.

That places two artefacts at the same trust level as root on every host you own:

The control node. Named accounts only, no shared login, MFA on the path to it, its own hardening baseline, and its auth logs shipped somewhere it cannot itself write to. It should not run anything else.

The playbook repository. Branch protection and required review on the branch that automation runs from. A code review on a CM repository is not a style exercise; it is the approval gate for a fleet-wide root change, and it deserves the reviewer attention that implies.

Two supporting habits:

# The vault password comes from a script that fetches it at run
# time, not from a file sitting next to the playbooks
ansible-playbook -i inventory/prod site.yml \
  --vault-password-file /usr/local/bin/vault-pass-from-agent

# Confirm you are on the branch you think you are, every time
git -C /srv/ansible status --short --branch

The --diff secret-leak hazard belongs here too and is covered in full by linux-config-drift-detection: --diff renders the contents of every managed file, so any task templating a credential needs no_log: true and the output needs to be treated as sensitive wherever it lands.

Knowledge check