Runbook: Onboard a managed host

Onboarding is the moment a host stops being something a person occasionally logs into and starts being something a playbook converges without asking. The first real run is the risky one: every role in its function group applies at once, to a host that has never had them applied, and every default that was fine on a freshly built machine now lands on one that has been in service for three years.

This runbook front-loads that risk into a check-mode review that the host owner reads.

When to use this runbook

• A newly provisioned host is joining the managed estate.
• A long-lived host is being brought under management for the first time.
• A host is moving from another configuration management system.
• A host that was removed from management is coming back.

Blast radius

One host - but every role that its function group applies.

The number that matters is not “one host”, it is “how many roles will run against it for the first time”. Read the group’s role list before Step 7, because that is the list of things about to change on a machine that has never seen them.

Inputs

• Hostname, management IP, SSH port, and the bastion if there is one.
• The function group it joins, and that group’s role list.
• The host owner and their agreement.
• The maintenance window for Step 9.
• The sudo policy the roles require.

Step 1: Record the pre-onboarding state

Do this before anything else. It is both your rollback material and your evidence for what the converge actually changed.

HOST=web03.example.com
ssh "$HOST" 'systemctl list-units --type=service --state=running --no-pager --plain' \
> "baseline-$HOST-services.txt"
ssh "$HOST" 'rpm -qa 2>/dev/null || dpkg-query -W' \
| sort > "baseline-$HOST-packages.txt"
ssh "$HOST" 'sudo tar -C /etc -cf - ssh sudoers.d sysctl.d' \
> "baseline-$HOST-etc.tar"

Keep these off the host. A baseline stored on the machine you are about to reconfigure is a baseline you may not be able to read when you need it.

Step 2: Establish the host key from a trusted channel

ssh-keyscan -t ed25519 web03.example.com | tee /tmp/newhost.pub

# Compare against a source that is NOT the network path you just used:
#   the provisioning system's record, the console, or the build log
ssh-keygen -lf /tmp/newhost.pub

Then add it deliberately:

- name: Trust the host key for the new managed host
ansible.builtin.known_hosts:
  name: web03.example.com
  key: "{{ lookup('file', '/tmp/newhost.pub') }}"
  path: /home/ansible/.ssh/known_hosts
  state: present
delegate_to: localhost

Step 3: Install the automation public key

The bootstrap connection here is a human one - your own account with your own key - because the automation account does not have access yet.

- name: Bootstrap automation access on a new host
hosts: web03.example.com
become: true
vars:
  ansible_user: "{{ bootstrap_admin_user }}"
tasks:
  - name: Automation service account exists
    ansible.builtin.user:
      name: ansible
      shell: /bin/bash
      create_home: true
      state: present

  - name: Controller public key is authorised
    ansible.posix.authorized_key:
      user: ansible
      key: "{{ lookup('file', '/home/ansible/.ssh/id_ed25519.pub') }}"
      state: present
      exclusive: false

Then confirm by hand before trusting it:

sudo -iu ansible ssh -o BatchMode=yes web03.example.com 'id; hostname -f'

BatchMode=yes matters. Without it, SSH may fall back to a password prompt and you will conclude that key authentication works when what worked was you typing a password.

Step 4: Grant the narrowest sudo rule that works

# /etc/sudoers.d/50-ansible  (mode 0440, root:root)
ansible ALL=(root) NOPASSWD: ALL
Defaults:ansible !requiretty

NOPASSWD: ALL is the honest starting point for a general-purpose configuration management account, and pretending otherwise produces sudoers files full of command paths that the next role breaks. What makes it defensible is the rest of the design: a dedicated account, a key that only the controller holds, and an audit trail. If your estate requires narrower, narrow it to the specific commands the group’s roles run and expect to revisit the file whenever a role changes.

Validate before installing, always:

sudo visudo -cf /etc/sudoers.d/50-ansible
sudo install -m 0440 -o root -g root /tmp/50-ansible /etc/sudoers.d/50-ansible

A syntactically invalid file in sudoers.d breaks sudo for everyone on the host, including the account you would use to fix it. visudo -c before install is the difference between a change and an incident.

Confirm it works the way automation will use it - non-interactive, no TTY:

sudo -iu ansible ssh -T -o BatchMode=yes web03.example.com 'sudo -n id'

ssh -T suppresses the TTY, which is exactly how Ansible connects. A sudo rule that works when you are logged in and fails without a TTY is the classic requiretty failure, and it will present as a become failure on the first converge.

Step 5: Confirm the remote interpreter

ansible -i inventories/staging web03.example.com -m setup \
-a 'filter=ansible_python*'

Read ansible_python_interpreter and ansible_python.version. Ansible discovers an interpreter per host, and two hosts in the same group choosing different ones is a real source of “works on one, fails on the other”.

If no Python is present at all, setup fails and the bootstrap needs ansible.builtin.raw, which is the one module that does not require a remote interpreter:

- name: Install a Python interpreter before anything else can run
ansible.builtin.raw: |
  test -e /usr/bin/python3 || (apt-get update && apt-get install -y python3)
changed_when: false

raw is a bootstrap tool, not a general escape hatch. It has no idempotency, no check-mode support and no return structure. The moment Python is present, stop using it.

Step 6: Add to inventory - function group only

all:
children:
  web:
    hosts:
      web03.example.com:
        ansible_host: 192.0.2.13
  # deliberately NOT yet a member of wave1 / canary / patch-tuesday

This is the step people skip and regret. Adding the host to its lifecycle groups at the same time as its function group means the next scheduled fleet run picks it up before anyone has converged it deliberately. Onboard first, enrol in rollouts afterwards.

ansible-inventory -i inventories/staging --graph
ansible-inventory -i inventories/staging --host web03.example.com

Step 7: Ping, facts, then check mode

ansible -i inventories/staging web03.example.com -m ping -o
ansible -i inventories/staging web03.example.com -m setup >/dev/null && echo 'facts OK'
ansible-playbook -i inventories/staging site.yml \
--limit web03.example.com --check --diff | tee "check-web03.txt"

Three connections of increasing depth: transport, interpreter, then the whole site playbook without changing anything.

Step 8: Read the diff with the host owner

This is the review that makes the difference between onboarding and breaking someone’s server.

Look specifically for:

• A config file being replaced wholesale that the host owner had hand-edited.
• A service being disabled because the role’s default says it should be.
• A user or group being removed because it is not in the declared list.
• Firewall rules being replaced with the group’s standard set.
• A package being downgraded to the version the role pins.

Check mode also has limits worth stating out loud here: modules that do not support it are skipped, and tasks conditioned on the result of a command that did not run are evaluated against nothing. A clean check run is necessary, not sufficient.

Step 9: Converge for real

Inside the maintenance window, with the owner available.

ansible-playbook -i inventories/staging site.yml \
--limit web03.example.com --diff | tee "converge-web03.txt"

Point of no return. From this command onwards the host has been changed. Inventory edits no longer undo anything; only the restore path in the rollback section does.

Watch it to completion. Do not start it and walk away on a first converge - the failure you want to catch is a service that restarts and does not come back, and that is a minute of attention, not an hour.

Step 10: Verify, then enrol in rollouts

# Idempotence: the second run must change nothing
ansible-playbook -i inventories/staging site.yml \
--limit web03.example.com --diff

# Services that were running before are running now
ssh web03.example.com 'systemctl list-units --type=service --state=running --no-pager --plain' \
> after-web03-services.txt
diff "baseline-web03.example.com-services.txt" after-web03-services.txt

A second run reporting changed=0 is the real definition of converged. A run that keeps reporting changes on every execution contains a non-idempotent task, and that task will show up as permanent noise in every drift report from now on - which is how drift reports stop being read.

Only now add the host to its lifecycle groups:

    wave1:
    hosts:
      web03.example.com:

Rollback

Before Step 9, rollback is removing the inventory entry and the key. After Step 9, the host has been changed and must be restored.

HOST=web03.example.com

# 1. Restore the configuration captured in Step 1
scp "baseline-$HOST-etc.tar" "$HOST:/tmp/"
ssh "$HOST" 'sudo tar -C /etc -xf /tmp/baseline-'"$HOST"'-etc.tar'

# 2. Restart what the roles touched, using the run log as the list
ssh "$HOST" 'sudo systemctl daemon-reload && sudo systemctl restart nginx'

# 3. Remove automation access
ssh "$HOST" 'sudo rm -f /etc/sudoers.d/50-ansible'
ssh "$HOST" 'sudo userdel -r ansible'

# 4. Remove the inventory entry, then prove access is gone
ansible -i inventories/staging "$HOST" -m ping -o ; echo "expect rc=4, got rc=$?"

The verification in step 4 is the part people omit. An onboarding that was backed out but left the key in place is a host that is not in inventory and still trusts the controller - invisible and reachable at the same time.

Record the host as unmanaged, or the next inventory reconciliation against the CMDB will add it straight back.

Common patterns

Escalation

Escalate when:

• The check-mode diff shows a change to something the host owner did not expect to be managed. That is a conversation about ownership, not a technical decision.
• The host needs broader privilege than the roles justify.
• Another configuration management system is already enforcing state here. Two systems with different opinions produce a host that flips between them, and the symptom is drift that “fixes itself” and comes back.
• No supported interpreter can be installed.