Drift remediation - bringing hosts back to desired state

Drift remediation brings hosts back to the desired state and prevents the drift from recurring. This lesson covers the runbook.

The order of the runbook is the whole lesson: capture, investigate, decide, remediate. Reverting first feels efficient and is the single most common way an automated drift programme re-opens a resolved incident.

Step 1: Capture the evidence

Before you touch anything, take a copy and find out who changed it:

# Snapshot the drifted file with its metadata intact
sudo cp -a /etc/nginx/nginx.conf \
  /var/tmp/drift-$(date +%FT%H%M)-nginx.conf

# Who changed it, and when (requires an auditd watch on the path)
sudo ausearch -f /etc/nginx/nginx.conf -ts recent

# What exactly differs from desired state - reports only, changes nothing
ansible-playbook -i prod site.yml --check --diff --limit web01

--check --diff is the read-only form: it prints the diff the CM run would apply without applying it. Run it scoped to the single affected host with --limit.

Step 2: Investigate

Why did the drift happen?

• Was it a manual change by an operator?
• Was it a package update that rewrote a config?
• Was it an emergency change?
• Was it a test that was not cleaned up?

The decisive question is the third one. Check the incident record and the change log for this host before you decide anything. If a change was made during an incident, confirm the incident is closed and that the mitigation is no longer load-bearing.

Document the cause in the runbook. The same drift should not recur.

Step 3: Decide

Step 4: Remediate

Only now apply the desired state, and scope it to the host you investigated:

# Apply the CM to the affected host only
ansible-playbook -i prod site.yml --limit web01

# Verify drift is gone
ansible-playbook -i prod site.yml --check --diff --limit web01

The output should be empty (or near-empty for known non-managed files).

The --limit is not optional. Without it this command drives a run across every host in the prod inventory - a fleet-wide production change, triggered by a single host’s drift, with no batching and no canary.

Update CM to include intentional changes

If the drift is intentional (e.g. an operator made a needed change):

1. Add the change to the CM.
2. Re-apply CM; the change is now desired state.
3. Drift detection no longer flags it.

If the drift is unintentional, do not add it to CM. Revert the change.

Prevent future drift

The prevention patterns:

• No manual changes: all changes through CM.
• Ad-hoc SSH alerting: detect manual changes via SSH sessions.
• Configuration file monitoring: alert on changes to /etc outside CM.
• Periodic reapply: run CM on a schedule to revert drift automatically.

Periodic reapply is the one that needs care. Automatic remediation is itself a production change, so it has to obey the same controls as any other production change.

Consider what this apparently harmless crontab line does:

# WRONG - do not deploy this
0 4 * * * root ansible-playbook -i /etc/ansible/prod /etc/ansible/site.yml

There is no --limit, so the inventory is the whole fleet. Every host that runs this line drives its own fleet-wide run. On 200 hosts that is 200 concurrent full-fleet runs starting in the same minute, all hammering the inventory source and the secrets backend, and overlapping on each other’s package state. Any task that notifies a service restart restarts that service on every host at once - a total outage of that service, with no gradual signal that anything is wrong.

The correct pattern is a pull model: each host remediates only itself, on a jittered schedule, with a lock so runs cannot overlap.

# /etc/systemd/system/config-pull.service
[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /run/config-pull.lock \
  /usr/bin/ansible-pull -U https://git.example.com/ops/cm.git -C main \
    --only-if-changed local.yml

# /etc/systemd/system/config-pull.timer - spread the fleet out
[Timer]
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=1800

flock -n makes a second run exit immediately rather than stack on the first. RandomizedDelaySec=1800 spreads the fleet across half an hour so a bad commit does not land everywhere at the same instant. --only-if-changed skips the run entirely when the repository has not moved.

If you must keep a push model, scope it to the local host and batch it:

ansible-playbook -i /etc/ansible/prod /etc/ansible/site.yml \
  --limit "$(hostname -s)"

and set an abort threshold in the play itself:

- hosts: web
  serial: "5%"
  max_fail_percentage: 0

serial: "5%" runs the fleet in waves. max_fail_percentage: 0 stops the whole play the moment a wave fails, so a bad commit reaches one wave rather than all of production.

Document the remediation

For each drift remediation:

DRIFT REMEDIATION
=================
Host: web01
Date: 2026-08-09
Drift: /etc/nginx/nginx.conf was manually modified
Evidence: /var/tmp/drift-2026-08-09T0912-nginx.conf (copy taken before any change)
Cause: Operator edit during INC-4471 on 2026-08-08 - added
       limit_req to shed load from a scraper
Incident state: INC-4471 closed 2026-08-08 21:40; upstream rate
       limiting now handled at the edge, so the local limit is
       no longer load-bearing (confirmed with incident owner)
Decision: safe to revert
Action: ansible-playbook -i prod site.yml --limit web01
Prevention: config-pull.timer on web01, jittered, host-scoped
Followup: none

Read the Incident state line. That line is the reason this revert was safe. Without it, the correct action was to leave the file alone and escalate.

The documentation is the audit trail. The next drift investigation can look back and see what happened.

Knowledge check