Configuration management rollback — Ansible idempotency and the next-run restoration

Configuration management rollback is the most counterintuitive of the five mechanisms because it is not a mechanism at all. There is no ansible rollback. There is no special procedure. The rollback is the next playbook run, which reconciles reality to whatever the playbook currently declares. To roll back, edit the playbook to declare the previous state, run the playbook, and the system is restored. Idempotency is the property that makes this work: the run that restores the previous state is the same run that would have applied the previous state originally.

Why there is no rollback command

The reason is in the model. A configuration management tool like Ansible does not apply a sequence of changes; it declares a desired state and converges reality to that declaration. The play is not “do these things”; it is “the system should look like this”.

flowchart LR
    A["Playbook declares desired state"] --> B["Run"]
    B --> C["Reality reconciled"]
    C --> D{"Matches desired?"}
    D -- "yes" --> E["No changes; OK"]
    D -- "no" --> F["Apply minimal changes to reconcile"]

The reconciliation is idempotent. A second run, against a system that already matches the declaration, makes no changes. A third run, against a system that has drifted, makes the minimum changes to restore the declaration. The run is the rollback because the run is the only thing that exists.

The rollback procedure:

1. Edit the playbook to declare the previous desired state. This is typically a git revert of the commit that introduced the bad change.
2. Run the playbook. ansible-playbook site.yml or equivalent.
3. The run reconciles the system to the previous declared state. No special flags; no separate command.

The cost of the model is that every change is a forward declaration; the benefit is that every change is also a rollback declaration. The two are the same operation.

The next-run restoration property

The property that makes this work is that the playbook run is a function of the playbook’s current content and the system’s current state. Given a playbook, the post-run state is deterministic; given a different playbook, the post-run state is different in a deterministic way.

ansible-playbook --check --diff site.yml

The --check flag runs the playbook in dry-run mode; no changes are applied, but the playbook reports what would change. The --diff flag adds a unified diff of the changes. Together they are the rollback’s pre-flight: the operator can confirm that the playbook, as currently written, will reconcile reality to the previous declared state before actually applying it.

A team that runs the playbook in check mode after a rollback commit knows whether the rollback will succeed before touching production. The check is the production discipline of declared-state rollback.

What declared-state rollback can and cannot do

The boundary declared-state rollback crosses is the playbook’s declaration. What it can do:

• Restore files to the content and permissions the playbook declares.
• Restore packages to the version the playbook pins.
• Restore services to the running state the playbook declares.
• Restore configuration to the values the playbook templates.

What it cannot do:

• Restore data that the previous run’s managed services produced. A database whose content the playbook does not manage is invisible to the rollback.
• Roll back an action that has no inverse in the declared state. A user account that the previous playbook created is not removed by the rolled-back playbook unless the new playbook declares “this user should not exist”.
• Roll back resources Terraform owns. Configuration management and Terraform overlap; the playbook cannot restore a cloud resource that Terraform deleted.
• Roll back across playbook format changes. A playbook that was rewritten from imperative to declarative cannot roll back to the imperative version’s behaviour.

When the previous declared state is the problem

The failure mode unique to declared-state rollback is when the previous declared state is itself the problem. Two cases:

• The previous state is wrong. The rolled-back playbook declares a state the team now knows is wrong. The rollback restores the bad state. The fix is a new commit that declares the correct state, not a revert of the bad commit.
• The previous state has external dependencies that have moved. A playbook that declares “the configuration file at /etc/api/config.yaml should contain this API key” rolls back to the previous API key; if the API key has been rotated, the rollback restores a stale credential. The fix is a credential rotation, not a rollback.

The discipline is to verify the declared state is correct before rolling back to it. A git revert of a bad commit that lands on a state that is itself bad is two bad states.

When to use check mode

The --check flag is the rollback’s pre-flight. The cases:

• Before any rollback that affects production. The check confirms the playbook’s reconciliation plan matches the operator’s intent.
• After a long-running rollback that touches many hosts. A partial rollback may have left some hosts in an intermediate state. The check identifies the gap.
• When the playbook’s idempotency is in doubt. A playbook that uses imperative modules may not be idempotent; the check identifies what the run would change.

ansible-playbook --check --diff --limit=$HOST site.yml

The --limit flag scopes the run to a single host; the check on one host is a safe way to verify the rollback plan before applying it fleet-wide.

Production discipline

1. Write declarative playbooks. Use modules that declare state (file, service, package, template); avoid imperative modules (command, shell) for stateful changes.
2. Test every rollback with --check --diff before applying it. The dry-run is the rollback’s pre-flight.
3. Verify the previous declared state is correct before rolling back to it. A revert of a bad commit that lands on a bad state is two bad states.
4. Treat configuration management and Terraform as separate domains. A resource Terraform owns is not Ansible’s to roll back; a resource Ansible owns is not Terraform’s to roll back.
5. Run the playbook after every commit, including rollback commits. The run is the verification; the no-change result is the confirmation.

Cross-course references

• Ansible for Production Sysadmins - Part XXXIV (ConfigMgmt) covers idempotency and the declared-state model.
• This course, Part LI (Ansible CI) covers the pipeline checks that validate idempotency in CI.
• This course, Part XXXIV (ConfigMgmt) covers the Linux-system-level analogue of declared-state management.

Quiz