Why manual does not scale - the case for configuration management

A single host configured manually is manageable. Ten hosts configured manually is annoying. A hundred is impossible. At scale, manual configuration is the source of every snowflake server, every drift, and every “works on my host” bug.

The problems with manual configuration

• Drift: two hosts that started identical end up different because someone ran apt install on one.
• Snowflakes: each host is unique; you cannot reason about the fleet.
• Reproducibility: a new host takes hours to set up manually.
• Auditability: no record of what was changed.
• Error-prone: typos, missed steps, partial configuration.
• Recovery: a disaster recovery requires rebuilding manually.

The discipline: configuration management

Configuration management (CM) is the discipline of:

• Desired state: declare what a host should look like.
• Idempotency: applying the same configuration multiple times yields the same result.
• Version control: configuration is in git, with history and review.
• Drift detection: regularly check that reality matches the declared state.
• Automation: changes go through automation, not manual edits.

CM tools

For most Linux sysadmin work, Ansible is the right choice: agentless (just SSH), YAML (declarative), and large ecosystem.

What CM does

• Provisioning: install packages, create users, configure services.
• Configuration: write config files with the right contents.
• Idempotency: applying twice does not break anything.
• Orchestration: roll out changes across the fleet.
• Drift detection: detect when reality differs from declared.

What CM does not do

CM is not:

• A replacement for monitoring (CM configures; monitoring observes).
• A replacement for backup (CM restores state; backup protects state).
• A replacement for change management (CM automates; change management approves).

CM is the substrate. It enforces what was approved.

The migration path

Most teams do not start with CM. They start with a script, then a bigger script, then a framework. The migration:

1. Document the current state of one host.
2. Capture it in CM (Ansible playbook).
3. Test on a non-production host.
4. Apply to one production host.
5. Verify.
6. Roll out to the fleet.

The first CM-managed host takes days. The hundredth takes minutes.

Knowledge check