What automation costs · Ansible

The case for automation is usually made with a single number: the time saved per change, multiplied by the number of hosts. It is a good number and it is real.

It is also the only number in the pitch. Everything below is a cost that the estate pays afterwards, indefinitely, and none of it appears in the business case. Not one of these is a reason to avoid automation. They are the things you should know you are buying, because a team that adopts the model without budgeting for them produces the outcome at the end of this lesson — a repository everyone is afraid of.

The costs

Review

Every change to a fleet-wide declaration is a fleet-wide change. That means every change wants a reviewer, and the reviewer needs to understand not just the diff but its blast radius.

That second part is what makes this expensive. A three-line diff that changes a variable in group_vars/all.yml may affect four hundred hosts. A hundred-line diff adding a new role may affect three. Nothing about the size of the diff tells the reviewer which situation they are in, so a competent review involves working out what the change actually touches — which is real work, on every change, forever.

The cost is not the review process. It is that the review requires someone who can read Ansible and knows the estate, and that is a narrower set of people than either skill alone.

A place to test

“Test it on a non-production host first” assumes a non-production host. Estates that do not have one either test in production or do not test, and the second is more common than teams admit in writing.

A test environment that is genuinely useful costs:

Hardware or cloud spend that produces no revenue.
Its own maintenance, because a test environment that has drifted from production tests nothing.
Data, or a defensible substitute for data. Testing a database role against an empty database proves less than it appears to.
Somebody’s time to keep the resemblance current as production moves.

The last one is the one that decays. A test environment is at its most accurate on the day it is built.

A pipeline

Once automation can change production, the things that check it before it does become load-bearing: syntax checking, linting, secret scanning, a check-mode run against a representative host, and integration tests where the risk justifies them.

That pipeline is software. It breaks, it needs upgrading, its dependencies move, and when it is broken people start merging without it — which is exactly the moment it was supposed to be protecting.

The person who understands the repository

This is the largest cost and the one least often named, because it does not look like a cost. It looks like an asset.

An automation estate concentrates knowledge. The person who wrote the roles knows why the variable precedence is arranged that way, why db01 is excluded from one play, what the third when: clause is working around. That knowledge is largely not in the repository, because the repository records what rather than why.

Manual administration spreads knowledge thinly and badly across a team. Automation concentrates it, deeply and well, in one or two people. The second arrangement is far more effective and far more fragile, and it converts into an outage the moment those people leave — not immediately, but at the first change nobody remaining feels able to make.

The estate that now depends on it

Before automation, a host could be changed by anyone with access and a text editor. After automation, changing a host correctly means changing the declaration, getting it reviewed, and running it — which is slower, and which requires the automation to be working.

That is the intended trade. But it means the automation is now on the critical path for routine work, and an outage of the automation is an outage of your ability to change anything. Controllers, credentials and the repository become production infrastructure, which is why this course has a part on controller security and another on rebuilding a lost controller.

Unrun automation is worse than none

This is the failure mode that the whole part has been building toward, and it deserves stating as sharply as possible.

Automation that is not run is worse than no automation, because it is believed.

A team with no automation knows they do not know the state of their fleet. That is uncomfortable and it is accurate, and it makes them check things before they act.

A team with an eight-month-old repository believes they know the state of their fleet. They will answer an auditor’s question by reading the repository. They will plan a migration on the assumption that all application servers are configured the way the role says. They will be confidently wrong, which is a worse position than being aware that you are uncertain.

The repository nobody will run

A team has a comprehensive Ansible estate. It was applied routinely until an incident fourteen months ago, when a run intended for the staging inventory was executed against production, restarted a service fleet-wide during business hours, and caused a forty-minute outage.

The response was reasonable in the moment: a rule that playbooks are run only with explicit approval. In practice approval was hard to get, so runs became rare, so each run had more to correct, so each run was riskier, so approval became harder to get.

Fourteen months later the repository is applied to production approximately never. It is still maintained — changes are still merged into it — because everyone agrees it should be kept up to date. Nobody has run it against production since March.

What the team believes: their fleet is configured as the repository describes. Every conversation about the estate uses the repository as the reference.

What is true: the repository describes an intention that has been diverging from reality for over a year, and the divergence is unmeasured because measuring it would mean running something.

Why the obvious diagnosis is wrong: it looks like a process problem caused by an over-cautious approval rule, and removing the rule is the suggested fix. It is not. The rule was a response to a real incident, and removing it restores the conditions that caused the outage. The actual defect is that the team has no way to run the declaration without changing anything, so every question about state requires permission to make changes.

The way out is measurement decoupled from correction. A scheduled check-mode run against production changes nothing, needs no approval on change-management grounds, and produces a per-host divergence report. That report converts an unmeasurable risk into a work list, and — more importantly — it means the next real run is small.

Read-only / Safemeasurement decoupled from correction— Check mode changes nothing on any host. This is the run that a team stuck in the approval loop above can execute today, because it is a read of state rather than a change to it.

$ ansible-playbook -i inventories/production site.yml --check --diff --limit web

PLAY RECAP *********************************************************************
web01   : ok=22  changed=3  unreachable=0  failed=0
web02   : ok=22  changed=3  unreachable=0  failed=0
web03   : ok=20  changed=5  unreachable=0  failed=0
web04   : ok=25  changed=0  unreachable=0  failed=0

Illustrative output

Read that as a drift report. Three of the four web servers differ from the declaration in three respects; web03 differs in five; web04 matches. That is a work list, produced without changing anything, and it is available to a team that is not permitted to change production.

When the estate cannot afford it

The honest conclusion of a cost list is that some estates should not pay it, and this course would be dishonest not to say so.

Consider not adopting configuration management if:

The estate is small and static, and will stay that way. Three hosts changed twice a year. The setup cost will not be recovered.
There is no second person. A single administrator who leaves takes the repository with them, and the next administrator inherits an artefact they cannot read describing hosts they have not seen. Written runbooks would have transferred better.
There is nowhere to test and no appetite to build one. Automation applied to production without a test path is a fleet-wide change mechanism with no safety net. That is a worse position than manual work.
Nobody will own it. Not “nobody has time this quarter” — nobody, structurally, for the lifetime of the estate. An unowned declaration becomes the snowflake in lesson 5.

Note what is not on that list: team inexperience, and the estate being on physical hardware. Both are commonly given as reasons and neither is one. Inexperience is a reason to start small. Physical hardware is the case where convergence is most clearly the right model, because replacement is not available.