Manual administration, honestly

Most introductions to configuration management open by making manual administration look foolish. A cartoon of an engineer SSHing into a hundred servers in sequence, and then the tool arrives.

That framing is both wrong and counter-productive. It is wrong because manual administration is genuinely the correct choice under conditions that occur constantly in real work. It is counter-productive because a reader who has been told that everything manual is bad will automate things that should not be automated, which is its own failure mode and a more expensive one.

This lesson makes the case for doing it by hand, and then says precisely where the case stops holding.

When manual is right

One host, one time

A change that will be applied to exactly one host, exactly once, and never again. Enabling a debug flag on a single application server to capture a stack trace. Adjusting a queue depth on the one machine with the unusual disk controller. Clearing a stuck lock file.

Automating this costs more than doing it. Writing the automation, testing it, reviewing it and storing it is real work, and it produces an artefact that will be read exactly zero more times. The cost of the automation is paid in full and the benefit is never collected.

The trap in this category is not the first change. It is the fourth change that you also classified as one-off, having forgotten the three previous ones. Lesson 8 gives you a way to notice that.

A diagnosis you do not yet understand

This is the strongest case, and the one most often missed.

Automation encodes a decision. You cannot encode a decision you have not made. During diagnosis you are not applying a known fix — you are forming a hypothesis, testing it, discarding it, and forming another. That loop is fundamentally interactive and it is fast by hand:

$ ss -tnp state established '( dport = :5432 )' | wc -l
journalctl -u app --since '10 min ago' | grep -c 'pool exhausted'
cat /proc/sys/net/ipv4/tcp_fin_timeout

412
88
60

Nobody writes that as a playbook, because the third command only exists because of what the second one returned. Writing it as automation before you know the answer means writing automation for a problem you have not identified.

The discipline that matters here is not “automate it”, it is capture the outcome once you have one. The diagnosis is manual; the fix, if it will recur, is not.

The estate is genuinely small and genuinely stable

Three hosts, changed twice a year, run by two people who both understand them. This is a real configuration, particularly for internal infrastructure, small hosting setups and appliances.

The honest calculation is below, and for this estate it comes out against automation. It is worth stating plainly because the industry consensus does not: at this size the automation is likely to be a net cost for years.

The system does not permit it

Some things cannot be automated with the tools under discussion. A firmware update that requires a physical console. A vendor appliance whose only interface is a web UI. A change that requires two-person authorisation by policy. A system whose vendor support contract is void if you touch the configuration outside their tooling.

Naming these is not defeatism. It is scope control, and the alternative — building fragile automation against an unsupported interface — is a recognisable failure mode that produces the worst of both worlds.

Where the curve turns

Here is the arithmetic, stated honestly. There is no published constant for this and anyone who gives you one has made it up; what follows is a model you populate with your own numbers.

For a change applied by hand to N hosts:

• Cost of doing it is roughly linear in N. Ten hosts take about ten times as long as one. There are small economies — you get faster by the fourth host — and small diseconomies, because attention degrades. Call it linear.
• Cost of an error is also roughly linear in N, but with a much larger constant. A typo applied to one host is a five-minute fix. The same typo applied to forty is an incident with a duration measured in the time it takes to work out which hosts got it.
• Probability of at least one error rises with N and rises faster with repetition, because the work is monotonous. This is the term that actually decides the question, and it is the one people leave out.

For the automated version:

• Setup cost is a fixed sum paid up front: writing it, testing it, reviewing it, and building somewhere to test it. It is substantially larger than most estimates.
• Marginal cost per host is near zero. This is the whole point.
• Cost of an error is N times the per-host cost, applied simultaneously, with no opportunity to notice partway. This is the term that people leave out on the other side.

Put together, automation wins when

setup_cost < N × repetitions × (manual_cost_per_host + error_rate × error_cost)

and the variable that dominates in practice is repetitions, not N. A change applied to two hundred hosts once may still not be worth automating. A change applied to five hosts every week almost certainly is.

The error term deserves its own treatment

The reason manual work fails at scale is not that it is slow. Slow is tolerable and often affordable. It fails because of what repetition does to accuracy.

Consider a change applied by hand to forty hosts. Suppose you are good and make a mistake on one host in fifty. The probability that all forty are correct is 0.98^40, which is about 0.45. A better-than-even chance that the fleet is now inconsistent — and you will not know which host, because the mistake you made is by definition the one you did not notice.

Now apply the previous lesson: that inconsistency is drift, it produces no symptom today, and it surfaces during an unrelated incident in four months.

This is why “be more careful” is not a solution. At one error in five hundred — considerably better than most people sustain across forty repetitions of a monotonous task — the fleet is still wrong about 8% of the time.

The two kinds of work

A useful separation, and one worth applying deliberately:

The failure at the top-right — automating exploration — produces playbooks full of conditionals encoding hypotheses their author later disproved. The failure at the bottom-left — doing repeatable work by hand — produces the arithmetic above.

Most real work starts in the left column and moves to the right. The skill is noticing when it has moved.

$ sort ~/.bash_history | uniq -c | sort -rn | head -15

     31  systemctl restart app
   22  ssh web03
   14  journalctl -u app -n 200
   11  sed -i s/old.ntp.example.com/new.ntp.example.com/ /etc/chrony/chrony.conf
    9  systemctl restart chrony

The fourth line in that output is the interesting one. A sed -i against a configuration file, run eleven times, is repeatable work being done by hand — and it is also being done in a way that leaves no record of which hosts received it.

What manual administration cannot give you

Stated once, so the rest of the part can build on it. Manual administration, however careful, does not produce:

1. A machine-readable statement of intended state. The procedure is prose. You cannot compare a host to prose.
2. Per-host structured evidence of outcome. “I did all forty” is a claim by a tired person, not a record.
3. Cheap repetition. Re-running the change to confirm it took is as expensive as the original change, so nobody does it.
4. A review artefact. There is nothing for a second person to read before the change is applied.

Item 3 is the one to hold on to. Almost everything valuable about configuration management follows from making a second run cheap — which is what makes convergence, drift detection and safe resumption after a partial failure possible at all.

Knowledge check