Choosing an approach for a given estate

Four approaches have now been argued on their merits: manual administration, shell scripting, configuration management, and replacement from an image. This lesson is the framework for choosing between them, and it closes Part I without concluding that the answer is Ansible — because for one of the three estates worked below, it is not.

Two framing points before the dimensions.

Choose per tier, not per estate. The question “what should we use?” has no answer. The question “what should we use for the application servers, and separately for the database, and separately for the switches?” has three answers, frequently different ones. Every estate worked below comes out hybrid.

The approaches compose. Replacement needs configuration management to build the image. Configuration management needs a script to bootstrap the controller. Everything needs manual work during diagnosis. Treating them as rival camps is the most common analytical error in this area.

The six dimensions

1. Host lifetime

How long does a host live before it is destroyed?

• Hours to days. Drift has no time to accumulate. Replacement is natural and convergence is largely wasted work.
• Weeks to months. Either model works. The decision moves to other dimensions.
• Years. Replacement is a migration, not a deployment. Convergence is the model.

This is the single most decisive dimension, which is why it is first.

2. Change frequency

How often does the intended configuration change?

Low frequency with long-lived hosts is the case where careful manual work with a written runbook genuinely competes. High frequency is where the setup cost of automation is recovered fastest, and where the error term from lesson 3 does the most damage if you do not.

Note this is the frequency of intended change, not of incidents.

3. Statefulness

How much irreplaceable local state does the host hold?

• None. Destroy it freely.
• Cache or rebuildable data. Destroying it costs rebuild time.
• Authoritative data. Destroying it is a restore, and replacement stops being a deployment mechanism.

State is what converts “replace the host” from a routine operation into a maintenance window with a rollback plan.

4. Evidence requirements

Does somebody outside the team need to be shown what changed, when, by whom, and that it matched an approved intention?

Regulated estates, and any estate with a serious audit obligation, need an artefact trail. Automation produces one almost as a side effect — version control history, review records, run logs. Manual work produces one only through discipline that must be sustained indefinitely.

If the answer here is a firm yes, it pushes hard toward automation even when the other dimensions are marginal, because the evidence is a deliverable in its own right.

5. Team size and continuity

How many people, and how long do they stay?

• One person. Automation concentrates knowledge in that person, and their departure takes it. Written runbooks transfer better to a successor than an unfamiliar repository does. This is an uncomfortable finding and it is the honest one.
• Small team with turnover. Automation is a substantial win, provided review is real, because it makes the estate legible to people who did not build it.
• Large team. Automation is close to mandatory. Coordinating manual changes across many people is where drift is generated fastest.

6. Who is on call, and what they will be asked to do

The dimension people leave out.

If the on-call engineer is expected to fix things at 03:00, the estate needs to be fixable at 03:00 by someone who did not build it. That argues for procedures a person can read and follow, and against automation whose failure modes require its author to interpret.

If the on-call response is “replace the instance”, the estate needs replacement to be fast, reliable and safe at 03:00 without a human composing a plan.

An estate whose on-call runbook says “run the playbook” and whose playbook nobody on call is confident running has a genuine problem that neither model fixes.

The two axes that carry most of the signal

Dimensions 1 and 2 dominate often enough to be worth a picture.

flowchart TB
  subgraph HIGH["High change frequency"]
    A["Short-lived hosts<br/>REPLACEMENT<br/>build image, deploy, discard"]
    B["Long-lived hosts<br/>CONFIGURATION MANAGEMENT<br/>declare, converge, report"]
  end
  subgraph LOW["Low change frequency"]
    C["Short-lived hosts<br/>SCRIPTED BUILD<br/>a build script is enough"]
    D["Long-lived hosts<br/>MANUAL + RUNBOOK<br/>if the estate is also small"]
  end

Read the four cells as regions rather than points, and read the boundaries as soft. An estate near a boundary is genuinely ambiguous and the remaining four dimensions decide it.

Two cautions about the diagram. The bottom-right cell says “if the estate is also small” for a reason: long-lived hosts changed rarely but numbering four hundred still need convergence, because the error term across four hundred manual repetitions dominates everything else. And the bottom-left cell is the one people most often over-engineer — a short-lived host built rarely does not need an image pipeline, it needs a build script somebody can read.

Three estates

Estate A: the regional hosting provider

Forty physical servers in two data centres. Debian, running customer workloads. Hosts live five to eight years. Configuration changes maybe monthly — a package update, a firewall rule, a monitoring change. Three-person team, one of whom has been there nine years. A compliance obligation requires demonstrating that security patches are applied within a stated window, with evidence.

Answer: configuration management, unambiguously. Every dimension points the same way. Replacement is unavailable because the hardware is the hardware. The evidence requirement alone would justify it: a convergence run produces a per-host record of what was applied and when, which is exactly the artefact the auditor wants and which manual patching produces only through sustained discipline.

The patching workflow is the highest-value thing to automate first — it is frequent, it is fleet-wide, it is what the audit asks about, and it is where manual repetition does the most damage.

Manual administration remains correct for the storage array and for diagnosis. That is not a gap in the answer; that is the answer.

Estate B: the consumer web platform

Two hundred application instances in a cloud, autoscaled between 80 and 400. Instances live about eighteen hours. Deployments happen six to twelve times a day. All state is in a managed database and an object store. Twelve engineers. On call is a rotation, and the standing instruction for an unhealthy instance is to terminate it.

Answer: replacement, and configuration management is the wrong tool for the application tier. Converging an instance that will not exist tomorrow is wasted work, and worse, it introduces a second source of truth about instance state competing with the image.

This is the estate where a reader who finished Part I convinced that Ansible is the answer would do damage — building a convergence estate for hosts whose correct lifetime is eighteen hours, and slowing deployments to do it.

Where configuration management does belong here: building the image. Something declares what goes into it, and that is a declaration of desired state applied to a build host. The tool is the same and the target is different.

Estate C: the university research group

Eleven Linux workstations and two GPU servers. Users are researchers who install their own software constantly and need root on their own machines. Configuration changes are rare at the base-OS level and constant at the user level. One part-time systems administrator, a PhD student, in the role for about two years before they graduate.

Answer: manual administration with written runbooks, plus a small narrow automation for the two things that genuinely repeat.

Work through why. Host lifetime says convergence. Statefulness says do not replace. But dimension 5 is decisive in the other direction: a single administrator with guaranteed turnover in two years. An Ansible repository written by this year’s PhD student and inherited by next year’s is an artefact the successor cannot read describing hosts they have not seen — which is a worse handover than a written build runbook.

The genuine automation candidates are narrow: user account provisioning, and the backup verification. Both repeat, both are error-prone, both are small enough to be readable by a successor.

Readiness: can this estate adopt it at all?

Suppose the framework says configuration management. There is a second question, and it is the one that determines whether the adoption succeeds.

Work through these honestly. A “no” is not disqualifying on its own, but two or more is a signal to fix the gap before writing any automation.

1. Can you enumerate your hosts? Not approximately. Is there a list, is it current, and does anything keep it current? Automation against an inventory that is missing hosts silently excludes them, and those hosts become the drift.

2. Do you know who owns each service? Somebody has to be able to approve a change to it and be paged when it breaks. An estate with unowned services will produce automation nobody dares apply to them.

3. Is there a non-production environment? Or a credible plan to build one. See lesson 7 on what happens without it.

4. Is there somewhere the declaration will live, with review? A repository with protected branches and a second reader. Automation in a directory on one person’s laptop is a fleet-wide change mechanism with no review step.

5. Will somebody own it after the project ends? Structurally, by role, not by goodwill.

6. Can you reach every host from one place? Push automation requires the controller to open connections. NAT, air gaps and DMZ segments are design constraints, not blockers, but they need to be known before the controller is placed.

$ comm -3 \
<(sort inventory-hosts.txt) \
<(ssh mon01 'curl -s http://localhost:9090/api/v1/targets' | jq -r '.data.activeTargets[].labels.instance' | cut -d: -f1 | sort -u)

	build-legacy02
jump01
web09

Three findings in that output, and each is a different problem. Two hosts are being monitored but are not in the inventory, so nothing will ever converge them — they are unmanaged and nobody has noticed. web09 is in the inventory but is not being monitored, which is either a decommissioned host still listed or a live host with no observability. Both are worth knowing before you write a line of automation.

Knowledge check