Host enrolment and decommissioning - joining and leaving the fleet

The management plane lesson described the components a fleet needs. This lesson is about the two transactions that connect a machine to all of them at once, and about the fact that only one of those transactions usually gets automated.

Every fleet has hosts that are serving production traffic and are not in monitoring. Nobody put them there deliberately. They are the residue of an enrolment that half succeeded and did not say so.

What “enrolled” means

A host is enrolled when every one of these is true. Fewer than all of them is not a smaller degree of enrolment; it is a host that will surprise somebody.

The fourth row carries a subtlety worth stating: “is being scraped” and “an alert would fire” are different claims. A host in the scrape config with no alert rule matching its labels is monitored in the sense that data exists and unmonitored in the sense that matters.

The bootstrap trust problem

Enrolment needs the host to prove it is entitled to a machine identity. But the machine identity is what the plane uses to authenticate hosts. Something has to break the circle.

Trust on first use, with human approval. The host generates a key, submits a signing request, and an operator approves it.

# Puppet
puppetserver ca list --all
sudo puppetserver ca sign --certname web-03.example.com

# Salt
salt-key -L
sudo salt-key -a web-03.example.com

Honest and auditable at small scale. It degrades into rubber-stamping the moment approvals outnumber the attention available, and a rubber-stamped approval is not a control.

Short-lived bootstrap tokens. The provisioning system mints a single-use, time-limited token and hands it to exactly one host. The host exchanges it for a durable identity and the token is spent. This is the general answer: the trust comes from the provisioner, which already knew it was building this host.

Platform attestation. In a cloud, the instance identity document is signed by the platform and states the instance ID, image and account. On bare metal, TPM-based attestation plays the same role. The strongest option where it is available, because the host proves what it is rather than proving it knows a secret.

Enrolment must be idempotent and must fail loudly

Two properties turn enrolment from a script into a process:

Re-runnable. Enrolment gets interrupted - a reboot, a network blip, a step that timed out. Running it again must converge rather than create a second inventory record and a second certificate. Every step is “ensure”, never “create”.

Loud on partial failure. A script that enrols a host into seven systems and fails on the sixth must not exit zero. This is where half-enrolled hosts come from: the provisioning job went green because the last step succeeded, and the log shipper was never configured.

#!/usr/bin/env bash
# Enrolment driver: any failure is a failed enrolment.
set -euo pipefail

FQDN=$(hostname -f)

enrol_identity
enrol_inventory
enrol_config_management
enrol_monitoring
enrol_logging
enrol_backup
enrol_patching

# Prove it, from outside the host, before declaring success.
verify_enrolment "$FQDN"

The last line is the one that matters. Verification asks the management systems whether they know about the host, rather than asking the host whether it thinks it told them.

Finding half-enrolled hosts

Reconcile the systems against each other. Every pairwise difference has a name and a meaning:

The mechanic is a sorted list from each source and comm:

# One host per line, sorted, from each authority
inventory_hosts   | sort -u > /tmp/inventory.txt
cm_hosts          | sort -u > /tmp/cm.txt
monitoring_hosts  | sort -u > /tmp/mon.txt

# In configuration management but not in inventory
comm -13 /tmp/inventory.txt /tmp/cm.txt

# In inventory but not monitored
comm -23 /tmp/inventory.txt /tmp/mon.txt

Run it on a schedule and alert on a non-empty result. A reconciliation that runs quarterly finds the same problems a quarter late; the value is in it running daily and staying empty.

The fourth row - reachable and in nothing - needs a source outside the management plane, because by definition the plane does not know about those hosts. Cloud provider instance listings, DHCP leases, switch ARP tables and IP address management records all work.

Decommissioning

Almost every fleet automates enrolment and performs decommissioning by hand, which is why the monitoring system carries dead hosts, the CA holds three certificates per machine, and DNS resolves names for hardware that left the building two years ago.

The reverse checklist, in an order that matters:

1. Drain. Remove from load balancers and service discovery. Confirm traffic is zero before anything else.
2. Take the last backup, if the data has any value, and record where it is.
3. Rotate anything shared. Any credential this host held that other hosts also hold must be rotated, because the disk is about to leave your control.
4. Revoke the machine identity. Certificate revoked, keytab deleted, cloud role detached.
5. Remove from configuration management.

sudo puppetserver ca clean --certname web-03.example.com
sudo salt-key -d web-03.example.com

6. Remove monitoring and log shipping. Late in the order deliberately - see the callout.
7. Remove DNS records, forward and reverse.
8. Wipe or destroy the storage, per the data classification.
9. Mark the inventory record decommissioned. Do not delete it. You will need to answer questions about this host for years, and the record is the only thing that can.

One script, both directions

Enrolment and decommissioning touch the same eight systems. When they are separate scripts owned by separate people, they drift, and the drift is always in the same direction: enrolment gains a system, decommissioning does not learn about it, and that system slowly accumulates dead entries.

Keep them in one place with one list of components and a direction, so adding a ninth system to the plane is a single change that necessarily handles both.

# The shape that stays consistent
fleet-lifecycle enrol      --host web-03.example.com --role web --env production
fleet-lifecycle verify     --host web-03.example.com
fleet-lifecycle decommission --host web-03.example.com --confirm

verify earns its place separately from both: it is what a reconciliation job calls, what an on-call engineer runs when a host is behaving oddly, and what the enrolment script calls at the end before reporting success.

Knowledge check