Golden images, drift and the reconstruction gap · Backup & DR

Configuration management could not reconstruct everything on the previous lesson’s list, and the usual answer to that shortfall is to pair it with an image. The image supplies a base too awkward to describe in code, the code supplies whatever is specific to the host, and between them they claim to reproduce a working system on demand. That claim is a reconstruction promise, stated in the present tense about a future event. Drift is the interest it accrues: every change made to a running system and not made to its definition raises the balance quietly, and the balance falls due at the one moment nobody wants a surprise.

A definition describes the system as defined, not as run

A backup and a definition are both described as “what we need to come back”, and the phrase hides a difference that decides how a recovery goes. A backup is a copy of what existed. It carries whatever was on the volume, including the things nobody remembered were there, because copying does not require anyone to have understood the contents. A definition is a description of what should exist. It carries exactly what somebody wrote down, and nothing at all beyond that, however many years the system has been running.

A golden image is a definition with a timestamp. It is the output of a build that ran once, from a specification, on a date, and it is frozen at that date in the same way a photograph is: accurate about the moment it captured and silent about everything since. Configuration code is a definition evaluated later, at convergence time, against whatever it finds. Together they cover more ground than either does alone, and they still share the same limitation. Both of them are statements about the system as defined. Production is the system as run.

While the two agree, the distinction is academic and nobody has any reason to raise it. The definition is the artefact everyone trusts, sitting in a repository with a commit history, reviewed at merge time, referenced in the recovery plan. It is also, in most estates, the artefact nobody exercises. It is read constantly and executed against a blank machine almost never, which is an unusual combination for something a recovery depends on.

The Kubernetes capture used in this part makes the shape of a definition concrete, because a manifest bundle is a definition with no ambiguity about its boundaries: what is in the YAML is in the definition and what is not, is not.

Configuration changeapplying the desired state — the part that lives in the repository— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 1 of the captured desired-state run; the four objects were applied from a single manifest bundle.

$ kubectl apply -f rbdr-shop.yaml

==================================================================
STEP 1  Apply the desired state - this is what lives in Git
==================================================================
namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0

Four objects, exit code 0. That set is what the definition contains, and it is also the whole of what the definition can ever produce. The capture then enumerates the three things that exist once the workload has been running — the YAML in Git, the cluster state in etcd as the API server holds it, and the application data in the PersistentVolume — and draws the consequence in a line worth carrying out of this lesson unchanged: Only the first is in your repository.

Four ways production stops matching its definition

Drift is not a sign of an undisciplined team. It is what happens to a system that is operated. Four mechanisms produce nearly all of it, and they differ in how visible they are, which is why estates that have eliminated the first two still have a gap.

The first is the emergency change. At three in the morning somebody edits a file on the host, restarts a service, and the incident ends. The follow-up commit is the last item on a list that stops being urgent the moment the graphs recover. Nobody decides not to commit it; it simply does not happen, and no artefact anywhere records the omission.

The second is packages moving forward in place. The image was built on a date and holds the versions available on that date. The running host has been patched since, by an operator or by an unattended upgrade mechanism, and its package set has moved. The definition typically says a package should be present rather than which version, so it continues to report agreement while the two diverge underneath. That is a defensible way to write the code and it still means the rebuilt host lands on whatever is current at rebuild time, which is not what production has been running.

The third is manual changes that were never emergencies. A kernel parameter raised during a load test and left. A firewall rule added for a vendor’s migration window. A mount point created for a one-off data import that turned out to be recurring. Each was reasonable, each was made by someone competent, and none of them was ever a change anyone thought needed a commit.

The fourth is the one that cannot be eliminated by discipline: configuration the software writes for itself. Certificate renewal replaces files on disk. Cluster daemons write membership and identity state. Databases maintain their own generated configuration alongside the one an operator edits. Package hooks generate caches and unit files. Nobody typed any of it, no repository holds it, and to a comparison tool it is indistinguishable from a manual edit. An estate can drive the first three sources close to zero with genuinely good process. The fourth remains, which is why the honest goal is a measured gap rather than an absent one.

The rebuild succeeds, and that is the problem

Drift is usually filed as a tidiness concern: things are inconsistent, someone should clean them up. That framing understates it, because drift changes what a rebuild produces. The rebuilt system is the system as defined. If production diverged from the definition, then rebuilding produces a system that has never actually run the workload — not a slightly stale copy of production, but a machine in a configuration that has no operational history at all.

The failure mode this creates is not an error. It is a success. The rebuild completes, the exit code is zero, the service starts, the health check passes, and the difference appears later — under load, at the next certificate rotation, when a batch job needs the tuned kernel parameter, when a peer expects a firewall rule that is not there. The same capture makes the point at the other end of a disaster: after the namespace was deleted, the identical definition was applied again.

Configuration changere-applying the same definition after the namespace was deleted— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 5 of the same capture, immediately after `kubectl delete namespace rbdr-shop` reported the namespace fully removed.

$ kubectl apply -f rbdr-shop.yaml

==================================================================
STEP 5  Recovery attempt 1: re-apply the Git desired state
==================================================================
This is what teams mean when they say "everything is in Git".
namespace/rbdr-shop created
persistentvolumeclaim/rbdr-orders created
configmap/rbdr-config created
pod/rbdr-orders-writer created
>>> exit code: 0

Compare the two captures line by line. They are the same four created lines and the same exit code: 0. Nothing in the second run’s output distinguishes a cluster that has been running the workload from one that was empty a moment ago, because the output is a report about the definition, not about the system. The same capture then looked at what the workload had accumulated.

Data-loss riskthe objects came back; what the running system had produced did not— EXECUTION VALIDATED on k3s v1.36.3+k3s1. The same step 5, immediately after the apply above reported every object created and the pod Ready.

$ kubectl exec rbdr-orders-writer -- sh -c 'ls -la /data; cat /data/orders.csv'

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Every declared object was present and the pod was 1/1 Running, so every signal derived from the definition read as a completed recovery. The capture draws the conclusion in its own words: the desired state restored perfectly, and the orders did not come back because they were never in the desired state. That is the reconstruction failure in miniature, with the divergence made total rather than partial.

Notice also when the divergence becomes visible. Drift is created during ordinary operations, when there is time to write it down, and it is discovered during recovery, when there is not — by people already handling an outage, on a rebuilt machine, against a production system that in the worst case is the thing they just lost. That inversion, cheap to record and expensive to discover, is the same structure the first lesson of this course described for backup and restore, arriving here in different vocabulary.

What a drift-detection run is actually iterating over

The reason a green drift report is compatible with a large gap is mechanical, and it comes down to what the loop iterates over.

A convergence tool run in check mode walks the definition. For each declared resource it queries the corresponding live state, compares the two, and reports a difference. Ansible’s --check predicts what each task would change without changing it, and --diff shows the textual difference for the files and templates it manages; both operate task by task, over the tasks that exist. Terraform’s plan refreshes the resources recorded in state, compares them against the configuration, and with -detailed-exitcode returns 0 for an empty diff, 2 for a non-empty one and 1 for an error.

Now consider a file, a package, a sysctl or a cloud resource that exists in production and is named nowhere in the definition. It is not part of the iteration. The tool does not miss it, fail to notice it, or decide it is unimportant — the tool was never asked about it. For Terraform the resource is absent from state, so there is nothing to refresh and nothing to compare, and the plan is empty with respect to it. For Ansible there is no task, so there is no comparison and no line in the recap.

That splits the estate into two sets. The declared set is what the definition names, and drift inside it is exactly what check mode is built to find. The undeclared set is everything else that is real. The reconstruction gap lives entirely in the second set, and no amount of running the first tool more often samples it, because sampling it would require iterating over the machine rather than over the definition.

Only one procedure iterates over the machine: building a system from the definition and comparing the result against production. The comparison is then between two running systems, both enumerable, and the difference between them is the gap. Everything else is a statement about the definition’s internal consistency.

Three detection methods and the boundary of each

Each method is worth running, and each is worth describing by what it cannot see, because a method used outside its boundary produces false confidence rather than no information.

Periodic re-convergence in check mode is the cheap one. It runs on a schedule, needs no additional infrastructure, and produces a per-host change count that trends usefully over weeks. Run it with diff output so the finding is legible rather than a number, and treat a non-zero result as a question about who changed what, not as a task to silence by converging for real. For a Terraform estate the same signal is available as an exit code, which makes it schedulable without parsing output:

PLAN=/var/lib/recon/drift.tfplan
terraform plan -detailed-exitcode -out="$PLAN"
RC=$?
case "$RC" in
  0) echo "no difference for any tracked resource" ;;
  2) echo "tracked resources differ - review $PLAN" ;;
  *) echo "plan did not complete, exit $RC" ;;
esac

Its boundary is the one the mechanism above describes: the declared set only.

Rebuilding staging from the definitions and diffing it against production is the expensive one and the only one that reaches the undeclared set. Build a host or a namespace from definitions alone, then compare the two machines along every axis you can enumerate cheaply — installed packages, listening sockets, unit files, mounts, kernel parameters, the contents of the configuration directories. Package sets are the usual starting point because they are trivial to enumerate and they catch the second drift source directly:

OUT=/srv/recon
REBUILT_HOST=recon-rebuild-01
PROD_HOST=web-prod-01
mkdir -p "$OUT"
for host in "$REBUILT_HOST" "$PROD_HOST"; do
  ssh "$host" 'dpkg-query -W -f "${binary:Package}\n"' | sort >"$OUT/$host.packages"
done
diff -u "$OUT/$REBUILT_HOST.packages" "$OUT/$PROD_HOST.packages" >"$OUT/package-gap.diff"

Read-only / Safethe package sets of a rebuilt host and the host it was rebuilt from— Illustrative output. No transcript in this course covers a paired rebuild, so this shows the shape of the comparison rather than a measured result.

$ diff -u /srv/recon/recon-rebuild-01.packages /srv/recon/web-prod-01.packages

--- /srv/recon/recon-rebuild-01.packages
+++ /srv/recon/web-prod-01.packages
@@ -1,4 +1,6 @@
nginx
openssl
+libjpeg-turbo8
python3
+redis-tools
rsync

Illustrative output

Each + line is a candidate member of the reconstruction gap: present in production, absent from what the definition builds. Some will turn out to be legitimate — a dependency pulled in by a package the definition does install — and the triage is the work. Its boundary is what you thought to enumerate, so the list of axes is itself a maintained artefact.

Treating any manual change as an incident is the process method, and it is the only one that closes the loop rather than measuring it. What makes it work is not a norm but a record: the follow-up commit gets an identifier, an owner and a close condition, tracked with the same machinery that tracks the incident itself. What makes it fail is leaving it as a cultural expectation, because a cultural expectation produces no artefact when it is not met, and an unmet expectation that leaves no artefact is indistinguishable from compliance.

The reconstruction gap is a dated list, not a claim

The measurement that matters here has a definition worth stating precisely. The reconstruction gap is the set of things present in production and absent from the definition. Not a percentage, not a maturity score — a set whose members have names, useful exactly because you can read it.

It is measured by rebuilding. Stand up the system from definitions alone, put the workload on it, and record every intervention needed to make it behave the way production behaves. Each intervention is one member of the set. Then each member gets one of two dispositions: it is committed to the definition, or it is recorded as a known exclusion with an owner and a reason. A member with neither disposition is the interesting case, because it is a thing your recovery depends on that nobody has agreed to own.

The number that gets reported is the count of members and the date of the rebuild that produced it. “Twelve items, measured 2026-08-14” is a fact. “Zero” by itself is not a measurement at all, and it is worth being blunt about why: an unrebuilt definition and an unmeasured gap are the same thing described from two directions. Nothing has been established either way.

Like a restore test, the measurement has a shelf life. The gap shrinks when items get committed and grows every week the estate is operated without one, which means the useful cadence is tied to the rate of change rather than to the calendar, and the reading from six months ago describes an estate that no longer exists.

Two sentences that a rebuild would have corrected

“Everything is in Git, so we can rebuild.” This is the anti-pattern the whole lesson is aimed at, and it is worth naming precisely what the claim assumes: that a repository which has never been used to build the running system will build it correctly the first time it is asked to, during an incident, under time pressure, possibly without the production system available for comparison. Nothing about a definition being reviewed, merged and stored supports that. The k3s capture is the miniature version — the same four created lines and the same exit code: 0, before the disaster and after it.

“Our nightly convergence is always green, so there is no drift.” The green report covers the declared set, which is the part the definition already knows about. It cannot report on state it was never told to look for, so it is silent about precisely the category that will surprise a rebuild. Read it as evidence that the declared set is converged, and read the absence of a recent rebuild as the absence of any evidence about the rest.

Production discipline

Rebuild from the definition on a schedule, not on demand. A definition that has produced a working system this quarter is a different artefact from one that has only been read. Put the rebuild in the calendar next to the restore test, and give it the same status: an exercise that produces evidence, not a task that succeeded.
Report the reconstruction gap as a dated list of named items. Twelve items with owners is a usable statement about recovery. Zero without a rebuild date behind it is an assertion, and it is the assertion this lesson exists to make uncomfortable.
Give every emergency change a tracked follow-up, not a good intention. The commit needs an identifier, an owner and a close condition recorded before the incident channel closes, because an expectation that leaves no artefact when it is unmet cannot be audited.
Pin what the rebuild must reproduce and name what it may not. Where the version matters, say the version; where it does not, say so explicitly. A definition that silently accepts whatever is current is not wrong, but it should be a decision that somebody made rather than a default nobody noticed.
Use check-mode convergence for the declared set and read its silence correctly. It is the right tool for finding edits to managed files and the wrong tool for finding anything the definition never mentioned. Treat a clean run as a statement about the declared set and nothing wider.

Cross-course references

Ansible for Production Sysadmins — Part XXXVI (Drift and Convergence) builds the check-mode and re-convergence practice that this lesson uses as its first detection method, and it is where the task-by-task mechanics live; what this lesson adds is the boundary of that practice, namely that iterating over the playbook can never enumerate the undeclared state a rebuild would miss.
Terraform for Production Sysadmins — Part XVII (Drift Detection and Reconciliation) covers state refresh, -detailed-exitcode and the reconciliation decision for resources that changed outside the workflow; the reconstruction gap defined here is the complementary quantity, made of the resources that never entered state at all and therefore never appear in a plan.
Docker & Containers for Production Sysadmins — Part IV (Images) explains how an image is built from a definition and why anything written into a running container after build is outside it, which is the same definition-versus-runtime split this lesson applies to hosts; a container estate makes the split unusually visible, because the rebuild happens on every deployment rather than once during a disaster.