Classifying data by what it costs to lose · Backup & DR

A failure model tells you what can go wrong; it does not tell you which of the things that go wrong actually matter. Point the same failure — a host gone, a bucket emptied, a directory removed by a command that ran in the wrong shell — at different parts of an estate and the outcomes are not comparable.

Some of what disappeared comes back with a package install. Some comes back by running a pipeline. Some does not come back by any means, because nothing else in the world ever held it. Treating those three identically is expensive in both directions at once: it buys premium storage for material a build server would happily regenerate, and it leaves the material no build server can regenerate on the same nightly schedule as a compiler cache.

Rebuild, re-derive, or unable to continue

One question sorts an estate, and it is worth asking in exactly this form: if this were gone right now, and no copy existed anywhere we control, what would we actually do? There are only three honest answers.

The first is we would rebuild it. The item came from somewhere outside the estate that still has it — a distribution mirror, a container registry, a vendor download page. Operating system packages, base images, runtime interpreters and third-party binaries live here. Nothing about these bytes is ours; we fetched them, and we can fetch them again.

The second is we would re-derive it. The item was produced by something we still hold, from inputs we still hold: compiled artefacts, images built from our own Dockerfiles, generated configuration, search indexes, materialised views, thumbnail caches. The value is in the recipe and the inputs, and the output is a performance optimisation that happens to occupy disk.

The third is we could not continue. There is no upstream and there is no recipe. The bytes are the only surviving record of something that happened in the world: an order that was placed, a payment that settled, a message a customer sent, a signature that was produced, a key that was generated once from randomness that no longer exists. Nothing regenerates them, because nothing else ever knew them.

Notice what the question asks about. Not what kind of file this is — what the estate around it looks like. The same PostgreSQL data directory is irreplaceable on the primary and re-derivable on a reporting replica rebuilt from it nightly. The same container image is reconstructable while its build definition and base image remain available, and irreplaceable the moment that base image tag is overwritten upstream. Classification is contextual, which is exactly why it has to be written down rather than assumed.

Data with no upstream

Work through the third category first, because it is the smaller list and the one that justifies the whole budget.

Business records and transactional history. Orders, invoices, ledger entries, contracts, the audit trail. These are the reason the organisation is subject to retention obligations at all, and they are the class where the cost of loss is not measured in engineering hours but in whether the business can answer questions it is legally required to answer.

Customer-supplied content. Anything a person outside the organisation typed or uploaded. Worth naming separately from business records, because it tends to live somewhere unglamorous — a bucket wired up in an afternoon, a volume mounted into one container — and because the organisation that loses it has to tell the people who created it.

Accepted database writes. A database is two classes glued together and it is a mistake to protect it as one thing. The schema, roles, extensions and tuning are configuration: they live in migration files in a repository and are re-derivable. The rows are not. Every write the database acknowledged is a promise that the data survived, and the gap between what it acknowledged and what the last backup holds is the only number that matters here.

Key material. Encryption keys, repository passwords, CA private keys, SSH host keys, signing keys, the credentials that unlock a secrets manager. This class is dangerous out of all proportion to its size. Key management guidance treats the recovery of archived keys as a function to be designed deliberately, with procedures of its own, rather than as a by-product of copying servers around. What earns it that separate treatment is the asymmetry: a few kilobytes of key decide whether terabytes of ciphertext are data or noise. A repository encrypted under a key nobody can produce is operationally identical to one that was never taken — the storage bill is the only difference.

Two ways a correctly-running backup produces nothing recoverable

“Store the key on the backed-up server.” The repository password lives in a file on the machine whose filesystem is being backed up, so it is inside every snapshot. The machine is destroyed. The key is now available exclusively inside the thing it is required to open, which is the same as not existing. This is not exotic; it is the default outcome of configuring encrypted backups without ever asking where the key goes.

“The image is in the registry, so we don’t need to back it up.” True while the registry exists, while the tag still points at the digest it pointed at last month, and while the base image it was built FROM is still published. A reconstructability claim is a claim about other people’s systems continuing to behave, and it can become false without anyone touching your estate.

“Reconstructable” is a claim about a recipe, not about a file

The second category is where most of the volume lives and where most of the savings are, but the word carries three conditions that are easy to state and easy to forget.

The recipe must exist and be protected at least as well as the thing it produces. A compiled artefact is reconstructable from source; if the source repository is the only copy and it is unprotected, the artefact was never reconstructable, merely undamaged so far. This is the commonest classification error after key material: an estate correctly decides it need not back up /var/lib/docker, then discovers nobody backs up the repository holding the build definitions either, because “it’s in Git” was treated as a property rather than as a location.

The inputs must still be reachable. Re-derivation depends on upstreams you do not control: mirrors that retire old versions, registries that garbage-collect untagged digests, dependencies whose maintainers deleted the project, vendor downloads behind a lapsed support contract. A build that reproduced perfectly a year ago is not evidence that it reproduces today.

The re-derivation must fit the recovery time you promised. An index that takes longer to rebuild than the recovery time objective agreed for its service is, for recovery purposes, not reconstructable at all — it is slow-to-restore data that happens to have a recipe. Protect it like data you cannot re-derive, not because the recipe stopped working, but because the clock does not care that a recipe exists.

Classification is a property of the dependency graph, not of the file

The reason classification resists a simple checklist is that it is not a per-file attribute at all. It is the answer to a graph traversal.

Model every item you hold as a node, and draw an edge from each item to what could regenerate it: from a compiled binary to its source and toolchain; from a container image to its build definition and base image; from /etc to the configuration repository and the runner that applies it; from a search index to the database it was built from. Now walk outward from any node.

If every path leaves your estate and terminates at an external source that still exists and is still reachable, the item is rebuildable and needs no protection of its own.

If a path terminates at inputs you hold, the item is re-derivable — and here is the consequence that matters: those inputs inherit the class of everything downstream of them. The class of an item is the class of the weakest node on its reconstruction path. A build cache protected like a ledger, sitting above a source repository protected not at all, has achieved nothing: the traversal ends at an unprotected node, so the whole path is unprotected.

If no edge leaves the node at all, the item is irreplaceable. Not because it is important — importance is a separate axis — but because the graph offers no route back.

Two properties follow. Classification is transitive, which is why protecting outputs while ignoring recipes is a null operation. And it is perishable: the graph changes when a mirror retires, when a vendor stops publishing, when a build depends on a service that is later decommissioned, when someone hand-edits a file that used to be generated. An item’s class is a dated assertion about the estate, not a permanent label on a file.

Configuration inherits whatever protects its source of truth

Configuration is the class most estates get almost right, which is more dangerous than getting it obviously wrong.

The reasoning is sound: if everything under /etc that matters is generated from a role, module or manifest in a repository, and the repository is protected, and the system that applies it can be stood up again, then configuration is re-derivable and needs no backup line of its own. Every one of those clauses is a condition, and estates fail on them in predictable places.

The first is drift. Declared and running configuration agree only until someone fixes something at three in the morning, and an emergency edit never pushed back into the repository is, from the moment it is made, irreplaceable data living in a directory everyone has classified as reconstructable. The second is the material that was never declarative at all: machine identity such as SSH host keys and cluster join tokens, node UUIDs other systems have recorded, values a vendor’s web interface wrote into its own database, DHCP lease state, the internal database of the configuration tool itself. The third is the pipeline — configuration applied by a CI system is reconstructable only as far as that system, its runners, its credentials and its pipeline definitions are themselves recoverable.

The material that looks disposable and is not

There is a recognisable family of items that are small, are not “data” in the sense the business uses the word, have no owning team, and end recoveries.

Infrastructure state. Terraform’s state file records the mapping between the configuration you wrote and the real resources that exist — information held nowhere else, since the provider knows the resources, the repository knows the configuration, and only state knows which is which. Losing it does not lose infrastructure; it loses the ability to manage infrastructure, and the next plan proposes to create everything that already exists. Rebuilding the mapping means importing every surviving resource back into state by hand, under time pressure, and state is also a place sensitive values come to rest, which puts it in two classes at once.

The backup catalogue. The index recording which snapshot holds which file version is not the data, and it is routinely excluded from protection on exactly that reasoning. Some repository formats can rebuild an index by re-reading the repository and some cannot; where they can, that rebuild costs a full pass over the repository — a cost you are asked to pay at the moment you are trying to restore.

Monitoring configuration. Alert rules, recording rules, dashboards, silence and SLO definitions. Lose these and the estate comes back with nobody able to tell whether it came back correctly, which is how a partially-successful restore gets declared complete. Note the split inside this class: the configuration is reconstructable from a repository, while the historical metrics are irreplaceable, because they record what actually happened.

Licences and entitlements. Offline activation files, licence keys, vendor portal credentials, the support contract identifier. Nothing in the estate regenerates them, and replacing them runs at the vendor’s pace on the vendor’s ticket queue rather than at the pace of a restore.

What classification decides

Classification is not documentation for its own sake. Four operational decisions fall directly out of it, and none of them can be made without it.

Retention follows the reason the data exists. Irreplaceable records are kept as long as an obligation or an investigation might require. Anything reconstructable is kept only while it makes a rebuild faster than a restore; past that point, keeping it is pure cost.

RPO is only meaningful for one of the classes. Asking how many minutes of a build cache you can afford to lose is a category error: you do not restore it to a point in time, you re-derive it to current. The question of how much accepted work may be lost applies to irreplaceable data and to nothing else, and confining it there is what makes the number arguable with a business owner rather than negotiated with a tool.

Storage tier and copy count follow from cost of loss divided by size. Key material is small and catastrophic, arguing for the most copies, the most expensive media, and at least one copy that production cannot alter — either genuinely offline, or held under a retention mode that refuses deletion for a defined window. Bulk reconstructable data is large and cheap to replace, arguing for the cheapest tier or none at all.

Restore order is the output nobody expects. The same traversal that produced the classification produces the sequence: keys before the repositories they open, the catalogue before the data it indexes, state before the infrastructure it manages, identity and DNS before the applications that authenticate against them. An estate that has not classified its data cannot state its restore order, and will discover it live.

A register makes all four decisions inspectable, and can be checked mechanically for the error that matters — an item claiming a class its dependency graph does not support.

# One row per protected item: item, class, and the route back if it is lost.
REGISTER=/srv/recovery/data-classes.tsv
while IFS=$'\t' read -r item class route; do
  case "$class" in
    irreplaceable)
      [ -n "$route" ] && printf 'review: %s is keyed irreplaceable but names a route\n' "$item"
      ;;
    reconstructable)
      [ -z "$route" ] && printf 'misclassified: %s has no reconstruction route\n' "$item"
      ;;
  esac
done < "$REGISTER"

Production discipline

Classify by reconstruction route, not by file type or by directory. The question is what you would do if the item were gone, and the answer depends on the estate around it, not on the extension. The same data directory can be irreplaceable on one host and re-derivable on another.
Protect every recipe at the class of the thing it produces. Source repositories, build definitions, migration files and configuration repositories inherit the class of their outputs. “It’s in Git” names a location, not a protection level, and the traversal stops at the weakest node on the path.
Give key material its own recovery path, outside everything it protects. A key that exists only inside the system it decrypts has already failed. The test is whether you can produce the key while assuming the estate is gone, and it is worth rehearsing separately from any restore.
Register the small things nobody owns, with a named owner each. Infrastructure state, the backup catalogue, monitoring configuration, licence files, machine identity, DNS zone data. These convert a recoverable incident into an unrecoverable one, and they are unowned by default.
Date every classification and re-check it when the graph moves. A retired mirror, an overwritten tag, a decommissioned build service or a hand-edited config file all silently demote an item from reconstructable to irreplaceable without changing a single line of your backup policy.

Cross-course references

Terraform for Production Sysadmins — Part XII (State Recovery and Backup) treats the state file as an object with its own recovery procedure, which is the concrete instance of the category this lesson calls “looks disposable and is not”: the mapping between declared configuration and real resources exists nowhere else in the estate, so it is irreplaceable by the test applied here even though it contains no business data.
Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) covers where key material is allowed to live and how it is recovered, which is the mechanism behind the rule stated above that key material needs a recovery path outside everything it protects; this lesson establishes why keys are their own class, that course establishes how to hold them.
Git, CI/CD & GitOps for Infrastructure Engineers — Part XCVIII (Git Hosting Failure) examines what happens when the repository that everything is reconstructed from is itself unavailable, which is the exact condition under which this lesson’s “configuration is reconstructable from code” claim stops being true and the whole reconstructable class silently becomes irreplaceable.