Reading an estate: inventory and the first dependency map

The previous lesson listed what a backup system never promised to supply: the target hardware, the network, DNS, the identity provider, the certificate a client will check, the licence an appliance wants at boot. Every one is a dependency of something, and none appears in a backup schedule, because a schedule is a list of sources while a dependency is a relationship. This lesson builds the artefact that records relationships, because that artefact — not the schedule — is what a recovery order can be derived from. Most estates have never built one. They have an asset inventory, which is a different thing, and a monitoring target list, which is different again.

Services are what the business names; servers are an implementation detail

An estate can be enumerated from either end. Starting from machines produces a list of hosts, disks and owners, and it is usually the list that already exists, because it can be generated. Starting from services produces the names a customer, a contract or an executive would use during an outage: the ordering site, the billing run, payroll, the ticketing system. How many a given estate has is not worth guessing at. The distinction that matters is a different one: the service list has a completeness test and the machine list does not.

Ask the business whether anything they care about is missing from a list of services and the answer is usable, because the names are already theirs. Ask the same of a list of virtual machines and nobody can answer, because you cannot notice the absence of a host you never heard of. Recovery is negotiated in the units the business uses, so the map is built in those units.

Enumerate services first, then attach implementation to each. That direction catches the systems the operations team never thinks about because it does not run them — payroll, the e-signature provider, the status page itself. They hold state, and the same three questions apply. Leaving a SaaS tenant off because there is no server to back up is how an estate learns mid-incident that the export it needs is a manual download.

Where the state actually lives, written as an address

The second column is the one people believe they already have. Write it as an address rather than a category: not “a database” but orders-db.prod.internal:5432/orders, not “some files” but /var/lib/orders/uploads on app-01..03, not “object storage” but s3://acme-invoices-prod. A category cannot be verified, restored or handed over. An address can.

Two questions make the column honest. If this machine were deleted now and rebuilt from its image, what would be different? Everything that would be different is state, and the list runs longer than expected — the cron spool, the local ACME account key, the firewall configuration accumulated through a web UI over four years, the files that arrived through a form and were never called data. And which copies are written by something other than the service itself? A bucket the application and a nightly job both write to has two writers and two ways to be wrong.

A third distinction belongs in the same column, because a device frequently keeps two versions of itself. Many appliances hold a running configuration and a separately saved one, and only the saved copy survives a reboot, so a record taken from the running state describes a device that may never boot back into it. The same split appears in software as state that is not a file at all: the messages sitting in a queue, the position a consumer has reached in a stream, the jobs a scheduler believes it has already run. None of these is a path on a filesystem, which is why they are quietly left out of a column that expects one, and each of them is a way for a restored system to start cleanly and then behave wrongly — which is the failure that costs the most to diagnose, because nothing about it looks like a failed restore.

SERVICE="orders-api"
MAP_DIR="/srv/recovery-map/${SERVICE}"
mkdir -p "$MAP_DIR"

# Address, owning team, and everyone able to delete it.
printf '%s\n' \
  "postgres|orders-db.prod.internal:5432/orders|dba-team|dba-team, cloud-admin" \
  "objects|s3://acme-invoices-prod|orders-team|cloud-admin, lifecycle rule" \
  "volume|/var/lib/orders/uploads on app-01..03|orders-team|root on each host" \
  > "${MAP_DIR}/state.psv"

The format is unimportant. The fourth field is not: a row naming a location and an owner but no destroyer is half a row.

The column nobody fills in: what it needs in order to start

The third column records what must be true before the service can run at all, and it comes from six families. Name resolution: the service addresses its database, its queue and its identity provider by name, and a recovered host with no resolver logs connection errors instead of starting. Identity: an issuer or directory that must be up before any human or workload authenticates. Secrets: the credential, the store holding it, and the credential that reads the store. Certificates and trust: the server certificate, its key, and the authority clients trust. Network reachability: routes, firewall rules, tunnels, egress to whatever the software phones home to. Licensing: keys pinned to a hostname or a MAC address, served by a licence server that is itself a row on the list.

Write each one as an edge, not a tick box. A checklist says the service needs DNS. An edge says this service cannot start until that DNS service has started, and that DNS service is itself a row with its own state and dependencies. Only the second form composes, and only the second turns “what do we bring up first” from opinion into something readable off the drawing.

Two properties of the edge set do the work once it exists. The first is transitivity: a service’s real prerequisites are not the edges drawn from it but the transitive closure of those edges, so a service that appears to need only its identity provider also needs everything the identity provider needs, and the honest prerequisite list reaches further than the one anyone writes down unprompted. The second is that an edge is a testable statement rather than an assertion. The test is a cold room: if nothing else in the estate were running, would this start, and at what point would it stop? Working through that one service at a time is slow, which is the argument for building the map once and reusing it, but it is the only question that separates a dependency the software genuinely has from one an operator has assumed it has for years.

The map has three bands, and the interesting rows sit outside them

Three bands are enough: the services the business names, the state each owns, and what must already exist before any of them can start. Solid edges run from a service to its state and its dependencies; dotted edges record what a row would have to be reconstructed from.

flowchart LR
  subgraph SVC["Services the business names"]
    PORT["customer-portal"]
    ORD["orders-api"]
    BIL["billing-worker"]
  end

  subgraph STATE["Where the state lives"]
    PG[("orders-db.prod<br/>PostgreSQL")]
    OBJ[("s3://acme-invoices-prod<br/>object store")]
    VOL["/var/lib/orders/uploads<br/>on app-01..03"]
    FW["edge-fw-01<br/>running configuration"]
  end

  subgraph DEP["Must already exist to start"]
    DNS["internal DNS zone"]
    IDP["identity provider"]
    SEC["secret store"]
    CA["internal CA private key"]
  end

  subgraph ORPH["Recovery machinery, owner unnamed"]
    TFS["Terraform state"]
    CAT["backup catalogue"]
  end

  PORT --> ORD
  PORT --> IDP
  PORT --> CA
  ORD --> PG
  ORD --> OBJ
  ORD --> VOL
  ORD --> DNS
  ORD --> SEC
  BIL --> PG
  IDP --> SEC
  SEC --> DNS
  DNS -->|zone deployed with a credential from| SEC
  PG -.->|rebuilt from| TFS
  VOL -.->|restored via| CAT
  FW -.->|restored via| CAT

Two features matter more than the drawing itself. The pair of edges between the secret store and DNS forms a cycle. And the fourth band holds rows that no solid edge points at: nothing needs them in order to serve a request, and every edge that reaches them is dotted.

The rows that turn out to have no owner

Four items appear, estate after estate, with an address, no owner and no backup job. They are the machinery of recovery itself, or they belong to devices rather than servers, which is why assigning owners to servers never reaches them. Two of the four sit in the diagram’s fourth band, unreferenced by any solid edge. The other two hide among the ordinary rows — one filed as state, one filed as a dependency — which is why an unowned row is not something a reader can spot by looking at the shape of the drawing.

The running configuration of an appliance — firewall, load balancer, switch, storage array — is years of small changes made through a web UI. It is in no repository, because it was never code, and in no backup job, because the device runs no agent.

The Terraform state maps declared configuration to the real resource identifiers it created. Without it a rebuild is not a replay of the configuration but an import exercise performed against an account that is on fire, and it is commonly stored in a bucket inside the account being rebuilt.

The private key of the internal certificate authority signs every certificate the estate trusts internally. If it is gone, recovery needs a new root distributed into every trust store on every host and in every image, while the estate is down and the tooling that distributes things is itself waiting on certificates.

The backup catalogue is the index recording which snapshot holds which file and where the bytes are. Where a product keeps that index only on the backup server, losing the server turns a shelf of tapes into data with no map — the row found latest, because the data really is all there.

Ownership is what gets a row tested. The useful question is not who owns the firewall, but who restores edge-fw-01’s configuration, from what, onto what, and when they last did it. The first phrasing gets a name; the second gets a procedure or an honest gap.

Recording who can destroy each copy

The fourth column asks, for each copy, which identities are able to delete it. Enumerate people, roles, workload identities and automation together, because an automated destroyer belongs in the column as squarely as a person does: a bucket lifecycle rule that expires objects after a set number of days, an apply against the wrong workspace, and the backup product’s own retention policy all delete copies, and they do it on a schedule rather than on a bad day.

Then run the collapse test. Take the destroyer sets of every copy of a given service’s state and intersect them. If the intersection is non-empty — one credential, one role, one pipeline reaching all of them — the number of copies that survive that credential being misused is zero, whatever the drawing shows. Three copies reachable by a single cloud administrator role is one copy with extra storage cost.

Two refinements stop the test from flattering the estate. Destruction is not only deletion: an identity that can overwrite a copy in place, shorten the retention that governs it, or re-encrypt it under a key that identity controls has made it unusable just as thoroughly, so the column asks who can render a copy unrecoverable rather than who holds a permission named delete. And the identities have a closure of their own. A role that is allowed to assume a second role inherits everything the second one reaches, which means intersecting only the principals named literally in each policy understates the overlap; the set that matters is the set that can reach the copy by any path, and it includes the pipeline credential that is allowed to edit the policy.

The output is not a document to file. It is a graph with four columns, and it is the input to everything the rest of this course does: objectives negotiated per service, the ordering of a failover, the scope of a restore test, and which copies sit outside the reach of the identity that runs production.

Production discipline

1. Enumerate services before servers. Start from the names the business would use during an outage, then attach machines, buckets and appliances to each. A service list can be checked for completeness by the people who own the outcome; a machine list cannot.
2. Write every location as an address. s3://acme-invoices-prod can be verified, restored and handed over; “object storage” cannot, and a category in the state column is how an estate finds a bucket nobody backed up.
3. Record dependencies as edges, not tick boxes. Only edges compose into an order, and only edges expose the cycles that have to be broken before an incident rather than during one.
4. Give every row an owner, including rows that are not servers. The appliance configuration, the Terraform state, the CA private key and the backup catalogue reliably have none, and an unowned row never acquires a tested restore procedure.
5. Record who can destroy each copy, then intersect the sets. If one identity reaches every copy of a service’s state, the estate has one copy. That number, not the count of backup jobs, survives a compromised credential.

Cross-course references

• Observability for Production Sysadmins — Part VIII (Service Discovery) describes how a monitoring system assembles its target list from labels and platform APIs. That list is the artefact most often mistaken for the inventory built here, and this lesson is the argument for why it cannot substitute: service discovery enumerates what is running and answering, never where a service keeps its state or who can delete it.
• Terraform for Production Sysadmins — Part XII (State Recovery and Backup) treats one of the four unowned rows this map exposes at course length, and matters here because the state file makes a rebuild a replay rather than an import exercise.
• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XVII (Inventory, Discovery and Monitoring) builds the same enumeration for keys and certificates, and joins this lesson at the internal CA private key: the dependency band records that services will not start without a trusted chain, and that part records what must be inventoried to rebuild one.

Quiz