Inventories, maintenance procedures and escalation paths

Three documents in this part are not procedures. Nobody executes them step by step, so they are written once and then rot quietly, and each one turns into an incident in its own characteristic way.

The inventory rots into a host nobody knew was running. The maintenance record rots into a certificate that expires. The escalation path rots into a vendor support account with no active entitlement, discovered at 02:40 on a Sunday.

The inventory: two lists, and the gap between them

There is a mechanical part to inventory - collecting facts, naming conventions, dynamic sources - covered in linux-naming-and-inventory and linux-inventory-and-facts. This lesson is about the part that makes it useful, which is that you always have two inventories and the interesting information is in their difference.

The declared inventory is what should exist: the CMDB, the Ansible inventory, the Terraform state, the spreadsheet. It is authoritative about intent.

The discovered inventory is what actually responds: DNS records that resolve, hosts that answer on the management network, machines reporting into monitoring or configuration management in the last 24 hours. It is authoritative about reality.

Reconciling them takes one command and produces two lists, both of which are findings:

sort -u /srv/inventory/declared-hosts.txt > /tmp/declared
sort -u /srv/inventory/seen-last-24h.txt  > /tmp/discovered
comm -23 /tmp/declared /tmp/discovered    # declared, not seen
comm -13 /tmp/declared /tmp/discovered    # seen, not declared

Declared but not seen is a host that is down and unnoticed, or one that was decommissioned without the record being updated. The first is an outage nobody has reported; the second is why your capacity numbers are wrong.

Seen but not declared is worse. It is a host running on your network that no process created and no process patches: a machine built for a migration that was never turned off, a test box someone stood up with a copy of production data, or something that is not yours at all. Nothing else in your tooling will find it, because every other tool starts from the declared list.

Run the reconciliation on a schedule and treat a non-empty result as work, not as a report.

What a host record needs

Most inventories record the technical facts, which configuration management can rediscover anyway. The fields that matter are the ones no tool can derive:

The decommission gap

A host is rarely “removed”. It is powered off, and then it persists in:

• DNS, including reverse records
• the monitoring system, where it now alerts as down forever or was silenced and forgotten
• firewall rules and security-group memberships
• authorized_keys and credential grants on other hosts
• backup job configuration, failing nightly
• load balancer pools, health-checking a dead address
• the inventory itself

Each of these is small. Collectively they are why nobody can tell which alerts matter. Decommissioning deserves a checklist as much as building does, and the checklist should end with the inventory record being marked retired rather than deleted - a deleted record cannot answer the question “what was 192.0.2.47 six months ago?”, which is the exact question a forensic timeline asks.

Maintenance procedures

Maintenance is the work that has no ticket because nothing is broken: patch cycles, certificate renewals, backup restore tests, log retention review, capacity review, credential rotation, cluster failover exercises.

Documenting it is necessary and not sufficient. The property that decides whether maintenance happens is whether it is scheduled - attached to a date and an owner - rather than merely described.

# Anything already running on a timer is maintenance that will happen
systemctl list-timers --all --no-pager

For each recurring item, record four things:

MAINTENANCE: <name>
=========================
Cadence     : <how often, and any window constraint>
Owner       : <rota>
Trigger     : <timer, calendar entry, or ticket automation>
Procedure   : <link to the runbook that performs it>
Evidence    : <what proves it was done - a log, a ticket, a report>

The Trigger line is what separates a maintenance programme from a wish list. If the only trigger is that someone remembers, the item will be skipped during the first busy month and never resume.

The Evidence line is what makes it auditable, and it answers a question that comes up in every post-incident review: was the maintenance actually performed? “It is in the schedule” is not an answer. A ticket, a timer’s journalctl record, or a dated report is.

Escalation paths

Escalation documentation is read exactly once per incident, under maximum time pressure, by someone who needs a phone number rather than an org chart.

What it must contain, per path:

• The trigger. The condition under which you escalate, stated as a criterion rather than a feeling: “the service has been down for 20 minutes and the cause is not identified”, “the failure involves data loss”, “a security control has been bypassed”.
• The destination. A rota name and the exact route - which paging tool, which channel, which phone number. Not a team page on the wiki.
• What to have ready. The three or four facts the receiving team will ask for first. This turns a five-minute handover into a one-minute one, and it is the part responders most appreciate.
• The out-of-hours difference. The route at 15:00 and the route at 03:00 are rarely the same, and the document should say both.

Vendor escalation needs more, because it involves a contract:

VENDOR: <name>
====================
Support portal   : <url>
Account / site ID: <id>
Contract number  : <id>, expires <date>
Entitlement      : <24x7 or business hours; response time per severity>
Severity mapping : <what the vendor calls a P1, in their words>
Who can open a case: <named accounts, or the shared credential path>
Phone (24x7)     : <number>
Last tested      : <date and by whom>

Knowledge check