Documentation that stays true and can be found

Operational documentation has two failure modes and neither one announces itself. The document drifts out of agreement with the system, or the reader never finds it. In both cases the document still exists, the wiki still shows it, and the metric that says “we have runbooks for 82% of our services” still reads 82%.

Both failures are mechanical, and both have mechanical fixes. That matters because the usual proposed fix - a quarterly review - is a calendar entry that competes with production work and loses.

Documentation nobody can find does not exist

The responder is not browsing a documentation site. They are looking at an alert, a dashboard, a failing unit, or a config file, at 03:00, and they will spend perhaps ninety seconds looking before they start improvising.

So the question is never “where should this be filed”. It is “where will the reader’s eyes already be”, and the answer is almost always one of four places.

The alert. Every alert should carry the link to the runbook for that alert. This is the single highest-leverage change in this lesson, and most alerting systems have a field for it:

- alert: MyAppHealthCheckFailing
  expr: probe_success{job="myapp-health"} == 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: 'myapp health check failing on {{ $labels.instance }}'
    runbook_url: 'https://docs.example.com/runbooks/myapp-health-failing'

An alert without a runbook_url is asking the responder to guess the document’s title. An alert whose runbook_url points at a page that does not exist is worse, and is worth checking automatically - a link checker over the alert definitions catches it before an incident does.

The config file. A comment at the top of a file that a human will edit under pressure:

# Managed by Ansible: roles/myapp/templates/myapp.conf.j2
# Local edits are overwritten on the next run.
# Change procedure: https://docs.example.com/runbooks/myapp-config-change

The login banner. /etc/motd is read by everyone who logs into the host, which during an incident is exactly the population you want to reach:

cat /etc/motd

Keep it to the essentials - what this host is, who owns it, and where its runbooks live. A motd that has grown to thirty lines is skipped as thoroughly as no motd at all.

The search box. People search for the string they are looking at, which is the error message. If the runbook for a filesystem-full incident does not contain the literal text No space left on device, it will not be found by the person reading that error. Put the actual error strings in the document, in a code block, spelled exactly as the system emits them.

Drift is detectable

Documentation drifts because the system changes and nothing connects the change to the document. Three mechanisms close that gap, in increasing order of effectiveness.

Keep the document next to the thing

When the runbook lives in the repository that holds the service, a change to the service and a change to its documentation are the same pull request, reviewed by the same person, at the moment the author still has the context.

A CODEOWNERS entry makes it structural rather than cultural: a change under roles/myapp/ requires review from the team that owns docs/myapp/, so the two cannot silently diverge.

Make the document executable

The strongest version: the assertions in the document are run by something.

This course does exactly that to itself. Every command in a lesson lives in a shell block that a checker parses, every cross-reference resolves to a file that exists, and both are enforced rather than reviewed:

$ node scripts/check-frontmatter-refs.mjs linux && node scripts/check-shell-blocks.mjs linux

check-frontmatter-refs: linux - 0 dangling prerequisites, 0 has_quiz mismatches
check-shell-blocks: linux - 412 shell blocks parsed, 0 failures

The same idea applies to operational documentation. A verification step in a runbook - systemctl is-active myapp, curl against a health endpoint - can be run nightly against a staging environment. When the command stops working because a unit was renamed, the check fails and someone fixes the runbook, months before an incident would have found it.

You do not need to automate a whole runbook. Automating the pre-checks and the verification steps is most of the value, because those are the parts made of commands with deterministic results.

Date it honestly

Where automation is not practical, a last_verified date is a weak signal used well. It does not make the document correct; it tells the reader how much to trust it, which is information they otherwise have to guess.

grep -h '^last_verified:' src/content/courses/linux/lessons/*.mdx \
  | sort | uniq -c | sort -k2

Sort the estate by that date and the oldest entries are your review queue - a queue derived from data rather than a calendar reminder that competes with production work.

Deleting is a maintenance action

A document that is wrong is worse than no document. No document sends the reader to first principles, which is slow and correct. A wrong document sends them confidently in the wrong direction, and they will trust it for longer than they should because it looks maintained.

So deletion is a legitimate and under-used response:

• A runbook for a service that was decommissioned: delete it. It will otherwise be found by someone searching for a similar-sounding service.
• A procedure superseded by automation: replace the body with a pointer to the automation and one paragraph on what to do when the automation fails.
• A document nobody can vouch for: mark it clearly. A banner reading “Unverified since 2024-03; treat as historical context, not as a procedure” is honest and takes a minute.

Knowing whether it worked

Two signals, and both come from the incident records described in the previous lesson:

Was the runbook opened? If a runbook existed for the failure and the responder did not use it, that is a defect regardless of the runbook’s quality - it was either not findable or not trusted. Both are fixable, and neither is fixed by writing more content.

Where did the responder stop following it? The step where someone left the procedure and started improvising is the step that was wrong, missing an expected result, or assumed knowledge they did not have. Add a line to the incident record: “followed the runbook to step 6, then diverged because the output did not match”. That single sentence is the most specific documentation feedback anyone will ever give you.

Ask both questions in every post-incident review. They take thirty seconds and they produce a defect list drawn from real use, which is worth more than any review cycle.

Knowledge check