Documentation and runbook repositories — keeping docs in version control alongside code

Documentation and runbooks are code: they change, they need review, they need history, and they need to be auditable. A runbook that lives in a wiki outside version control is one that does not exist when the wiki is down, cannot be reviewed through the team’s normal review surface, and has no commit history tying it to the change it describes. The fix is to put the runbook in a version-controlled repository and tie it to the change it describes through commit hashes.

The two shapes for documentation

flowchart TB
    subgraph COLOC["Co-located: docs live with the code"]
        C1[infra-app/README.md]
        C2[infra-app/runbooks/]
    end
    subgraph CENTRAL["Centralised: separate docs repository"]
        D1[infra-docs/runbooks/]
        D2[infra-docs/architecture/]
    end
    COLOC -. cross-link .-> CENTRAL

A co-located documentation set lives next to the code it describes. A centralised documentation set lives in a separate infra-docs repository that the whole organisation reads from.

The co-located shape is the easier one to keep current: a change to the VPC module is a change to the module’s README, and the two are reviewed in the same pull request. The centralised shape is the easier one to discover.

The right answer is usually a hybrid: co-locate the documentation that is about a piece of code with the code, and centralise the documentation that is about the organisation in a separate infra-docs repository.

The runbook pattern

A runbook is a step-by-step procedure for responding to an alert. The minimum useful runbook has four sections:

• Alert or trigger. What fired, and where? Include the alert name, the service name, and the dashboard link.
• Diagnosis. What to check first, what to check second.
• Mitigation. The first action to take, with the exact command and the expected output.
• Recovery. The follow-up actions, with the exact command and the expected output.

A runbook without all four sections is one that will not be followed at 03:00 during an incident.

The cross-reference pattern ties the runbook to the code:

## References
- Commit: <commit-hash>
- Service: <service-name>
- Dashboard: <dashboard-url>
- On-call: <rotation-url>

The commit hash is the link to the code. git log -S "<alert-name>" -- runbooks/ finds the runbook’s history.

The architecture document pattern

An architecture document is the high-level description of a service: what it does, what it depends on, and what the SLOs are. The minimum useful architecture document has a diagram (Mermaid, in plain text) showing the components, a list of dependencies, a list of SLOs, a list of known issues, and a list of owners.

The architecture document is what a new engineer reads on their first day.

Cross-references and the commit anchor

The cross-reference pattern is the most important part of the documentation repository. The mechanism is a footer in every runbook that names the commit hash, the service name, the alert name, the dashboard URL, and the on-call rotation URL.

When the runbook needs to be updated, the update is a pull request against the documentation repository.

When to co-locate, when to centralise

The rule of thumb:

• Co-locate the documentation that is about a piece of code - the module’s README, the runbook for a specific alert, the architecture diagram.
• Centralise the documentation that is about the organisation - the onboarding guide, the security policy, the change calendar, the SLO catalog.
• Cross-link between the two. The module’s README says “see the security policy in infra-docs/security/”.

Production discipline

1. Co-locate the README with the code it describes. A module without a README is one that nobody can use.
2. Centralise the organisation-level documentation in a separate infra-docs repository.
3. Every runbook has a footer with a commit hash, an alert name, a dashboard URL, and an on-call rotation URL.
4. Update the runbook in the same pull request as the code change that affects the runbook. A code change that changes an alert’s name without updating the runbook is a bug.
5. Use git log -S to find the runbook’s history.

Cross-course references

• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the role-based layout where a role’s README is co-located with the role.
• Terraform for Production Sysadmins - Part IX (State) covers the per-environment state files that the architecture document should reference.
• Linux for Production Sysadmins - Part XXXIV covers the runbook pattern for firewall changes.

Quiz