Change documentation and tickets — the audit trail that survives the operator

A change without documentation is a rumour. The next operator who looks at the rule, the alias, or the NAT entry has to guess why it exists. The guess is usually wrong; the wrong guess is the cause of the next incident. Documentation is what turns a change from “someone did this” into “this was done, for this reason, with this evidence, with this rollback”.

This lesson covers the change ticket as the unit of change documentation: the required fields, the peer-review sign-off, the before/after evidence, and the failure modes of undocumented changes.

What a change ticket is

A change ticket is a single record of a single change. One ticket per change. The ticket is created before the change is applied, updated during the change, and closed after the change is verified. The ticket is the audit trail: who did what, when, why, with what evidence, with what rollback.

The ticket is not the operator’s memory. The ticket is not a Slack thread. The ticket is not an email that scrolls away. The ticket lives in the ticketing system the organisation uses — Jira, GitHub, ServiceNow, GitLab — and persists beyond the operator’s tenure.

$ configctl config backup && cp /conf/backup/config-*.xml ~/backups/ && configctl filter show | head -20 > ~/backups/before-rules.txt

Writing backup to /conf/backup/...
drwxr-xr-x  2 root  wheel    512 Aug 14 02:14 backups
-rw-r--r--  1 root  wheel    892 Aug 14 02:14 before-rules.txt
@0 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = https
@1 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = dns
@2 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = ssh

Required fields

The change ticket has seven required fields. Each field is filled in before the change is applied. The operator who skips a field is the operator who, three months later, cannot answer “why does this rule exist?”.

• Identifier. A unique ticket id. CHG-2026-0814-0042, or whatever the ticketing system issues.
• Summary. A one-sentence statement of the change. “Add rule to LAN to block egress to 198.51.100.50:443.”
• Rationale. The reason. The incident, the request, the compliance requirement. “INC-2026-0814-0042: 198.51.100.50 was compromised; block all egress to that destination.”
• Expected effect. The traffic pattern after the change. “Hosts on 192.0.2.0/24 cannot reach 198.51.100.50:443. Other destinations unaffected.”
• Worst-case effect. The most damaging failure mode. “Source mistyped as 0.0.0.0/0; all LAN egress blocked; GUI unreachable. Recovery: console restore from CHG-2026-0814-0042-backup.”
• Rollback. The procedure to reverse the change. “Remove the rule; reload the ruleset; verify egress to 198.51.100.50:443 resumes.”
• Evidence. Before-evidence and after-evidence. The backup, the ruleset snapshot, the log lines, the state table entries.

The fields are filled in this order. The order matters because the operator who fills in “expected effect” first might not name a worst case. The operator who fills in “worst-case effect” before “expected effect” is forced to think about failure before thinking about success.

Peer-review sign-off

The peer-review sign-off is a field in the ticket. The sign-off records:

• The name of the peer reviewer.
• The timestamp the peer reviewed.
• The outcome of the review: approved, approved with conditions, sent back for revision.

The peer reviewer reads the ticket before the change is applied. The reviewer reads the change summary, the expected effect, the worst-case effect, and the rollback. The reviewer decides whether the procedure is sufficient for the worst case. If the procedure is not sufficient, the reviewer sends the change back.

A self-review is weaker than a peer review. The self-review is recorded as “self-reviewed” with the operator’s name; the auditor who reads the ticket later can see that the change was not independently reviewed.

$ configctl filter show | grep -E '198.51.100.50|192.0.2.0/24' > ~/backups/after-rules.txt && diff ~/backups/before-rules.txt ~/backups/after-rules.txt

--- before-rules.txt
+++ after-rules.txt
@@ -3,3 +3,4 @@
@0 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = https
@1 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = dns
@2 pass in quick on igb0 inet proto tcp from 192.0.2.0/24 to any port = ssh
+@3 block return in quick on igb0 inet proto tcp from 192.0.2.0/24 to 198.51.100.50 port = https

Before/after evidence

Evidence is what makes the change auditable. A change ticket without evidence is a change ticket the auditor cannot verify. The evidence falls into three categories:

1. Configuration evidence. The before and after configuration backups. The before and after compiled ruleset. The before and after pfctl -s rules output. The before and after pfctl -s nat output (if NAT changed).
2. State evidence. For stateful changes, the before and after state-table size. For alias changes, the before and after alias membership. For NAT changes, the before and after translation table.
3. Behaviour evidence. The log lines that show the new rule matching. The packet captures that show the new flow. The test transaction (an HTTP request to the destination, a curl to the blocked service) that confirms the rule is doing what it is supposed to do.

The before-evidence is captured immediately before the change. The after-evidence is captured immediately after the change. The two are diffed; the diff is attached to the ticket.

Failure modes of undocumented changes

The undocumented change has a half-life. Within a week, the operator who made the change has forgotten the exact details. Within a month, the rationale is fuzzy. Within a quarter, the rule is mysterious. Within a year, the rule is removed by the next operator who does not know why it exists.

The failure modes:

• Orphaned rules. Rules whose rationale is lost. The next rule review marks them obsolete; the rule is removed; the incident that prompted the rule reappears.
• Lost context. The next incident investigation cannot determine whether the change is related. The change ticket would have shown the link; without it, the investigator is guessing.
• Unrepeatable rollback. The operator who needs to roll back cannot reconstruct the pre-change state. The backup is the only evidence, and the backup may be too coarse (it covers many changes, not just this one).
• Drift. The configuration drifts from the documentation. The documentation says one thing; the configuration does another. Without an evidence trail, the operator does not know which is correct.

Summary

• A change ticket is the unit of change documentation: one ticket per change, with required fields filled in before the change is applied.
• Required fields: identifier, summary, rationale, expected effect, worst-case effect, rollback, evidence.
• Peer-review sign-off is a ticket field. The reviewer reads the ticket and decides whether the procedure is sufficient for the worst case.
• Before/after evidence is captured as configuration, state, and behaviour artifacts, diffed, and attached to the ticket.
• Undocumented changes produce orphaned rules, lost context, unrepeatable rollback, and drift.