Prevent recurrence — the controls that should have caught it

The prevention controls are the changes that close the door the leak walked through. The controls are layered because no single control catches every leak path: a pre-commit hook catches the secret before it leaves the workstation, a server-side scanner catches it before it reaches the default branch, a CI scan job catches it before it merges, branch protection prevents direct pushes, and secret manager adoption removes the credential from the repository entirely.

The five layers of prevention

The five layers are stacked from the developer workstation outward to the production consumer. A leak that escapes one layer should be caught by the next:

flowchart LR
    A["developer workstation"] --> B["pre-commit hook"]
    B --> C["git push"]
    C --> D["server-side scanner"]
    D --> E["CI scan job"]
    E --> F["branch protection"]
    F --> G["secret manager adoption"]
    G --> H["production consumer"]
    A -. "secret committed" .-> X["leak"]
    B -. "block" .-> X
    D -. "block" .-> X
    E -. "block" .-> X
    F -. "block" .-> X
    G -. "remove credential" .-> X

• Pre-commit hook. Runs on the workstation before git commit. Runs a scanner against the staged changes and refuses the commit on a match.
• Server-side scanner. Runs on the git host when a push is received. Inspects every commit and blocks the push or opens an alert.
• CI scan job. Runs in the pipeline on every pull request. Fails the build on a match.
• Branch protection. Requires a pull request, a successful CI, and a code review before merge.
• Secret manager adoption. Replaces the credential with a reference to a secret stored in Vault or AWS Secrets Manager.

Choosing the layer for each leak path

• Developer laptop. Pre-commit hook.
• Pull request. CI scan job.
• Default branch. Server-side scanner.
• Production consumer. Secret manager adoption.

The post-incident review

The review turns the response into a permanent change. Four outputs:

• The timeline finalised. Appended to the incident record.
• The leak path identified. The exact sequence is reconstructed.
• The control gaps listed. Each gap has an owner and a target date.
• The policy changes committed. Recorded as a new commit in the policy repository.

Production discipline

1. The prevention controls run on the workstation, the detection controls run on the host. Both are required.
2. Every control gap has an owner and a target date. A review without owners is a confession, not a change.
3. The credential is removed, not redacted. The remediation is secret manager adoption, not a more aggressive scanner.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (KeyMgmt) covers secret manager adoption.
• Ansible for Production Sysadmins - Part XXXVIII (IncidentRunbooks) covers review patterns.
• Terraform for Production Sysadmins - Part XV (CredentialRotation) covers rotation patterns.

Quiz