The postmortem and the policy update — what changes after the incident

A generic postmortem records what happened, identifies the root cause, and lists action items. A GitOps-aware postmortem does all of that and produces a policy update - a concrete change to one of four policy categories - that prevents the incident from recurring. The postmortem without a policy update is a narrative the team reads once and forgets; the postmortem with a policy update is a change the team deploys and the cluster enforces.

The four policy categories that change after a GitOps incident:

• The runbook. The break-glass procedure, the on-call checklist, the incident-response playbook. The runbook is the operational artefact the next responder reads at 03:00.
• The sync windows. The automated-sync policy on the affected Application, the cluster, or the project. Sync windows determine when Git changes are allowed to reach the cluster; the incident may have revealed a window that was too permissive or too restrictive.
• The review gates. The PR template, the required reviewers, the CI checks that catch the kind of change that caused the incident. The review gates are the preventative controls.
• The controller configuration. The Argo CD or Flux settings on the affected Application - revisionHistoryLimit, retry, prune, self-heal, the health check. The controller configuration is the runtime contract.

Each category has an owning artefact, a deadline, and a deploy path. The postmortem produces a delta in each category where the incident demands one.

flowchart LR
    A["Postmortem"] --> B["Runbook delta"]
    A --> C["Sync window delta"]
    A --> D["Review gate delta"]
    A --> E["Controller config delta"]
    B --> F["Policy update in production"]
    C --> F
    D --> F
    E --> F

The timeline for policy updates

The four categories have different deadlines. The postmortem assigns each delta to an owner with a date; the team’s weekly meeting tracks open deltas.

• Runbook delta: 48 hours. The runbook is what the next on-call engineer reads. The delta must be merged into the runbook within 48 hours of the postmortem, while memory of the incident is fresh and the next responder can be warned.
• Sync window delta: one week. Sync windows are cluster-level configuration; changes require platform-team review. The delta is opened as a PR, reviewed, and deployed within a week.
• Review gate delta: two weeks. Review gates are CI changes; they require build pipeline updates and a rollout to every pipeline that runs the affected check. The delta is a tracked issue with a two-week target.
• Controller config delta: one week. Controller configuration changes are per-Application or per-project; the delta is a PR against the manifests repository, reviewed and deployed within a week.

The deadlines compose a fast loop. A team that hits all four within a week has a fast loop; a team that takes a month has a slow loop and will pay the incident cost again before the policy is in production.

What each category’s delta looks like

The four categories produce different artefacts. The postmortem’s job is to specify each delta precisely enough that the owner can execute it without re-deriving the reasoning.

• Runbook delta. A diff against the break-glass procedure from LXXXVII-02: a new trigger added, a duration tightened, a repair step clarified. The diff is reviewed by the team and merged into the runbook repository.
• Sync window delta. A manifest change in the argocd-app.yaml (or equivalent Flux resource): a new syncWindow block restricting automated sync to a narrower window; a manual sync policy replacing automated for the affected Application.
• Review gate delta. A change to the PR template: a new review question that catches the kind of change that caused the incident. A CI check that validates the manifest against the new gate. The check is added to the pipeline and required for merge.
• Controller config delta. A change to the Application’s spec: revisionHistoryLimit raised to retain more rollback history; retry.retryLimit increased; selfHeal re-enabled after a temporary suspension; a new health check that catches the failure mode that produced the incident.

Each delta has an owner, a deadline, and a deploy path. The postmortem is the document that ties the four deltas to the incident and gives the team a checklist to track them.

flowchart LR
    A["Runbook delta"] --> B["Runbook repository PR"]
    C["Sync window delta"] --> D["Manifests PR"]
    E["Review gate delta"] --> F["CI repository PR"]
    G["Controller config delta"] --> H["Application spec PR"]
    B --> I["Merged within 48h"]
    D --> J["Merged within 1 week"]
    F --> K["Merged within 2 weeks"]
    H --> L["Merged within 1 week"]

Production discipline

1. Every postmortem produces at least one policy delta. A postmortem with no policy delta is an open incident.
2. Each delta has an owner and a deadline. Unowned deltas are undelivered deltas.
3. The weekly meeting tracks open deltas. A delta open for more than two weeks is a loop violation.
4. The loop closes when the delta is in production. The postmortem is not “done” until the change deploys.

Cross-course references

• This course, Part LXXXV-06 (Post-rollback investigation)
  • the audit trail discipline that feeds the postmortem.
• This course, Part LXXXVII-04 (Post-incident Git reconciliation) - the Git-side closure that precedes the policy update.
• Google SRE Book - Postmortem Culture - the blameless framework the GitOps postmortem extends.

Quiz