Emergency drift — when incident response breaks the model

Emergency drift is the category produced when an incident responder deliberately breaks the GitOps model to stop the bleeding. The model assumes all changes flow through Git; the incident assumes that flow is too slow. The reconciliation loop does not know an incident is in progress; it will keep running, and the emergency edit will either be reverted by self-heal or be preserved by suspending self-heal.

flowchart TB
    subgraph Normal["Normal operation"]
        G["Git commit"] --> C["Controller applies"]
        C --> K["Cluster state"]
    end
    subgraph Incident["Incident response"]
        K --> Op["Operator: kubectl edit"]
        Op --> K
        Suspend["Suspend reconciliation"]
        K --> Suspend
    end
    subgraph Post["Post-incident repair"]
        Commit["Commit emergency change to Git"]
        Resume["Resume reconciliation"]
        Validate["Validate convergence"]
    end
    Incident --> Post

The break-glass question

The break-glass question is the only question that matters in the first minutes of an incident where the GitOps model is in the way: can this change wait for a commit?. The answer is almost always yes for non-emergency changes; it is sometimes no for emergency changes.

The cases where the answer is no:

• A resource is crashing and a one-line fix unblocks recovery. A missing environment variable, a wrong image tag, a sidecar that needs to be removed. The fix can be applied with kubectl edit in seconds; the commit-and-push cycle takes minutes, even when the on-call engineer’s laptop is working.
• A controller is in a destructive loop and the only way out is to delete a resource. The deletion must happen now; the commit-and-apply cycle would take too long.
• The GitOps controller itself is the cause of the incident. When the controller is the source of the divergence, the fix is to suspend the controller, not to commit around it.

In each case the change is necessary, urgent, and outside the GitOps model. The change is emergency drift. The discipline that follows is the discipline that makes the break repairable.

The rule: record the break

Every emergency edit must be recorded. The record is not the Kubernetes audit log (it captures that, not why); the record is a structured artefact that an incident commander can read at the post-incident review. The standard artefacts are:

• Incident channel message. A timestamped message in the incident channel naming the resource, the field changed, the old value, and the new value. The message is part of the incident record and is preserved with the post-incident report.
• Commit after the fact. Once the bleeding has stopped, the emergency edit is committed to Git with a message that links to the incident ID. The commit is the durable record of the change; the audit log records that it was made; the incident channel records why.
• Self-heal suspension recorded. If self-heal was suspended to preserve the emergency edit, the suspension is recorded with a timestamp and an associated resume time. The default rule is “self-heal is suspended for the duration of the incident, never longer”.

The discipline is that the break is documented during the incident, not retrospectively. A retrospective commit without a contemporary incident record is one where the why has been lost.

Post-incident repair

The repair is the part of incident response most teams skip, and the part that determines whether the GitOps model survives the incident. The repair has three steps:

• Commit the change. The emergency edit is committed to Git as a new revision, with a commit message linking to the incident ID. The desired state now matches the observed state.
• Resume self-heal. If self-heal was suspended, it is resumed. The controller runs the next reconciliation tick, sees the now-matching desired state, and converges cleanly.
• Validate convergence. The team’s standard dashboards are consulted: the Application is Synced, the diff is empty, the cluster resource matches the Git manifest. A validation step that finds lingering drift is the signal that the repair is incomplete.

The repair is the production discipline that turns emergency drift into a documented exception rather than an undocumented break. The GitOps model survives incidents that are repaired; it does not survive incidents that are not.

Production discipline

1. Document the break before walking away. Every emergency edit must be in the incident channel within five minutes of being applied. The discipline is “if I do not document, I am the next incident”.
2. Suspend, never disable globally. Self-heal suspension is per-Application and time-bounded. Disablement is cluster-wide and permanent until reversed.
3. Repair is part of the incident. The post-incident review that does not include “is the GitOps model repaired and is self-heal resumed?” is an incomplete review.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on incident response cover the cluster-side mechanics of emergency interventions.
• Linux for Production Sysadmins - Parts on break-glass procedures cover the equivalent pattern in configuration management.
• Terraform for Production Sysadmins - Parts on state-file emergencies cover the analogous problem with cloud infrastructure.

Quiz