The incident versus GitOps tension — when the model fights the responder

The GitOps model assumes that every change to a running system is sourced in a Git commit and applied by a reconciliation loop. The loop is continuous, periodic, and context-blind: it does not know whether the operator who last touched the cluster is on-call, off-call, asleep, or in the middle of an incident. It reads the branch tip, computes the diff against the observed state, and applies the patch.

Incident response assumes the opposite: that the operator can act on the cluster with whatever urgency the situation demands, using whichever tool is closest, without waiting for a commit, a CI run, or a controller tick.

These two contracts collide. The collision is the GitOps-incidents topic of this part.

sequenceDiagram
    participant O as Operator
    participant K as Cluster
    participant G as Git
    participant R as Reconciler
    Note over O,R: Normal operation
    G->>R: new commit
    R->>K: apply diff
    Note over O,R: Incident begins
    O->>K: kubectl edit (emergency)
    K-->>R: observed state differs from Git
    R->>K: revert emergency edit (self-heal)
    O->>R: must pause reconciler

When the model fights the responder

The model fights the responder whenever the recovery action would otherwise be reverted by the next reconciliation tick. The tick is fast - typically ninety seconds in Argo CD, sixty seconds in Flux - so the responder has roughly one tick to apply the edit and document it before the controller reverts it. Three shapes produce the fight:

• The one-line fix. A missing environment variable, a wrong image tag, a configmap key that needs a single value. The fix is kubectl edit in seconds; the commit-and-push cycle is minutes. The controller reverts the edit on the next tick if the responder does not pause it first.
• The destructive deletion. A controller is in a loop and the only recovery is to delete the resource. The deletion is urgent; the commit-and-apply cycle would let the controller re-create the resource before the deletion takes effect.
• The controller-induced incident. The reconciler itself is the cause of the divergence. The fix is to suspend the controller, not to commit around it - committing while the controller is still running produces a new desired state that the controller continues to mis-apply.

In each shape, the responder and the reconciler want different things. The responder wants the edit preserved. The reconciler wants the cluster to match Git. The reconciler wins on the next tick unless the responder pauses it.

The reconciliation loop is blind to the incident

The reconciler has no signal that an incident is in progress. It reads Git, renders the desired state, queries the cluster, computes the diff, applies the patch. The “incident is in progress” state is invisible to it. The production answer is to make the incident state visible - by suspending the affected Application, by writing a structured message to the incident channel, by recording the suspension timestamp. The controller’s blind spot is by design; the operator’s job is to fill it.

flowchart LR
    A["Declared incident"] --> B["Suspend affected Application"]
    B --> C["Apply emergency edit"]
    C --> D["Document in incident channel"]
    D --> E["Stop the bleeding"]
    E --> F["Commit change to Git"]
    F --> G["Resume reconciler"]
    G --> H["Validate convergence"]

The rule: pause rather than fight

The rule is short. During a declared incident that requires breaking the model, the responder pauses the reconciler on the affected Application, applies the edit, documents the break, and commits after the bleeding stops. The responder does not disable the reconciler globally, does not re-apply the edit every tick, and does not walk away without committing. The model is broken for the duration of the incident and repaired at the end of it.

Production discipline

1. Suspend first, edit second. The order is non-negotiable. An edit applied while the controller is running will be reverted on the next tick.
2. Document the break during the incident. Every emergency edit appears in the incident channel within five minutes of being applied.
3. Commit before resuming. The reconciler is resumed only after the Git state matches the cluster state. A reconciler resumed before the commit produces a reversion loop.

Cross-course references

• This course, Part LXXV-04 (Emergency drift) - the drift category produced when the GitOps model is broken to stop the bleeding.
• This course, Part LXXXVI-05 (Unhealthy deployment) - the failure mode where the reconciler is Synced but the application is Degraded; the boundary that triggers the one-line-fix shape.
• Kubernetes for Production Sysadmins - Parts on incident response cover the cluster-side mechanics of emergency interventions.

Quiz