Break-glass procedures — the discipline of breaking the model on purpose

A break-glass procedure is a pre-written runbook for breaking the GitOps model on purpose. The phrase comes from the physical break-glass box that covers a fire alarm: the alarm exists behind glass to discourage casual use; the glass is there so the responder has to break something deliberate to use it. A GitOps break-glass procedure is the same idea applied to the reconciliation loop - the loop is the default; the procedure authorises breaking it; the break must be documented and repaired.

The discipline is to write the procedure before the incident. A procedure written during the incident is a procedure written under pressure, by the person under pressure, for an incident that may not recur. A procedure written before the incident is a checklist the responder follows under pressure, authored by the person who was calm, for an incident that recurs.

flowchart LR
    A["Pre-incident: write the procedure"] --> B["Incident declared"]
    B --> C["Operator follows the checklist"]
    C --> D["Break is documented"]
    D --> E["Post-incident: repair the break"]
    E --> A

The four elements of a GitOps break-glass procedure

Every GitOps break-glass procedure has four elements. The procedure is incomplete without any of them, and an incomplete procedure is the same as no procedure - the responder improvises under pressure and improvises wrong.

• The trigger. What incident shape authorises the break. The one-line fix, the destructive deletion, the controller-induced incident from LXXXVII-01. The procedure names each trigger and the action it authorises.
• The owner. Who is accountable for the break. The on-call engineer, named in the incident channel. The owner is the only person authorised to apply the break and is the only person accountable for repairing it.
• The duration. How long the break is allowed to persist. The default is “the duration of the declared incident, no longer”. A procedure that allows an indefinite break is a procedure that has effectively disabled the model.
• The repair. What commits and what controller changes close the loop. The manifest commit that matches the emergency edit, the resume of self-heal, the validation of convergence. The repair has a deadline - typically 24 hours from the end of the incident.

The four elements compose a contract. The trigger says when; the owner says who; the duration says for how long; the repair says how it ends. The responder who follows the procedure knows exactly what to do and exactly when they are done.

Why the procedure is written before the incident

The procedure is written before the incident for three reasons:

• The responder is calm. The author of the procedure is not on-call at 03:00. The procedure reflects how the team wants the responder to act under pressure, not how the responder is tempted to act under pressure.
• The procedure is reviewable. A pre-written procedure can be reviewed by the team, refined in tabletop exercises, and updated after each incident. A procedure written during the incident is one engineer’s best guess and is never seen by anyone until the next incident.
• The procedure is executable under pressure. A 200-word checklist is executable by an on-call engineer at 03:00. A 2,000- word policy is not. The procedure is the executable form of the policy.

argocd app set $APP_NAME --self-heal=false
argocd app set $APP_NAME --sync-policy manual

These two commands are the executable form of the suspend step in most Argo CD break-glass procedures. They are the checklist.

The break-glass procedure in the runbook

The break-glass procedure lives in the team’s runbook alongside the incident response procedures. The cross-link matters: the incident response procedure says “for GitOps incidents, see the GitOps break-glass procedure”; the GitOps break-glass procedure says “for incident declaration, see the incident response procedure”. The two procedures compose into a single runbook entry.

flowchart LR
    A["Incident declared in channel"] --> B["Read incident response procedure"]
    B --> C{"Is this a GitOps break?"}
    C -- no --> D["Standard incident response"]
    C -- yes --> E["Read GitOps break-glass procedure"]
    E --> F["Suspend affected Application"]
    F --> G["Apply emergency edit"]
    G --> H["Document in incident channel"]
    H --> I["Stop the bleeding"]
    I --> J["Post-incident repair within 24h"]

Production discipline

1. Write the procedure before the incident. The procedure is a pre-incident artefact, not an incident artefact.
2. Test the procedure in tabletop exercises. A procedure that has not been rehearsed is a procedure that has not been written.
3. Update the procedure after every incident. Every incident produces a procedure delta: a trigger added, a duration tightened, a repair step clarified.
4. Cross-link from incident response. The two procedures are one runbook; the cross-link is what makes them one.

Cross-course references

• This course, Part LXXXVII-01 (The incident versus GitOps tension) - the three shapes the procedure must cover.
• This course, Part LXXV-04 (Emergency drift) - the drift category the procedure produces.
• Linux for Production Sysadmins - Parts on break-glass procedures cover the equivalent pattern in configuration management and the parallel discipline of writing them in advance.

Quiz