Disabling self-heal — how, who, and for how long

Self-heal suspension is the executable form of the break-glass procedure from LXXXVII-02. The discipline is to scope the break to the affected Application, set a time bound, and resume as soon as the bleeding stops. The opposite discipline - disabling self-heal globally on the cluster - is the trap this lesson exists to help the responder avoid.

The two Argo CD commands that compose the surgical break:

argocd app set $APP_NAME --self-heal=false
argocd app set $APP_NAME --sync-policy manual

The first command disables automatic reversion of drift on the named Application. The second command switches the sync policy from automated to manual, so the controller will not even apply non-drift changes without an explicit sync. Both are per-Application; both are reversible by re-running with the inverse flag values.

The two flags and what each does

--self-heal=false is the surgical disablement. The controller stops reverting drift on the named Application but continues to apply Git changes that differ from the cluster (under automated sync). The flag is the minimum break needed to preserve an emergency edit; it does not stop new Git changes from being applied.

--sync-policy manual is the broader disablement. The controller stops applying any Git changes to the named Application without an explicit argocd app sync $APP_NAME invocation. This is the break to use when the incident involves the controller itself or when the responder needs full control over when the next Git change takes effect.

The production rule: use --self-heal=false for the one-line fix shape; use --sync-policy manual for the destructive deletion and the controller-induced incident shapes from LXXXVII-01. Do not disable both globally; do not disable either globally.

flowchart LR
    A["Incident declared"] --> B{"Which shape?"}
    B -- "one-line fix" --> C["--self-heal=false"]
    B -- "destructive deletion" --> D["--sync-policy manual"]
    B -- "controller-induced" --> D
    C --> E["Apply edit, document"]
    D --> E
    E --> F["Commit to Git"]
    F --> G["Resume"]

Flux’s equivalent

Flux’s equivalent is the spec.suspend field on a Kustomization or HelmRelease. Setting it to true pauses reconciliation on the named resource; setting it back to false resumes.

flux suspend kustomization $KS_NAME
flux resume kustomization $KS_NAME

The Flux CLI commands take the resource name and namespace as arguments. They are per-resource and reversible. There is no cluster-wide suspend in Flux; the closest equivalent is to set spec.suspend: true on every Kustomization, which is the wrong default for the same reason that disabling self-heal globally is the wrong default in Argo CD.

The time bound

The suspension has a time bound. The default rule is “the duration of the declared incident, no longer”. The bound is enforced in three places:

• The incident channel. A timestamped message naming the Application, the flags set, and the expected resume time. The resume time is the incident’s expected close time plus a small buffer.
• The runbook entry. The break-glass procedure’s duration element specifies the maximum suspension. For most teams this is “until end of incident, hard cap of four hours”.
• The post-incident repair. The repair step in LXXXVII-02 is the resume; the repair has its own deadline (typically 24 hours from end of incident) that re-enables self-heal even if the responder has not done so.

The hard cap matters because incidents are sometimes forgotten once the bleeding stops. The on-call engineer resolves the incident, walks away, and the suspension persists. The post-incident repair’s job is to catch the forgotten suspension and re-enable self-heal before the next change hits.

Production discipline

1. Use per-Application flags, never cluster-wide. The break is scoped to the Application in the incident.
2. Pair the flags with a time bound. The suspension has a documented expected resume time in the incident channel.
3. Verify the suspension. After running argocd app set, run argocd app get $APP_NAME and confirm the new flags appear in the Sync Policy block.
4. Resume as part of the repair. The resume command is part of the post-incident repair checklist, not a separate action.

Cross-course references

• This course, Part LXXXVII-01 (The incident versus GitOps tension) - the three shapes that motivate the two flags.
• This course, Part LXXXVII-02 (Break-glass procedures) - the procedure that authorises the flags.
• Kubernetes for Production Sysadmins - Parts on cluster operations cover the broader pattern of scoped versus cluster-wide operational changes.

Quiz