Skip to main content
RunBook Academy

Git, CI/CD & GitOpsLXXXVII · GitOps During IncidentsPragmatism

Pragmatism versus purity — when GitOps can wait

Advanced⏱ ~23 mingit

What you'll learn

  • Recognise the workloads and environments where the GitOps discipline is the wrong default
  • Distinguish the GitOps boundary - declarative, reproducible, audit-critical - from the CI/CD boundary - imperative, ephemeral, low-risk
  • Apply the team-cost calculus that determines whether GitOps is worth the discipline for a given system
  • Identify the anti-patterns that arise from forcing GitOps onto systems that do not need it

Prerequisites

Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x

Not yet marked complete on this device.

GitOps is a discipline, not a religion. The break-glass procedures in LXXXVII-02 exist because the discipline has real cost and real benefit. This final lesson asks the question the rest of the part takes for granted: is GitOps the right default for this system, or am I forcing a discipline that does not pay?

GitOps is the right default for some systems and the wrong default for others. A blanket policy pays the discipline cost on systems that do not need it; a selective policy pays it only where it returns value.

flowchart LR
    A["System"] --> B{"Needs declarative reproducibility?"}
    B -- yes --> C["GitOps"]
    B -- no --> D{"Needs audit-grade history?"}
    D -- yes --> C
    D -- no --> E{"Long-lived, regulated, multi-team?"}
    E -- yes --> C
    E -- no --> F["Traditional CI/CD"]

The GitOps boundary

The GitOps boundary separates systems where the discipline pays from systems where it does not. Three properties compose the boundary:

  • Declarative reproducibility. The system can be described declaratively (manifests, IaC, configuration as data) and reproduced from the description. A stateful service with imperative-only configuration is outside the boundary.
  • Audit-grade history. The system’s change history must be traceable to a commit, a PR, a reviewer, and a rationale. An ephemeral test environment or a single-engineer tool is outside the boundary.
  • Long-lived, multi-team, regulated. Operated by more than one team, lives longer than the on-call rotation, and is subject to compliance review. A one-engineer tool replaced every quarter is outside the boundary.

A system with none of the three is a traditional-CI/CD candidate. A system with one is a GitOps candidate. A system with all three is a strong GitOps candidate whose discipline pays for itself many times over.

The anti-patterns of forced GitOps

Forcing GitOps onto a system outside the boundary produces three recognisable anti-patterns:

  • Ceremony without benefit. Every change requires a commit, a PR, a CI run, and a controller tick. A single-line configmap change becomes a five-minute deploy. The team starts skipping the ceremony (“just kubectl edit it for now”) and the Git history stops being authoritative.
  • Self-heal revert wars. The controller reverts imperative edits the team makes for legitimate operational reasons - the kind that produced the break-glass procedure in LXXXVII-02. The team escalates to disabling self-heal, which removes the GitOps benefit.
  • Audit theatre. The team writes commits for changes that no one reviews and PRs that no one reads. The Git history records the changes; the audit trail is empty. The system has the cost of GitOps without the benefit.

The three anti-patterns are signals that GitOps is the wrong default for the affected system. The remediation is to roll back to traditional CI/CD, not to push harder on the discipline.

The team-cost calculus

The calculus that decides whether GitOps pays has three inputs:

  • Change frequency. One change per week pays a low GitOps overhead; fifty per week pays a high overhead. The break-even point is roughly the frequency at which the CI pipeline becomes a bottleneck.
  • Cost of an unreviewed change. A system where an unreviewed change can take down production has a high cost; a system where the change can be reverted in five minutes has a low cost. GitOps pays when the cost is high.
  • Team size. A team of one pays the discipline cost alone; a team of twenty amortises it. The break-even size is roughly three engineers - below that, the overhead exceeds the review benefit.

The calculus is a heuristic, not a formula. A team that applies it consciously to each system pays the discipline only where it returns value; a team that applies GitOps by default pays it everywhere, including on the systems that do not need it.

flowchart LR
    A["System N"] --> B{"Frequency > threshold?"}
    B -- yes --> C["High GitOps overhead"]
    B -- no --> D["Low overhead"]
    C --> E{"Cost of unreviewed change > threshold?"}
    D --> E
    E -- yes --> F["GitOps pays"]
    E -- no --> G["Traditional CI/CD pays"]

When GitOps can wait

Some systems are the right long-term GitOps default but the wrong short-term one. A new system in its first month does not have the change history to justify the discipline; a system in its fifth year does. The team that defers GitOps until the system has earned it is applying the calculus correctly.

  • Ephemeral infrastructure. Test environments, sandbox clusters, demo systems - replaced every quarter; the change history is the CI pipeline, not Git.
  • Exploratory systems. New products, proof-of-concept deployments - Git would record decisions about to be revisited.
  • Single-engineer-owned tools. Internal dashboards, one-person automations - the review is the engineer themselves.

The three cases defer GitOps, not reject it. The system migrates when frequency rises, unreviewed-change cost rises, or team size grows.

Production discipline

  1. Apply GitOps selectively. Each system gets the calculus before it gets the discipline.
  2. Watch for the anti-patterns. Ceremony without benefit, self-heal revert wars, audit theatre - each is a signal that the discipline is on the wrong system.
  3. Defer GitOps when the calculus does not justify it. Ephemeral, exploratory, single-engineer-owned systems can wait.
  4. Reconsider when the calculus changes. A system that graduates to multi-team or to long-lived operation is ready for the discipline.

Cross-course references

  • This course, Part LXXXVII-01 (The incident versus GitOps tension) - the tension the calculus must justify.
  • This course, Part LXXVI-06 (The architecture decision) - the architectural choices the calculus informs.
  • OpenGitOps principles - the declarative, versioned, pulled, continuously-reconciled defaults.

Quiz

Knowledge check · 4 questions

  1. Q1. A team runs a small internal dashboard used by 12 people, operated by one engineer, with one change every two weeks and a five-minute revert path if the change is wrong. According to the team-cost calculus, what is the right default?

  2. Q2. A blanket 'GitOps everywhere' policy is not necessarily the safest default because it enforces the discipline uniformly across the cluster.

  3. Q3. Name the three properties that compose the GitOps boundary and the three anti-patterns that arise from forcing GitOps onto systems outside the boundary.

  4. Q4. Diagnose why the team's GitOps rollout is producing friction, and recommend the right scope for the discipline.

    A team has applied GitOps to a fleet of 40 systems: 8 production services, 12 staging environments, 15 ephemeral test clusters, and 5 internal dashboards. The 8 production services benefit from the discipline. The 12 staging environments produce frequent self-heal revert wars because the test framework applies imperative changes the controller reverts. The 15 ephemeral test clusters are replaced every two weeks; the Git history is empty because the commits are never merged before the cluster is destroyed. The 5 internal dashboards have PRs that no one reviews. The team is considering abandoning GitOps entirely.

Passing score: 75%. Answers are checked in this browser.