When deployment without approval fails — failure modes and incident classes

A team that runs continuous deployment without the four compensating disciplines does not fail loudly. The CI pipeline is green; the workflow applies the artifact; the production system experiences a regression that CI could not have detected. The incident is reported hours or days later, sometimes by a customer, sometimes by an alert that fires on a derived metric. The post-incident review asks the same question every time: “CI was green; how did this happen?”

The false sense of safety

The most insidious failure mode of continuous deployment is not a particular bug. It is the belief that a green CI run proves a change is safe to ship. The belief is wrong - CI proves the run, not the change - but it is held strongly enough that teams act on it. The result is a feedback loop in which each green deploy increases the team’s confidence, and each production incident is treated as an aberration rather than as evidence that the feedback loop is wrong.

flowchart LR
    A[Green CI run] --> B{Deploy?}
    B -- "with gate" --> C[Human reviewer checks context]
    B -- "without gate" --> D[Automatic apply]
    C --> E[Safe deploy]
    D --> F[Production regression]
    F --> G[Customer reports]
    G --> H[Incident response]
    H --> I[Post-incident: CI was green]
    I --> A

The diagram shows the loop. With a gate, the human reviewer intercepts the regression before it reaches production. Without a gate, the regression reaches production, the customer notices, the on-call responds, and the post-incident review concludes “CI was green” - the same starting point that produced the regression. The gate is the only mechanism that breaks the loop.

The four incident classes

When continuous deployment fails without the compensating disciplines, it fails in four recognisable shapes:

• Silent regressions. A code change alters behaviour that tests do not cover. The change passes CI because no test exercises the affected path. Production users notice. The change reverts, but the missing test is not added; the same regression returns in a later change.
• Performance regressions. A code change increases latency, memory, or query cost in ways the test suite does not measure. CI passes because the tests check correctness, not cost. Production latency creeps up until an alert fires or a customer complains.
• Semantic breaks. A code change produces output the test suite does not validate against the production data shape. CI passes because the test fixtures are synthetic. Production behaviour diverges because the real data does not match the fixtures.
• Data-shape changes. A code change modifies a schema, a message format, or an external API contract. CI passes because the change builds. Production fails because downstream consumers of the schema or message format break in ways the pipeline does not exercise.

What each class looks like in the runbook

The runbook response is shaped by the class:

The common shape across all four classes is the same: CI was green, production broke, the runbook response is to revert and add the test or discipline that would have caught the regression. The prevention column is the discipline that continuous deployment requires and that the team, by definition, has not acquired.

The runbook gap

The most damaging consequence of the four incident classes is not the incidents themselves; it is the runbook gap. Teams that have not experienced a deployment-without- approval incident have not written the runbook for one. When the incident arrives, the response is improvised: who decides to revert, who executes the revert, how is the revert communicated, what is the rollback path. The incident expands because the response is uncoordinated.

The remediation is not a better runbook; it is the gate itself. A gate prevents the incident class. A runbook documents the response. The first is structural; the second is procedural. A team without the gate has removed the structural defence and is relying entirely on the procedural one, which they have not exercised.

Production discipline

1. Audit the failure modes. A team running deployment without the four disciplines has not yet experienced all four incident classes. The first incident in each class is a surprise; the runbook should not be.
2. A green CI run is not a safety claim. The CI pipeline verifies what it verifies. Production is what it verifies.
3. The runbook is the response, not the prevention. A documented rollback is not a substitute for a gate that prevented the change from reaching production.
4. Incident classes recur. The same silent regression returns in a later change because the missing test was not added. Track which classes recur; the recurrence is the metric.
5. The compensating disciplines are the prevention. Coverage floors, performance budgets, contract tests, and schema registries close the gaps CI leaves open.

Cross-course references

• This course, Part XXXVII (CIFundamentals) - lesson git-cicd-gitops-xxxvii-01-what-ci-is-and-is-not is the full treatment of what CI does not guarantee.
• This course, Part LII (KubernetesCI) - lesson git-cicd-gitops-lii-04-policy-conftest-and-kyverno covers policy-as-code, which is part of the semantic-break prevention column.
• Linux for Production Sysadmins - Part XXXII (IncidentResponse) covers the runbook discipline that deployment-without-approval teams typically lack.

Quiz