The failure types — flaky tests, real failures, infrastructure failures, race conditions

A pipeline runs three hundred jobs a day. Two hundred and ninety seven succeed. The three that fail look identical from the dashboard: red dot, exit code 1, “build failed”. The on-call engineer hits the “retry” button. Two of them pass on retry. The third still fails. The next hour is the question - was the retry a recovery or a mask? - and that question can only be answered by classifying the failure before deciding on the response.

The four failure categories

Every pipeline failure belongs to one of four categories, and each category has its own diagnostic signal and its own correct response:

flowchart LR
    A["Pipeline failure"] --> B{"Which category?"}
    B -- "passes on retry" --> C["Flaky test"]
    B -- "fails consistently" --> D["Real failure"]
    B -- "external system" --> E["Infrastructure failure"]
    B -- "timing-dependent" --> F["Race condition"]
    C --> G["Retry; then quarantine the test"]
    D --> H["Halt and fix the change"]
    E --> I["Isolate and degrade"]
    F --> J["Sequence and add waits"]

• Flaky test. A test that passes on retry without any code change. The diagnostic signal is a low pass rate on rerun: three of five reruns pass, two fail, with no diff in the failure message. The response is to retry the job, then quarantine the test so it does not block other pipelines, then fix the test before the next cycle.
• Real failure. A failure that reproduces on retry with the same error message and the same exit code. The diagnostic signal is a stable reproduction: every rerun produces the same log line, the same stack frame, the same exit code. The response is to halt the pipeline, not retry, because the retry is guaranteed to fail and the engineer needs the failure to stay visible until it is fixed.
• Infrastructure failure. A failure caused by the runner, the network, the artifact registry, or the cloud control plane - not by the change under test. The diagnostic signal is a log line from the platform layer: HTTP 5xx from the registry, timeout on runner provisioning, “no space left on device”. The response is to isolate the failure (retry only the affected job, not the whole pipeline) and degrade the pipeline rather than fail it.
• Race condition. A failure that appears when two jobs modify shared state - a database, a queue, a file system, an API rate limit - at the same time. The diagnostic signal is non-determinism: rerunning the failed job in isolation passes; rerunning the full pipeline fails. The response is to sequence the jobs (lock, mutex, distinct namespaces) or to add an explicit wait.

Why “retry on failure” is the wrong default

The most common pipeline antipattern is a global “retry on failure” rule applied to every job. The rule is wrong because it conflates four failures that need four different responses. A retry that masks a real failure is a retry that defers the fix by one cycle. A retry on an infrastructure failure can turn a recoverable outage into a sustained outage by sending more traffic against a system that is already degraded. A retry on a race condition is a retry that may pass this time and fail the next, leaving the team with a flaky pipeline that they believe is stable.

The correct pattern is to apply retry selectively, only to the categories where retry is the right response:

jobs:
  flaky-tests:
    retries: 3
    steps:
      - run: ./run-flaky-suite.sh
  build:
    retries: 0
    steps:
      - run: ./build.sh
  publish:
    retries: 1
    continue-on-error: false
    if: always()
    steps:
      - run: ./publish.sh

The third job demonstrates if: always() - the cleanup step runs even when the job fails, which is the topic of LXI-03. The first job demonstrates selective retry on flaky tests. The second job demonstrates a deliberate zero retry on a real failure: the build must halt, not retry.

Diagnostic signals in logs

The classification depends on reading the log. The signals that distinguish each category are:

The column on the right is the test that distinguishes a race condition from a flaky test: a flaky test still fails in isolation sometimes; a race condition fails only when other jobs run. The distinction matters because the fix is different. Quarantining a flaky test removes the test from the pipeline until it is fixed. Sequencing a race condition removes the concurrency without quarantining the test.

Production discipline

1. Classify before responding. Read the log, identify the signal, name the category. The first minute of triage is the classification.
2. Never apply a global retry. Each job’s retry policy belongs in that job’s config, not in a pipeline-wide default. The default is zero.
3. Track flakiness as a metric. A test that fails one in ten runs is not “occasionally flaky”; it is a metric. The team that tracks the metric fixes the test.
4. Quarantine flaky tests, do not delete them. A flaky test removed from the pipeline is a flaky test the next developer will reintroduce. Quarantine preserves the signal.
5. Sequencing is cheaper than retries. Two jobs that race can be sequenced for less compute cost than a retry loop.

Cross-course references

• This course, Part LVII (Approvals) - lesson git-cicd-gitops-lvii-06-approval-fatigue-and-bypass-risks covers the discipline of treating automation as the decision, not the answer.
• This course, Part LX (ForwardFix) - lesson git-cicd-gitops-lx-03-when-forward-fix-is-the-right-answer frames the real-failure case as a forward-fix scenario.
• Linux for Production Sysadmins - Part XXXII (IncidentResponse) covers the triage discipline that classification supports.

Quiz