Retries and backoff — automatic retry vs manual retry, and the exponential backoff pattern

A retry is a deliberate reattempt of an operation that failed. The word deliberate matters: a retry without a stated reason is a retry that hides the cause of the failure from the next runner. The pipeline runs, fails, retries, succeeds; the engineer never sees the failure; the test that needed attention continues to flake because no one fixed it. The correct retry is a retry with a stated policy, an upper bound, a delay pattern, and a metric.

Manual retry versus automatic retry

The two kinds of retry are manual and automatic. Each has a place; neither is the default:

• Manual retry. A human presses the “retry” button after reading the failure. The human has classified the failure (LXI-01), decided that retry is correct, and is willing to consume a CI minute to confirm. Manual retry is appropriate for failures that look new, ambiguous, or one-off.
• Automatic retry. The platform re-runs the failed job N times before declaring the pipeline failed. The policy belongs in the workflow file; the engineer sees the eventual outcome, not the individual retries. Automatic retry is appropriate for the categories where retry is correct: flaky tests (LXI-01), certain infrastructure failures (transient registry errors), and race conditions that sequencing does not yet cover.

A pipeline that retries automatically is a pipeline that has made the decision for the engineer. That decision is correct when the policy matches the failure category. The discipline is the match:

flowchart LR
    A["Failure observed"] --> B{"Policy matches category?"}
    B -- "Yes" --> C["Automatic retry"]
    B -- "No" --> D["Halt; manual triage"]
    C --> E["Success on retry"]
    C --> F["Failure persists"]
    E --> G["Pipeline continues"]
    F --> H["Engineer investigates"]

jobs:
  test:
    retries: 3
    steps:
      - run: ./test.sh
  publish:
    retries: 0
    steps:
      - run: ./publish.sh
  cache-warm:
    retries: 1
    steps:
      - run: ./warm-cache.sh

Each job gets the retry policy that matches its failure profile. The test job retries three times for flakiness. The publish job does not retry because the publish must halt on failure. The cache-warm job retries once because warming a cache can fail on a cold runner and succeed on a warm one.

The exponential backoff pattern

The worst retry pattern is a fixed delay. One hundred failing pipelines with a fixed ten-second delay all retry at the same instant; the downstream system sees a thundering herd; the retries amplify the failure rather than recover from it.

The correct pattern is exponential backoff with jitter:

import random

def backoff(attempt):
    base = 2 ** attempt           # 2, 4, 8, 16, 32 seconds
    jitter = random.uniform(0, 1)  # 0.0 to 1.0 seconds
    return base + jitter

The base doubles each attempt (so retry N waits roughly 2^N seconds). The jitter adds a random fraction of a second so that one hundred failing pipelines do not retry in lockstep. The result is a retry pattern that recovers the failing pipelines without producing a stampede.

GitHub Actions supports retry but does not natively back off; the delay is the platform’s internal scheduler. The correct place for an exponential backoff is in the script the job calls, not in the workflow itself:

attempt=0
max_attempts=4
until [ "$attempt" -ge "$max_attempts" ]; do
    if ./publish.sh; then
        exit 0
    fi
    attempt=$((attempt + 1))
    sleep $((2 ** attempt + RANDOM % 2))
done
exit 1

GitLab CI supports a per-job retry with when and max clause; a runner-side backoff is the job’s responsibility. Jenkins’ options { retry(n) } produces the same shape - the workflow retries N times, and the backoff is in the script.

The retry budget and the circuit breaker

Two additional controls complete the retry pattern:

• Retry budget. The maximum fraction of total pipeline runs that may be retries in a given window. A 10% budget means that if 100 pipelines ran in the last hour, no more than 10 may have been retries. When the budget is exhausted, retries are paused to give the downstream time to recover without amplification.
• Circuit breaker. A control that opens when the downstream is known to be down and short-circuits all retries until the downstream recovers. The breaker is separate from the retry policy; the breaker decides whether retrying is allowed, the retry policy decides how to retry.

A pipeline with a retry budget but no breaker can still amplify an outage by retrying within its budget. A pipeline with a breaker but no budget can still saturate the downstream when the breaker has not yet opened. The two controls are not substitutes; they cover different failure modes.

When automatic retry is the wrong choice

Automatic retry is wrong for:

• Real failures. A retry on a broken change masks the failure for one cycle and guarantees the same failure on the next.
• Infrastructure failures with shared state. A retry on a publish that holds a partial lock can compound the lock rather than release it.
• Operations that are not idempotent. A retry on a non-idempotent operation can re-apply a change that was partially applied - which is the topic of LXI-04 and LXI-05.

Production discipline

1. Default retry is zero. Per-job retry policy with a default of zero. The retries that exist are deliberate.
2. Backoff is exponential with jitter. Fixed delay is a pattern that produces storms; it is not a pattern that produces recovery.
3. Bound retries with a budget. A retry budget prevents the platform from amplifying an outage.
4. Open a circuit breaker when the downstream is down. The breaker is independent of the retry policy.
5. Treat automatic retry as a decision, not a default. The pipeline that retries automatically has decided what counts as recoverable; that decision belongs in the workflow file, not in the engineer’s instincts.

Cross-course references

• This course, Part LXI-01 (FailureTypes) covers the classification that justifies the retry policy.
• This course, Part LXI-05 (Idempotency) covers why retry is only safe when the operation is idempotent.
• Linux for Production Sysadmins - Part XXXIII (ServiceReliability) covers retry budgets and circuit breakers as platform patterns.

Quiz