Success rate and flake rate — the distinction and how to interpret a 5% flake rate

A test suite that passes 95% of the time can be a test suite that fails 30% of the time. The 95% is the success rate - the fraction of runs whose final conclusion is success. The 30% is the flake rate - the fraction of runs that pass only after one or more failures were retried. The two numbers describe the same system from different angles, and a team that watches only the first is a team that does not see the second.

Success rate

Success rate is the simplest CI metric: the fraction of runs in a window whose final conclusion is success. The GitHub Actions API returns the conclusion for every run:

OWNER=acme
REPO=platform
gh api repos/$OWNER/$REPO/actions/runs \
  --jq '[.workflow_runs[] | .conclusion] | group_by(.) | map({conclusion: .[0], count: length})'
# returns: counts grouped by conclusion (success, failure, cancelled, ...)

A team that graphs count where conclusion == "success" / total over time has a success-rate dashboard. The signal is meaningful when interpreted against the baseline: a team whose success rate drops from 95% to 88% over a week is a team with a regression; a team whose success rate is 95% consistently is a team with a stable pipeline.

But success rate alone is insufficient. A 95% success rate with automatic retries can be a 70% per-attempt success rate. The team that watches only the success rate sees “95% green”; the team that watches the per-attempt rate sees “30% of attempts are failing”. The two views produce different operational decisions.

Flake rate

Flake rate is the fraction of runs that pass only after one or more failures were retried. A flaky test is a test that sometimes fails for non-deterministic reasons (timing, network, shared state, race conditions) and sometimes passes. The CI system marks the run as success after the retry succeeds; the run is green but the suite is failing.

The flake rate is computed from the per-step retry count. A run with no retries has a flake rate of 0 for that run; a run with one retry has a flake rate of 1; a run with three retries has a flake rate of 1 (the run is counted once, not three times):

# flake rate over the last N runs, computed from the per-step
# retry count in the run logs
# (requires reading the step logs for each run)
TOTAL_RUNS=100
FLAKY_RUNS=$(gh api repos/$OWNER/$REPO/actions/runs \
  --jq '.workflow_runs | length')
# actual flake count requires per-run log inspection; the
# summary metric is `runs with >= 1 retry / total runs`

flowchart LR
    A["100 runs"] --> B["80 runs: pass on first attempt"]
    A --> C["15 runs: pass on retry 1"]
    A --> D["5 runs: pass on retry 2 or later"]
    B --> E["Success rate: 100/100 = 100%"]
    C --> E
    D --> E
    B --> F["Per-attempt success rate: ~80/120 = 67%"]
    C --> F
    D --> F

The diagram above shows the gap. A team that measures success rate sees 100%. A team that measures per-attempt success rate sees 67%. The 67% is the operational truth; the 100% is the system’s optimistic summary.

Interpreting a 5% flake rate

A 5% flake rate means 5% of runs pass only after at least one retry. The number sounds small; the operational consequence is large:

• With 1 retry as the median. A 5% flake rate means roughly 5% of runs require one extra attempt. The team that runs 100 deploys per week is a team that sees 5 retries per week - visible as occasional “CI is slow today” complaints.
• With 3 retries as the median. A 5% flake rate means 5% of runs require 3 extra attempts. The team that runs 100 deploys per week is a team that sees 15 retries per week - visible as a queue of slow runs and a dashboard of “test suite is slow today”.
• With 10 retries as the median. A 5% flake rate is a suite that is barely passing. The team that runs 100 deploys per week is a team that sees 50 retries per week - visible as a queue of failed runs that eventually succeed.

The 5% number is the same; the operational reality is different. A team that reports “5% flake rate” without the retry-count context is a team that is missing the operational signal.

Alert thresholds for flake rate

The right alert threshold for flake rate is lower than most teams expect. A 5% flake rate is a serious signal; a 1% flake rate is a noticeable signal; a 0.1% flake rate is a worth-investigating signal. The threshold depends on the team’s volume:

• High-volume team (100+ runs/day). Alert at 0.5% flake rate. A 0.5% flake rate at 100 runs/day is one flake every two days; the cause is usually a single non-deterministic test that needs fixing.
• Medium-volume team (10-100 runs/day). Alert at 2% flake rate. A 2% flake rate at 50 runs/day is one flake per day; the cause is usually a small cluster of non-deterministic tests.
• Low-volume team (<10 runs/day). Alert at 5% flake rate. A 5% flake rate at 5 runs/day is one flake per four days; the cause is usually the same small cluster, and the alert is rare enough to be actionable.

The alert message should include the test name, not just the rate. A team that gets paged with “test flake rate at 3%” is a team that has to investigate; a team that gets paged with “test flake rate at 3% - top offender: test_payment_flow.py” is a team that can act.

Production discipline

1. Measure per-attempt success rate, not just success rate. The two numbers tell different stories.
2. Disable automatic retries on tests that are not designed for them. A retry is a debugging primitive; it should not be a permanent feature of the suite.
3. Treat each flaky test as a P1 bug. A flaky test is a test that is failing for non-deterministic reasons; non-determinism is a correctness issue.
4. Alert on flake rate with the test name, not just the rate. The alert should be actionable, not just informative.
5. Track the per-attempt runtime separately from the per-run runtime. The cost of retries is in the per-attempt runtime, not in the run page’s reported duration.

Cross-course references

• Observability course - Part III (BurnRate) covers the alert-threshold discipline in detail: a low-rate signal with high consequence (a flaky test that ships a bug) deserves a low threshold.
• This course, Part LI-02 (Retries) covers the retry mechanism in CI; this part covers the metric that surfaces when the retry is masking a failure.
• This course, Part XXXIX-05 (PipelineStatus) covers the retry count in the step logs; this part covers the aggregate retry rate across runs.

Quiz