Status checks — the automated gate and the cost of flakiness

A status check is an assertion from a system — the CI pipeline, the forge itself, or an external integration — that the PR satisfies a policy. The forge aggregates these into a single green-or-red signal on the merge button. The cost of a flaky check is paid by every engineer who has to wait for a re-run and every auditor who cannot tell a real pass from a rerun-of-shame.

What a status check is

A status check has three properties: a name, a state (success, failure, pending, or neutral), and a URL pointing at the system that produced it. The check is registered against the commit SHA — so a force-push invalidates the old checks.

gh pr checks
gh pr status

There are three sources of status checks. CI checks come from the pipeline — GitHub Actions, GitLab CI, Jenkins. Forge-side checks come from the forge itself — branch protection, signed-commit enforcement. External checks come from third-party services — code scanning, secret scanning, dependency review.

flowchart LR
    PR["PR commit SHA"] --> A["CI: terraform plan"]
    PR --> B["Forge: signed commits"]
    PR --> C["External: secret scanning"]
    A --> G["required gate"]
    B --> G
    G --> M["merge button"]
    C -. "informational" .-> M

Required versus optional

A required check is one whose name appears in the protected branch’s required-checks list; the merge button enables only when every required check is green. A check that determines whether the change is correct — terraform plan, unit tests — is a required check. A check that determines whether the change is also good — code coverage, lint — is often optional. The trade-off is the cost of a false positive.

The cost of flaky checks

A flaky check fails on a re-run of the same commit. The CI literature reports one to five percent flakiness; the rate climbs with shared state, network, or time dependencies. A 3% rate means roughly one in thirty merges is delayed by a re-run.

The four-step recipe for retiring flaky checks:

1. Measure. Track which checks are re-run.
2. Quarantine. Move the flaky check out of the required list. The check still runs.
3. Fix or remove. Fix if possible; remove if not.
4. Restore or replace. Once reliable, return it. Otherwise find a different check.

Production discipline

1. Required checks are few and fast. Five at twenty minutes total beats fifteen at an hour.
2. Flakiness is measured. Track re-run rates; quarantine checks above 5%.
3. Required checks are deterministic. No required check should depend on shared mutable state, the network, or the wall clock.
4. Optional checks are visible. A reviewer should see them in the PR view.
5. Bypass requires reason. The reason field is what the auditor reads.

Cross-course references

• CI/CD Pipeline Patterns - Part III (FastCI): the fast-CI discipline that keeps required checks in minutes.
• GitOps with Argo CD - Part IV (SyncPatterns): the GitOps-side equivalent of required checks.
• Terraform for Production Sysadmins - Part XI (PRWorkflows): Terraform-specific required checks (terraform plan, tflint, checkov).

Quiz