The performance loop — measure, optimise, re-measure

Pipeline performance is not a one-time optimisation. The workload changes - new packages, new tests, new dependencies. The runner pool changes - new instance types, new regions, new capacity. New levers become available - new cache backends, new sharding strategies, new matrix options. The performance characteristic that was optimal last quarter is suboptimal this quarter. The right framing is a loop: measure, optimise, re- measure, compare, decide.

The loop

The loop has four steps:

1. Measure the baseline. Record the five metrics for a fixed observation window on the default branch. The observation window must reproduce the workload.
2. Pull one lever. Change exactly one configuration: add a cache step, change a runner size, adjust the matrix shard count, add a path filter. One change per loop iteration.
3. Re-measure against the same workload. Record the five metrics for the same observation window on the same branch.
4. Compare and decide. If the new metrics are better on the dimensions that matter, keep the change. If they are worse on any dimension that matters, revert. If the comparison is ambiguous, re-run with a longer observation window.

flowchart LR
    A["Measure baseline"] --> B["Pull one lever"]
    B --> C["Re-measure"]
    C --> D{"All five metrics better?"}
    D -->|yes| E["Keep change"]
    D -->|no| F{"Worse on a dimension that matters?"}
    F -->|yes| G["Revert change"]
    F -->|no| H["Re-measure with longer window"]
    H --> C
    G --> I["Loop continues"]
    E --> I
    I --> J["Workload or pool changes"]
    J --> A

The diagram shows the loop. The “all five metrics better” check is strict: a change that improves wall-clock duration but degrades cost per job is a regression, not an optimisation. The team decides which dimensions matter; the loop enforces the measurement.

What triggers a new iteration

The loop runs whenever any of these changes:

• The workload changes. A new package added to the monorepo changes the test suite’s serial duration. A new dependency added to the lockfile changes the dependency-download cost. A new platform added to the matrix changes the shard count.
• The runner pool changes. A new instance type becomes available. A new region is added. The pool’s capacity at peak shifts (a new team joins the org, an existing team changes their CI pattern).
• A new lever becomes available. A new cache backend is released. A new matrix option is added. A new actions version changes the cache step’s behaviour.
• A regression is observed. A metric that was stable begins to drift. This is the early-warning signal that the loop has not been run recently enough.

The loop is not run on a schedule; it is run on a trigger. A team that runs the loop quarterly will fall behind a team that runs the loop on every workload change. The trigger is the change, not the calendar.

The regression failure mode

A regression is a change that improves one metric and degrades another. The baseline would have caught the regression before ship; the post-change metrics would have shown it. A loop that checks only one metric - typically wall-clock duration, because it is the most visible - misses regressions on cost, queue time, cache-hit rate, and flakiness.

A team that watches only wall-clock duration sees the dashboard improve and ships the change. The monthly bill doubles because the runner change increased per-minute cost. The team discovers the regression at the end of the month when the invoice arrives; the loop should have caught it the week the change shipped.

The fix is to record all five metrics together, on every loop iteration, and to compare them together. A change that improves one metric at the cost of another is not an optimisation; it is a regression that happens to be visible on one dimension.

Documenting the loop

The loop produces a record. Each iteration has:

• The baseline (date, branch, five metrics).
• The change (workflow file diff, configuration change).
• The post-change metrics (same date format, same branch, same five metrics).
• The decision (keep, revert, extend observation).

gh api repos/ORG/REPO/actions/workflows/WORKFLOW.yml/runs \
    --paginate \
    --jq '.workflow_runs[] | {id, head_branch, conclusion, created_at, run_started_at, updated_at}' \
    > runs-iteration-N.json

The record is what the next iteration compares against. A team that does not keep the record cannot run the loop; the comparison has nothing to compare to. The record is the artefact the loop produces, and it is the artefact the next loop iteration consumes.

Production discipline

1. One lever per loop iteration. Two levers at once produce an unattributable result.
2. Record all five metrics per iteration. A change that improves one metric and degrades another is a regression.
3. Trigger on change, not on calendar. The loop runs when the workload, the pool, or the available levers change.
4. Keep the record. The baseline, the change, and the post-change metrics are the artefacts the next iteration consumes.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) applies the same loop to apt mirror performance and image build duration.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the loop to molecule run timing and inventory size.
• Terraform for Production Sysadmins - Parts IX-XII (State) apply the loop to plan/apply duration and provider plugin download time.

Quiz