Measure before optimising — the metrics and the baseline

A pipeline cannot be optimised until it is measured. The intuition that “this job feels slow” is not a measurement; it is a guess, and guesses lead to optimisations that move the wrong dial. The five numbers that characterise pipeline performance are wall-clock duration, queue time, cache-hit rate, runner cost per job, and flakiness. The baseline is the recorded set of those numbers taken against a stable workload, before any optimisation, so the change can be evaluated against the same workload.

The five metrics

The five metrics divide into two groups: the latency metrics (the first three) and the cost-and-quality metrics (the last two).

• Wall-clock duration is the elapsed time from the moment the job starts running to the moment it completes. This is the number the engineer pushing the commit sees.
• Queue time is the elapsed time from the moment the job is triggered to the moment it actually starts running. A job that is fast once it starts but waits twenty minutes for a runner is a slow job to the engineer, not a fast one.
• Cache-hit rate is the fraction of cache lookups that return an exact-match entry, averaged across all cache steps in the workflow.
• Runner cost per job is the billable minutes (for hosted runners) or compute-second equivalent (for self-hosted runners) divided by the number of jobs. Two workflows with the same wall-clock duration can have very different costs.
• Flakiness is the fraction of reruns triggered by transient failure rather than by code change. A pipeline that is fast but fails one run in ten on a flaky network test is not a fast pipeline.

flowchart LR
    A["Job triggered"] --> B["Queue time"]
    B --> C["Runner picks up job"]
    C --> D["Step runs"]
    D --> E["Cache lookup"]
    E --> F{"Exact hit?"}
    F -->|yes| G["Restore path"]
    F -->|no| H["Miss - install from scratch"]
    G --> I["Job completes"]
    H --> I
    I --> J["Recorded metrics: duration, cost, flakiness"]

The diagram shows where the latency metrics come from. Queue time is the gap between trigger and pickup. Wall-clock duration is the gap between pickup and completion. Cache-hit rate is the fraction of cache steps that hit exactly.

The baseline

A baseline is the recorded metrics for a workflow over a fixed observation window, on a fixed branch, with a fixed commit traffic. The observation window must be long enough to wash out day-of-week effects (a week of CI traffic is a common choice) and the branch must be the default branch, because pull-request builds have a different cost profile from main-branch builds.

gh api repos/ORG/REPO/actions/workflows/WORKFLOW.yml/runs \
    --paginate \
    --jq '.workflow_runs[] | {id, name, created_at, run_started_at, updated_at, conclusion, head_branch}' \
    > runs.json

The recorded fields are enough to compute duration (updated_at - run_started_at), queue time (run_started_at - created_at), and flakiness (conclusion == 'failure' followed by a rerun on the same SHA). Cache-hit rate comes from the per-step cache step output in the run logs; the GitHub Actions API exposes it as cache_hit on each cache step.

The baseline is the only evidence that an optimisation worked. Without a baseline, an engineer who halves cache-miss time by switching runners cannot tell whether the new cost per job is better, worse, or the same. The baseline is not optional.

Where the numbers come from

The five metrics have different sources:

• Wall-clock duration and queue time come from the workflow run timeline. The gh api call above captures both.
• Cache-hit rate comes from the cache step output. Each cache step emits Cache hit for primary key or Cache miss; the rate is the fraction of hit events.
• Runner cost per job comes from the billing report. Hosted runners bill per minute; self-hosted runners bill per compute-second of the underlying instance. The two are not directly comparable, but each workflow has its own cost-per-job number.
• Flakiness comes from the rerun pattern. A rerun on the same commit SHA, within minutes of the original failure, and with no intervening push, is a flake.

The five numbers together characterise the workflow. The baseline records them together. The optimisation records them again afterward. The comparison is the answer.

Production discipline

1. Record a baseline before any optimisation. The baseline is the recorded five metrics for a fixed observation window on the default branch. Without it, an optimisation is a guess.
2. Record all five metrics together. A change that improves one metric and degrades another is not an improvement.
3. Reproduce the workload. The optimisation must be evaluated on the same commit traffic the baseline was taken against.
4. Pin the observation window. A baseline taken on a quiet week and an optimisation evaluated on a busy week are not comparable.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) applies the same baseline-then-measure pattern to apt mirror performance.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) uses the same pattern for molecule run timing.
• Terraform for Production Sysadmins - Parts IX-XII (State) apply the pattern to plan/apply duration on the Terraform CI lane.

Quiz