Runner pool sizing — the workload model and the concurrency limit

A runner pool cannot be sized from a guess. It can only be sized from the workload it has to absorb. The two numbers that matter are how many jobs arrive and how long each one takes. From those two numbers, a single formula gives the concurrency the pool must support at any instant.

The workload model

A workload model for a runner pool is three numbers and one distribution:

flowchart LR
  A["Arrivals\njobs/day, peak/hour"] --> C["Concurrency limit"]
  D["Duration\nminutes/job, p50/p95"] --> C
  R["Retry factor\nx1.0 to x2.0"] --> C
  C --> S["Pool size\nwith headroom"]

• Arrivals. Jobs per day. From the forge’s API or metrics. Convert to jobs per minute (JOB_RATE). Identify the peak hour - a single hour can carry 8-15% of the day’s volume in CI workloads.
• Duration. Average and p95 runtime per job. Duration is not a constant; a pool sized for the average will starve during the long tail.
• Retry factor. Most CI systems retry failed jobs. A retry factor of 1.2 means 20% more job-arrivals than first-attempt triggers suggest.

The output of the workload model is peak-hour arrival rate. For a team running 1,200 jobs/day with a peak-hour factor of 0.10, the peak hour sees 120 jobs. Spread over 60 minutes that is 2 jobs/minute arriving. If average duration is 8 minutes, 16 jobs are in flight at any instant under steady state.

The concurrency limit

The relationship between arrival rate, duration, and in-flight count is Little’s Law:

concurrent_jobs = arrival_rate_per_second × average_duration_seconds

Worked example:

ARRIVAL_PER_MIN=2
DURATION_MIN=8
CONCURRENCY=$(awk -v a="$ARRIVAL_PER_MIN" -v d="$DURATION_MIN" \
  'BEGIN { printf "%.1f", a*d }')
echo "Concurrency limit: $CONCURRENCY jobs in flight"

The formula is exact. The pool must hold at least concurrency simultaneous jobs or the queue grows without bound. A pool of 10 runners, with a concurrency limit of 16, will queue six jobs at peak.

Headroom and burst

The concurrency limit is the floor. The pool needs headroom on top of it for:

• Retries. A failed job that retries adds a second arrival. Retry storms can double the arrival rate for minutes.
• Burst. A large merge at 10am can push twenty PRs into CI at once.
• Maintenance. One or two runners should be drainable at any time for image upgrades without starving the pool.

A practical rule is 2x the concurrency limit for the minimum pool size. A pool sized at exactly the limit will queue jobs the first time a retry fires.

When the model is wrong

The workload model is built from history. It will be wrong when:

• A new repository joins the org and adds a job class the pool has not seen before (e.g. a long integration suite).
• A dependency upgrade doubles build time across every job.
• The team adopts trunk-based deployment and starts running CI on every push instead of every PR.

Re-derive the model quarterly. The cost of a stale model is a pool that queues during the busiest week of the quarter.

Production discipline

1. Derive size from workload, not from budget. Reverse the derivation once the budget is known; otherwise the budget caps the SLA.
2. Use peak hour, not average hour. A pool sized for the mean will queue at the peak.
3. Include retries and re-runs. They are arrivals too.
4. Re-derive the model every quarter. The workload drifts.
5. Plan for 2x headroom. Exact-fit pools are queueing pools.

Cross-course references

• Git, CI/CD & GitOps - Part XL (Runners) covers the runner classes whose sizes feed this model.
• Observability for Production Sysadmins - Part V (QueueMath) covers the queueing-theory identities in more depth.
• Kubernetes for Production Sysadmins - Part XXII (HPAMath) covers the same identity applied to pod scaling.

Quiz