Queueing and wait times — what happens when jobs exceed capacity

A runner pool can be sized correctly for the average workload and still fail during the peak. The reason is the shape of the queueing curve. As utilisation rises, wait time does not rise proportionally - it rises exponentially, and by the time wait time is visible in user-facing metrics the pool is already deep in the danger zone.

The utilisation curve

For a single queue with random arrivals and variable service times, average wait time is approximately:

wait ≈ (utilisation² / (1 - utilisation)) × avg_service_time

A pool at 50% utilisation has almost no queue. At 80% the average wait time is 0.64 / 0.20 = 3.2 times the service time. At 90% it is 0.81 / 0.10 = 8.1 times. At 95% it is 0.9025 / 0.05 = 18 times.

flowchart LR
  U["Utilisation"] -->|low| W["Wait near zero"]
  U -->|0.8| W2["Wait 3x service"]
  U -->|0.9| W3["Wait 8x service"]
  U -->|0.95| W4["Wait 18x service"]

The curve is the trap. Doubling the arrival rate at low utilisation barely moves wait time. Doubling it near saturation breaks the pool.

Recognising saturation

Saturation has three observable signals that arrive in this order:

1. Queue depth rises. The forge’s queue (workflows waiting for a runner) holds jobs instead of dispatching them immediately.
2. Wait time rises. time_queued_to_start on individual jobs grows from seconds to minutes.
3. SLA breaches appear. PRs that promised a 10-minute CI turnaround take 25.

The first signal is the cheap one. Queue depth is observable from the forge’s metrics before any user notices. The second signal is the one the team notices. The third signal is the one the manager notices.

Capping concurrency vs scaling

When saturation appears, the team has two levers:

• Scale the pool. Add runners. The fastest response on ARC is raising maxRunners; on hosted runners, raise the per-org runner limit.
• Cap concurrency. Limit how many jobs a single workflow or PR can hold. protects the pool from a runaway build matrix.

ARC_NS=actions-system
kubectl get actionsrunnersets -n "$ARC_NS" \
  -o custom-columns='NAME:.metadata.name,MIN:.spec.minRunners,MAX:.spec.maxRunners'

This shows the current minRunners/maxRunners for each runner set. The fix for a saturated pool is almost always maxRunners, not minRunners.

What the queue does to retries

A subtle failure mode: when the pool is saturated, failed jobs retry into a queue that is already long. The retry adds to the arrival rate, deepening the saturation. This is retry-induced congestion, and it turns a 5-minute blip into a 30-minute outage.

The mitigation is two-sided. First, exponential backoff on retries so failed jobs do not re-enter the queue immediately. Second, a circuit breaker that stops retries when the queue depth exceeds a threshold - failed jobs sit in a failed state instead of a queued state until the pool recovers.

Production discipline

1. Alert on queue depth, not on wait time. Queue depth is the leading indicator; wait time is the lagging one.
2. Treat 80% utilisation as the operation ceiling. Push past it only with explicit sign-off and a documented reason.
3. Retries must back off. A retry that re-enters the queue immediately is a self-inflicted attack.
4. Scale on maxRunners, not minRunners. The ceiling is what protects SLAs.
5. Run a saturation drill. Take the pool to 90% utilisation on a test day and observe the curve.

Cross-course references

• Git, CI/CD & GitOps - Part LXIII (Observability) covers the queue-depth metric in the larger observability picture.
• Observability for Production Sysadmins - Part V (QueueMath) covers Little’s Law and the utilisation curve in depth.
• Kubernetes for Production Sysadmins - Part XXII (HPAMath) covers the same curve applied to pod autoscaling.

Quiz