Concurrency limits and throttling — runner concurrency, queue depth, and the queue effect

A team has ten runners. Twenty pull requests merge in an hour. Each pull request triggers a build job. The runners pick up ten of the twenty jobs; the other ten queue. The queued jobs wait for a runner to free up. The team is unhappy: builds are slow, queue depth is high, and developers are asking why CI is getting worse. The team is conflating two different limits: runner concurrency (how many jobs can run at once) and target concurrency (how many deploys can safely run at once). This lesson is about the difference and the right way to set both.

Runner concurrency and the queue effect

A runner concurrency limit is the maximum number of jobs that can run simultaneously on the runner pool. For GitHub-hosted runners, the limit is per-org (and tier-dependent). For self-hosted runners, the limit is the number of runner processes registered with the controller. The limit is a throttle: it caps throughput at limit / service_time jobs per unit time. When arrival rate exceeds that throughput, jobs queue.

flowchart LR
    A[Job arrival rate 20/hr] --> B[Runner pool size 10]
    B --> C[Service time 6 min/job]
    C --> D[Throughput 100/hr]
    A --> E[Queue: 20 - 0 = 20 in flight]
    E --> F[Latency grows with queue depth]
    F --> G[Developers complain]

Little’s Law, the queueing-theory identity that governs the system, says:

• L = λ × W

where L is the average number of jobs in the system (running plus queued), λ is the arrival rate (jobs per unit time), and W is the average time a job spends in the system (waiting plus running).

A team that doubles the arrival rate without changing the runner pool doubles L and W. A team that halves the service time (faster jobs) halves L and W. A team that adds runners reduces λ from the perspective of the queue - same arrival rate, higher throughput, shorter wait - but does not reduce the arrival rate itself. The queue is a function of arrival rate, service time, and capacity; it is not a function of “how good the CI is”.

Runner concurrency versus job concurrency

The two numbers are not the same and they protect against different things:

• Runner concurrency is the maximum number of jobs that can run simultaneously on the runner pool. It is a property of the CI infrastructure.
• Job concurrency is the maximum number of deploys that can target the same resource simultaneously. It is a property of the target system.

Runner concurrency is limited by the number of runner processes. Job concurrency is limited by the target system’s ability to absorb concurrent deploys (state lock, API rate limit, conditional writes).

A team that increases runner concurrency to “speed up CI” may also be increasing job concurrency against a target that cannot absorb it. Twenty runners running twenty deploys at once is faster than ten runners running them in pairs, but only if the target can safely handle twenty concurrent deploys. For most production targets (Terraform state, Kubernetes cluster, shared database), the answer is no: the target serialises deploys anyway, and the runner concurrency limit above the target’s capacity just produces a queue at the target level instead of at the runner level.

The right discipline is to set runner concurrency no higher than the target’s safe concurrency. A target that can safely absorb three concurrent deploys should be paired with a runner pool of three (or fewer), not with a pool of ten. The extra runners do not speed up deploys; they queue at the target.

When throttling is the right answer

Throttling is the right answer when the constraint is the target, not the runner. The standard cases:

• State-protected deploys (Terraform, Helm, database migrations). The target serialises deploys through a lock; running more deploys in parallel only queues at the lock. Throttle the runner to one deploy per target, or use a concurrency group to make the queue explicit.
• API rate-limited targets. Cloud providers impose rate limits per account, per region, per API. A deploy pipeline that exceeds the rate limit produces 429 errors and retries. Throttle the runner to stay under the rate limit; the queue at the runner is better than the rate-limit errors at the target.
• Resource-constrained targets. A small Kubernetes cluster with limited CPU, a database with limited connections, a shared service with limited worker threads. Throttle the runner to stay under the resource constraint.

Throttling is the wrong answer when the constraint is the developer’s patience. A team that throttles because “developers are complaining about CI latency” is treating the queue depth as the problem when the actual problem is under-sized runners. The right fix for CI latency is more runners or faster jobs, not throttling.

Setting the right limit

The right concurrency limit is a function of three things:

1. Target concurrency capacity. How many concurrent deploys can the target safely absorb? For Terraform, the answer is one per state file (the lock serialises the rest). For a Kubernetes cluster with admission control, the answer is bounded by the cluster’s request rate. For a shared database, the answer is bounded by connection pool size.
2. Runner pool size. The number of runner processes available to run jobs. The limit should not exceed this number.
3. Job isolation. Whether jobs of the same workflow can safely run in parallel against the same target. If yes (each job targets a different resource), the limit can be higher. If no (each job targets the same resource), the limit should be one.

The limit is min(target_capacity, runner_pool_size, job_isolation). For a typical production deploy pipeline against a single Terraform state, the answer is one. For a test pipeline where each job runs an isolated integration test, the answer is runner_pool_size. For a hybrid where tests run in parallel and deploys serialise, the answer is the deploy limit for deploy jobs and the runner pool size for test jobs.

The cost of throttling

Throttling is not free. The cost is queue depth, which manifests as:

• Latency. A job that would take 6 minutes to run takes 6 minutes plus the time it spent in the queue. Queue time grows with arrival rate and shrinks with throughput. A team that triples the arrival rate without changing the runner pool triples the queue time.
• Queue depth. A queue that grows past a certain depth becomes its own problem: developers stop watching the pipeline, old jobs block new jobs, the queue becomes a backlog of “stale” jobs that may no longer be relevant.
• Developer friction. A queue that produces 30-minute build times produces developers who run fewer builds, who batch changes into larger PRs, and who find ways around the pipeline. The discipline of “every change goes through CI” erodes.

The right response to throttling cost is not to remove the throttle. It is to size the runner pool to the workload, to optimise jobs to reduce service time, and to accept that some throttling is the cost of safety.

Production discipline

1. Set runner concurrency from the target, not the runner. The throttle protects the target system. The runner pool size is an upper bound, not a target.
2. Use concurrency groups for serialised targets. A target that can absorb one deploy at a time (Terraform state, a shared database) should be paired with a concurrency group keyed on the target, not with a runner limit. The group makes the queue explicit and visible.
3. Monitor queue depth, not just runner utilisation. A runner pool at 100% utilisation is healthy; a runner pool at 100% utilisation with a queue of fifty jobs is a backlog. The metric is queue depth.
4. Optimise service time before adding runners. A 6-minute job that takes 3 minutes halves queue depth without adding a runner. The right order is: measure service time, optimise jobs, then add runners.

Cross-course references

• This course, Part LXII-01 (ConcurrentDeploy) covers the target-side failure modes that throttling exists to prevent.
• This course, Part LXII-02 (EnvLock) covers the concurrency group mechanism that makes throttling explicit.
• This course, Part LXII-05 (Serialize) covers when full serialisation is the right answer instead of partial throttling.

Quiz