Jobs and concurrency — concurrency groups, cancel-in-progress, and queue limits

A CI system has a finite runner pool. Pull requests, pushes, scheduled runs, and manual triggers all want runners. The scheduler must decide: which runs go now, which runs wait, and which runs are superseded and can be cancelled. The wrong default - “run everything, ignore conflicts, never cancel” - wastes runner minutes, corrupts shared state (Terraform state, k8s deployments, package registries), and produces noisy logs from runs that no longer matter.

Concurrency versus parallelism

flowchart LR
    subgraph PAR["Parallelism (should run concurrently)"]
        J1["lint"]
        J2["plan"]
        J3["scan"]
    end
    subgraph CON["Concurrency (must not run together)"]
        K1["terraform apply production"]
        K2["terraform apply production"]
    end

• Parallelism is the property of jobs that should run concurrently. lint, plan, and scan against the same commit do not conflict; running them in parallel reduces wall-clock time.
• Concurrency is the property of jobs that must not run concurrently. Two terraform apply jobs against the same state file, two kubectl rollout jobs against the same deployment, two ansible-playbook jobs against the same inventory will all corrupt shared state.

The pipeline design is to maximise the first and strictly limit the second.

Concurrency groups

A concurrency group is a named bucket. At most one run in the group is active at a time. New runs either queue (wait for the active run to finish) or cancel the active run and start:

concurrency:
  group: production-deploy-${ github.ref }
  cancel-in-progress: false   # wait, do not cancel

concurrency:
  group: ci-feedback-${ github.ref }
  cancel-in-progress: true    # cancel superseded runs

Two patterns:

• Queue with cancel-in-progress: false. Two deploys against production must not race; the second waits for the first. Correct for any job that mutates shared state.
• Cancel with cancel-in-progress: true. CI feedback on a pull request is updated by every push. The older run is stale; cancelling saves runner minutes. Correct for feedback jobs whose output is superseded.

The three scheduling outcomes

flowchart TB
    NEW["New run enters\nconcurrency group"] --> Q{"Group\nactive?"}
    Q -->|"no"| GO["Run starts"]
    Q -->|"yes, cancel-in-progress: true"| CANCEL["Active run cancelled,\nnew run starts"]
    Q -->|"yes, cancel-in-progress: false"| WAIT["New run queues"]

1. Run starts. The group is empty, the runner pool has capacity, the run executes normally.
2. Active run cancelled, new run starts. Used for CI-feedback on pull requests.
3. New run queues, waits for active to finish. Used for production deploys. The wait can be long; this is a feature.

The choice between cancel and wait is the production decision. cancel-in-progress: true on production deploys will, one day, cancel a deploy mid-apply and leave the target system in a half-applied state.

Queue limits and runner pool exhaustion

flowchart TB
    POOL["Runner pool\n(capacity: N)"]
    POOL -->|"all N busy"| BACKLOG["Backlog\n(queued runs)"]
    BACKLOG -->|"wait time exceeds limit"| FAIL["Run fails to start"]

When the runner pool saturates:

• Backlog grows. New runs queue; wait time grows linearly.
• Queue timeout. Most CI systems have a maximum queue time (often 60 minutes for hosted runners). Runs that wait longer fail to start. The PR shows a “could not start” status.

The failure mode is silent. A pipeline that worked on Tuesday fails on Wednesday because the runner pool is undersized for the new load. Size the runner pool for peak load, not average.

# Inside the runner shell, confirm the runner identity
env | grep -E '^RUNNER_'
# prints RUNNER_OS, RUNNER_ARCH, RUNNER_NAME, RUNNER_TOOL_CACHE

Designing the concurrency strategy

flowchart TB
    subgraph FB["PR feedback (cancel-in-progress: true)"]
        FB1["lint + plan + scan on push"]
        FB2["lint + plan + scan on next push (cancels FB1)"]
    end
    subgraph DP["Production deploy (cancel-in-progress: false)"]
        DP1["apply on push to main"]
        DP2["apply on next push to main (waits for DP1)"]
    end

• PR feedback group. cancel-in-progress: true. Keyed on the PR ref. New pushes cancel the previous push’s feedback.
• Production deploy group. cancel-in-progress: false. Keyed on the environment or branch. Deploys queue. Shared state is protected.

The two groups do not interact. A PR feedback run is never queued behind a production deploy, and a production deploy is never cancelled by a PR push.

# Combined concurrency strategy
on:
  pull_request:
  push:
    branches: [main]
concurrency:
  group: ${ github.workflow }-${ github.ref }
  cancel-in-progress: ${ github.event_name == 'pull_request' }

Conditional cancel-in-progress: cancel on pull request, wait on push to main.

Production discipline

1. Use a concurrency group for every workflow that mutates shared state. Default to cancel-in-progress: false.
2. Use a separate concurrency group for PR feedback. Keyed on the PR ref, not on the workflow.
3. Size the runner pool for peak load. Backlog grows during peak; queue timeout fires during sustained peak.
4. Monitor queue depth, not just runner utilisation. A pool at 100% utilisation with no backlog is healthy. A pool at 80% utilisation with a growing backlog is about to fail.
5. Key concurrency groups narrowly. group: production serialises everything. group: production-${ github.ref } is appropriate for branch-scoped deploys.

Cross-course references

• Linux for Production Sysadmins - Part XXXIII (NetHard) covers the same pattern for apt mirrors.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) applies the pattern to AWX job templates.
• Terraform for Production Sysadmins - Parts IX-XII (State) apply the strictest version: Terraform Cloud’s queueing is mandatory for any operation against the same state file.

Quiz