Environment locking and mutexes — concurrency groups, cancel-in-progress, and the implementation

A workflow that deploys to production runs three times in one minute because three pull requests merge in quick succession. Three runners spin up. Three deploys begin at the same time. The team wants exactly one deploy to production at a time. The mechanism that makes that guarantee is a concurrency group: a named mutex that the runner provider holds on behalf of the workflow, releases when the workflow finishes, and refuses to give to a second workflow while it is held. This lesson is about what that mutex does and what it does not do.

What a concurrency group is

A concurrency group is a string key that the runner provider associates with at most one in-flight workflow run. When a workflow declares concurrency: group: prod-${ github.ref }, the runner checks the group before starting the workflow. If a previous workflow with the same group is still running, the new workflow waits - or, depending on the flag, cancels the previous one.

name: deploy-prod
on:
  push:
    branches: [main]
concurrency:
  group: prod-${ github.ref }
  cancel-in-progress: false
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: terraform apply -lock-timeout=300s -auto-approve

The group key is a string. ${ github.ref } is the standard choice because it produces a different key for each branch and each tag: prod-refs/heads/main for the trunk, prod-refs/tags/v1.2.3 for a tag, prod-refs/heads/feature/x for a branch. A push to main produces the group prod-refs/heads/main. A second push to main while the first is still running is queued. A push to a feature branch produces prod-refs/heads/feature/x and runs in parallel with the main-branch deploy because the keys differ.

cancel-in-progress: false versus true

The cancel-in-progress flag has two values and they mean different things:

• cancel-in-progress: false (the default in many templates). The first workflow keeps running; the second waits. When the first finishes, the second starts. This is the first-write-wins semantic: the deploy that started first is the deploy that lands.
• cancel-in-progress: true. The first workflow is cancelled when the second starts. The second runs alone. This is the last-write-wins semantic: the deploy that started last is the deploy that lands; the earlier one is killed mid-flight.

flowchart LR
    A[Workflow run 1] -->|group free| B[Running]
    B -->|run 2 starts| C{Group held}
    C -->|cancel-in-progress: false| D[Run 2 queued]
    D -->|run 1 finishes| E[Run 2 starts]
    C -->|cancel-in-progress: true| F[Run 1 cancelled]
    F --> G[Run 2 starts]

The choice between them is a deployment policy decision, not a technical one.

• false is correct when the first deploy should be allowed to complete because partial state is worse than late state. A database migration that is half-done is worse than a database migration that has not started.
• true is correct when the most recent commit is the one that should land and earlier deploys are obsolete by the time they would finish. A doc-only deploy is safe to cancel; a deploy that creates a new S3 bucket is not.

Scoping: workflow, job, or environment

The concurrency block can appear at the workflow level or at the job level. The scoping determines what the mutex covers:

• Workflow-level concurrency holds the group for the entire workflow run, including all jobs. A workflow with a plan job and an apply job holds the group while both run; a second workflow waits until the first finishes both.
• Job-level concurrency holds the group for one job only. A plan job can run in parallel with another plan job; an apply job holds a separate group that serialises only the apply.

For environment locking, the standard pattern is workflow-level concurrency scoped to a GitHub Actions environment. The environment provides the deployment gate (required reviewers, wait timer, branch restrictions); the concurrency group provides the deployment mutex. The two together give a workflow that deploys to one environment at a time, only from the right branches, with the right reviewers.

concurrency:
  group: prod-${ github.environment }
  cancel-in-progress: false

${ github.environment } resolves to the environment name from the job’s environment: field. A deploy to production produces the group prod-production; a deploy to staging produces prod-staging. They do not block each other. Two deploys to production produce the same group and serialise.

What concurrency groups do not do

A concurrency group is a runner-level mutex. It serialises workflow runs that share the group key. It does not serialise the target system. If two workflows use the same group, the second waits. If two workflows use different groups - different branches, different environments, different workflow files - they can run at the same time even if they mutate the same target resource.

The group does not lock Terraform state. It does not hold an S3 object. It does not take a database advisory lock. It is a string key in the runner provider’s memory (or its persistent store) that says “this group is currently in use”. That is enough to serialise runners; it is not enough to serialise two applies that bypass the runner entirely - a developer’s laptop, a cron job, an out-of-band runbook command. For target-level serialisation, you need a state lock, an S3 conditional write, a database advisory lock, or whatever the target resource provides.

Production discipline

1. Scope the group to the environment, not the workflow. A group keyed on ${ github.environment } serialises deploys to the same environment and allows deploys to different environments to run in parallel. A group keyed on ${ github.workflow } serialises runs of the same workflow but not deploys to the same environment from a different workflow file.
2. Default to cancel-in-progress: false. First-write-wins is the safe default; the deploy that started first is the deploy that should land. Use true only when the deploy is idempotent and the most-recent-commit-is-the-right-answer policy is explicit.
3. Pair the group with a target lock. The concurrency group serialises the runner; the state lock serialises the target. One without the other is incomplete.
4. Treat manual cancellation as a recovery path, not a control. Cancelling a workflow mid-deploy is what cancel-in-progress: true does; doing it manually is what the on-call engineer does when a deploy is stuck. Both leave partial state behind; both require the same discipline to recover.

Cross-course references

• This course, Part LXII-01 (ConcurrentDeploy) covers the failure modes that concurrency groups exist to prevent.
• This course, Part LXII-03 (StateLock) covers the target-level lock that the concurrency group complements.
• Terraform for Production Sysadmins - Part X (StateBackends) covers the DynamoDB lock backend and the -lock-timeout flag.

Quiz