The concurrency discipline — when to allow concurrency, when to prevent it

A team has a hundred microservices, three production accounts, two Terraform state backends, four Kubernetes clusters, and a dozen GitHub Actions workflows. Each pipeline has its own concurrency setting - some allow concurrency, some throttle, some serialise. The settings were chosen by different engineers at different times for different reasons. The result is a system that runs mostly safely but occasionally fails in surprising ways: a deploy that corrupts state because the concurrency setting was wrong, a queue that grows because the throttle was too tight, a deploy that races because no one added a lock. This lesson is the framework that produces consistent settings.

The four questions

Every deploy in the system should answer four questions before the team decides its concurrency policy. The questions are:

1. What is the target resource shape? Is the target a single mutable resource, a small set of resources, or a large set of independent resources? A single state file is one resource. A hundred microservices with separate state files are a hundred resources. The shape determines how much concurrency the target can safely absorb.
2. What lock does the target provide? Does the target serialise writes through a built-in lock (Terraform state lock, Kubernetes ResourceVersion, S3 conditional write, DynamoDB conditional update)? Or does the target accept writes without coordination? The lock determines whether the target serialises itself or whether the runner must serialise for it.
3. What is the recovery cost if concurrency corrupts state? If two concurrent deploys corrupt state, how long does it take to recover? A corrupted Terraform state with S3 versioning is recoverable in minutes. A corrupted IAM policy with no version history is recoverable in hours. The recovery cost determines how much concurrency risk is acceptable.
4. What is the blast radius of a bad concurrent deploy? If a bad deploy ships because concurrency produced an unsafe state, how many users are affected? A misconfigured S3 bucket is a small blast radius. A misconfigured IAM role with broad permissions is a large blast radius. The blast radius determines how much concurrency risk is acceptable.

The answers produce one of three policies:

• Allow concurrency. The target is a large set of independent resources, the target provides a lock, the recovery cost is low, and the blast radius is small. Run as many deploys as the runner pool can support.
• Throttle concurrency. The target has a limited capacity (rate limit, connection pool, API quota), the lock allows concurrency but caps it. Run up to the cap, no more.
• Prevent concurrency. The target is a single mutable resource, the recovery cost is high, or the blast radius is large. Run one deploy at a time.

flowchart LR
    A[Target shape] --> B{Single resource?}
    B -->|yes| C[Prevent]
    B -->|no| D{Lock available?}
    D -->|no| E[Throttle or prevent]
    D -->|yes| F{Capacity limit?}
    F -->|yes| G[Throttle to capacity]
    F -->|no| H{Blast radius small?}
    H -->|yes| I[Allow]
    H -->|no| J[Throttle or prevent]

Decision policy examples

The framework applied to common cases:

• Production Terraform apply to a single state. Target shape: single resource (the state). Lock: state lock with DynamoDB. Recovery cost: medium (S3 versioning helps). Blast radius: large (production). Policy: prevent concurrency. One deploy at a time, FIFO queue.
• Helm release upgrade to a single release name. Target shape: single resource (the release). Lock: Kubernetes ResourceVersion. Recovery cost: medium (helm rollback). Blast radius: medium (the release’s workload). Policy: prevent concurrency. One upgrade at a time.
• Database migration to a shared schema. Target shape: single resource (the schema_migrations table). Lock: migration tool’s advisory lock. Recovery cost: high (data changes). Blast radius: large. Policy: prevent concurrency. One migration at a time.
• Ansible playbook against a small fleet (10 nodes). Target shape: small set of independent resources (the nodes). Lock: SSH serial strategy or forks setting. Recovery cost: low (re-run playbook). Blast radius: small (one node at a time). Policy: throttle to the fleet’s safe parallelism (usually 5-10 forks).
• Container image build with buildx cache. Target shape: independent resources (per-commit cache layers). Lock: BuildKit’s content-addressable cache. Recovery cost: low (rebuild). Blast radius: small. Policy: allow concurrency up to the runner pool size.
• GitOps apply to a Kubernetes cluster with many CRDs. Target shape: large set of independent resources (one CR per resource). Lock: server-side apply with fieldManager. Recovery cost: medium (re-apply). Blast radius: variable. Policy: throttle to the cluster’s admission-controller rate limit.

The framework is consistent: when in doubt, prevent is the safe answer. A team that prevents concurrency everywhere loses throughput but gains safety. A team that allows concurrency everywhere gains throughput but loses the ability to reason about state. The framework produces the right answer for each target; the discipline is to apply the framework to every target.

Auditing an existing pipeline

The audit is to take every deploy pipeline and ask the four questions. For each pipeline, the answer is one of: allow, throttle, prevent. The existing configuration is then compared to the policy. Where the configuration matches the policy, no change is needed. Where the configuration is looser than the policy, the change is to tighten it. Where the configuration is tighter than the policy, the change is optional (the cost is throughput, not safety).

The audit produces a list of structural changes:

• A pipeline that allows concurrency against a target that needs prevention: add a concurrency group, add a state lock, or both.
• A pipeline that prevents concurrency against a target that allows it: relax the group (optional), or document why the conservative policy is in place.
• A pipeline that throttles above the target’s capacity: lower the throttle to the target’s capacity.
• A pipeline that throttles below the target’s capacity: raise the throttle (optional), or document why.

The audit is structural: the changes are to the pipeline configuration, not to the runbook. A runbook that says “engineers should manually serialise deploys by waiting for the previous one to finish” is a procedural workaround for a missing structural control. The structural control is a concurrency group; the workaround is the bug.

The four invariants

The discipline produces four invariants that should hold for every deploy pipeline:

1. The target is the source of truth for concurrency. The runner concurrency limit is at most the target’s safe concurrency; never above.
2. Every shared target has a lock. The state lock, the Kubernetes ResourceVersion, the database advisory lock, the S3 conditional write. The lock is what serialises concurrent writers; without it, the runner is the only protection.
3. The queue is visible. The developer can see the queue depth in the CI dashboard. A queue that is invisible is a queue that is misunderstood.
4. The recovery path is known. If state is corrupted, the recovery path is in the runbook and tested. A team that does not know how to recover from concurrent state corruption is a team that has not thought about the cost of being wrong.

A pipeline that satisfies all four invariants is a pipeline that has reasoned about concurrency. A pipeline that violates any of them is a pipeline that has not.

Production discipline

1. Apply the four questions to every target. The target shape, the lock, the recovery cost, the blast radius. The answers produce the policy.
2. Default to prevent concurrency. The safe default. The relaxation to throttle or allow is an explicit decision with documented justification.
3. Pair the runner control with the target control. A concurrency group on the runner, a state lock on the target. Both together; never only one.
4. Make the queue visible. The CI dashboard shows queue depth; the developer understands the latency; the policy is observable.
5. Audit the pipeline against the framework. A periodic review (quarterly, after major changes) that re-applies the four questions and adjusts the policy. Concurrency settings drift; the audit catches the drift.

Cross-course references

• This course, Part LXII-01 (ConcurrentDeploy) covers the failure modes that the discipline exists to prevent.
• This course, Part LXII-02 (EnvLock), Part LXII-03 (StateLock), Part LXII-04 (Throttle), Part LXII-05 (Serialize) cover the individual mechanisms.
• This course, Part LXIII (Preflight) (next part) covers preflight validation as the discipline that catches configuration errors before they reach the deploy.

Quiz