Concurrent deployments and isolation — what "concurrent" means and the failure modes

Two pull requests merge into main within thirty seconds of each other. The CI pipeline triggers twice. Two runners spin up. Two deploys begin at the same time, each one confident it is the only change in flight. The dashboard shows two green checkmarks. The target system now has two writers, each holding a partial view of the world. The question this lesson answers is: which resource, in your target system, is being written by both deploys, and what happens when both succeed?

What “concurrent” actually means

“Concurrent” in a deployment context is not the same as “parallel in CI”. CI parallelism is about runners and stages. Deployment concurrency is about two changes whose effective intervals overlap in the target system. The effective interval starts when the first mutating call reaches the system and ends when the last mutating call returns. Two deploys are concurrent when their intervals overlap, regardless of whether they run on the same runner, different runners, or different CI providers.

flowchart LR
    A[Deploy A starts] --> B[A applies change 1]
    B --> C[A applies change 2]
    C --> D[A finishes]
    E[Deploy B starts] --> F[B applies change 1]
    F --> G[B applies change 2]
    G --> H[B finishes]
    subgraph overlap
    B
    F
    end
    D --> I[Two non-overlapping deploys - safe]

The trap is to look at the runner. Two runners, two pipelines, two green checkmarks looks concurrent. But if the deploys serialise on a lock - a Terraform state lock, a database row lock, a Kubernetes ResourceVersion - the intervals do not overlap. The runner is parallel; the target is serial. That is safe. The opposite is also true: a single runner can hold two deploys that overlap on the target, which is unsafe. The runner is serial; the target is concurrent.

The four failure modes

Concurrency in the target system produces four failure modes. Each is a different shape of corruption, and each has a different detection signature.

• Interleaved writes. Two deploys write to the same field of the same resource in different orders. The final state depends on which write landed last. The audit trail shows both writes; the production state matches neither intent.
• State corruption. Two deploys read the same record, compute different next states from it, and write both. The record contains values that no single deploy ever produced. This is the classic lost-update problem, the same shape as two bank tellers reading a balance and writing withdrawals that assume the original balance.
• Lost updates. Two deploys write to the same resource; only one write survives. The losing deploy reports success because its write returned 200 OK; the survivor’s write overwrote it. There is no error and no warning; the system is in a state one of the deploys did not produce.
• Split-brain. Two deploys each provision a separate coordination resource - two load balancer target groups, two DynamoDB lock tables, two IAM role versions - and the system routes traffic to both. Both halves of the system believe they are the authoritative one; neither can reconcile the other.

Safe concurrency and unsafe concurrency

Not all resources are unsafe under concurrency. The test is whether the resource serialises its own writes. A resource that exposes a compare-and-set operation, an incrementing version, or a queue with a single consumer is safe under concurrent writes - the resource does the serialisation. A resource that exposes only read-modify-write, or that accepts “set this field to this value” without checking the previous value, is unsafe.

Examples of resources that are usually safe under concurrent writes: S3 with versioning and conditional writes, DynamoDB with conditional updates, Kubernetes resources with a resourceVersion field that the API server checks, Git commits with content-addressed hashing.

Examples of resources that are unsafe under concurrent writes: a Terraform state file without a lock backend, an RDS database without row-level locking, a configuration file on a shared filesystem, an IAM policy written by two parallel aws iam create-policy-version calls, an Ansible inventory that two playbooks mutate concurrently.

Reading the target, not the runner

The discipline is to look at the target system before deciding whether two deploys are safe to run concurrently. A team that serialises pipelines but parallelises the underlying API calls - two runners, two apply steps, two state files - has serialised the wrong thing. A team that parallelises pipelines but uses a state lock - two runners, one state file at a time - has serialised the right thing.

Production discipline

1. Classify every deploy by target resource. Before allowing concurrent deploys, list the resources each deploy mutates and ask: “what happens if two of these run at the same time against the same resource?”. If the answer is “the second overwrites the first”, the deploy is unsafe under concurrency.
2. Default to serial for shared state. A Terraform apply, an Ansible playbook against a shared inventory, a Helm release upgrade, and a kubectl rollout are all unsafe under concurrency by default. Serialise them until you have a specific reason not to.
3. Measure target concurrency, not runner concurrency. A dashboard that shows ten runners running is not a dashboard that shows ten safe deploys. The metric that matters is the number of writes in flight against each shared resource.
4. Test the concurrent case. A deploy that is safe in single runs is not automatically safe in concurrent runs. Run two deploys against a staging environment at the same time and observe the result.

Cross-course references

• This course, Part LXI-04 (PartialDeploy) covers resumable deploys - the prerequisite topic that this lesson builds on.
• This course, Part LXII-02 (EnvLock) covers the concurrency group mechanism that prevents two deploys from running at the same time against the same environment.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover Terraform state and the lock backend that prevents two applies from corrupting state.

Quiz