Queueing and serialization — when to serialise a deploy and the cost it carries

A team has a single production Terraform state. Every deploy acquires the state lock. Every apply holds the lock for ten minutes. Five deploys arrive in an hour. The first runs, the next four wait. The fourth deploy lands fifty minutes after it arrived. The developer who pushed the fourth commit is frustrated: “why is CI so slow today?”. The answer is that the deploy is serialised, the queue is the cost, and the cost is correct: this is the only safe way to deploy to a single state. This lesson is about when serialisation is the right answer and how to make the queue visible.

When to serialise fully

Full serialisation is the right answer when the target has a single mutable resource that all deploys must touch, and that resource cannot safely absorb concurrent writers. The canonical cases:

• Single Terraform state. One state file, multiple contributors, no safe way to split the state further without losing the unit of deployment. Every deploy must acquire the state lock; only one can hold it; the rest queue.
• Single Helm release. One Helm release name in a Kubernetes cluster. The release is a single resource in the cluster’s etcd; concurrent upgrades race on the same resource version. Only one upgrade at a time is safe.
• Single shared database schema. A migration tool like Flyway or Liquibase that owns a schema_migrations table. The table is the serialisation point; only one migration can run at a time.
• Single critical infrastructure resource. A specific Route 53 hosted zone, a specific S3 bucket, a specific IAM role that every deploy modifies. Concurrent writes are unsafe.

In each case, the choice is not “serialise or not”. The target serialises by its nature; the choice is whether the serialisation is at the runner level (concurrency group), at the target level (state lock), or both. A team that does not serialise explicitly is relying on the target to serialise, and the target’s serialisation is invisible until it fails.

flowchart LR
    A[Deploy 1] -->|lock| B[Running]
    C[Deploy 2] -->|wait| D[Queue position 1]
    E[Deploy 3] -->|wait| F[Queue position 2]
    G[Deploy 4] -->|wait| H[Queue position 3]
    B -->|lock released| I[Deploy 2 starts]
    I --> J[Deploy 3 starts]
    J --> K[Deploy 4 starts]

The cost of serialisation

The cost of serialisation is queue depth, which manifests as latency. For a serialised pipeline:

• Queue depth is the number of deploys waiting. With arrival rate λ (deploys per minute) and service time S (minutes per deploy), the steady-state queue depth is approximately λ × S / (1 - λ × S) for λ × S < 1. At 80% utilisation, the queue depth is 4. At 95% utilisation, the queue depth is 19.
• Latency is the time from “deploy triggered” to “deploy finished”. The latency for a deploy at queue position N is approximately (N + 1) × S. The first deploy waits 0 minutes; the second waits S minutes; the twentieth waits 20 × S minutes.
• Developer experience is the perception that “CI is slow today”. A deploy that took 30 minutes yesterday might take 90 minutes today because the queue is deeper. The developer does not know the queue depth; they only see the total time.

The cost is real but it is the cost of safety. A deploy pipeline that runs four concurrent deploys to a single state and avoids the queue is a pipeline that corrupts state under load. The choice is not “fast or slow”; it is “slow or unsafe”.

Queue disciplines: FIFO, LIFO, priority

When a serialised pipeline has a queue, the queue discipline determines which deploy runs next. The three common disciplines:

• FIFO (first-in, first-out). The deploy that arrived first runs first. This is the default for most queue implementations and the discipline that matches developer expectations: “I pushed first, my deploy should land first”. A concurrency group with cancel-in-progress: false produces FIFO behaviour.
• LIFO (last-in, first-out). The deploy that arrived last runs first. This is unusual in deployment pipelines; it appears in some push-based systems where the most recent commit is the most relevant. A concurrency group with cancel-in-progress: true produces LIFO behaviour with cancellation.
• Priority. Deploys with higher priority jump the queue. A hotfix deploy might be tagged with a priority label that moves it to the front of the queue. Priority queues add complexity (what if two hotfixes arrive at once?) but solve real problems (a hotfix should not wait behind a routine deploy).

The right discipline depends on the team’s deployment policy. FIFO is the safe default; LIFO with cancellation is correct when “latest commit is the right answer” is the explicit policy; priority is correct when some deploys are more urgent than others.

Serialisation by queue versus serialisation by lock

Two layers can serialise a deploy pipeline:

• Serialisation by queue (the concurrency group) is the runner-level mutex. It serialises the workflow runs that share the group key. The lock is held by the runner provider, not by the target.
• Serialisation by lock (the state lock) is the target-level mutex. It serialises the operations against the target resource. The lock is held by the target system, not by the runner.

Both layers are usually present for safety: the queue makes the serialisation visible at the runner (the developer sees the queued status, the CI dashboard shows the queue depth), and the lock prevents out-of-band writes from bypassing the runner (a developer’s laptop, a cron job, a runbook command).

A team that uses only the concurrency group has not protected against out-of-band writes; the lock is the target’s protection. A team that uses only the state lock has a queue at the target that is invisible to the runner metrics; the concurrency group makes the queue visible. Both together give a serialised pipeline where the queue is explicit, the lock is enforced, and the developer experience is predictable.

Production discipline

1. Serialise by target, not by habit. A serialised deploy pipeline is the right answer when the target has a single mutable resource that all deploys must touch. It is the wrong answer when the deploys target independent resources and the serialisation is just tradition.
2. Pair the queue with the lock. The concurrency group serialises the runner; the state lock serialises the target. Both together; never only one.
3. Use FIFO by default. First-in, first-out is the discipline that matches developer expectations. LIFO and priority are correct in specific cases but require explicit policy.
4. Make the queue visible. A queue that the developer can see in the CI dashboard is a queue that the developer understands. A queue that surfaces only as “build slow today” is a queue that produces confusion.
5. Serialise the deploy, not the test. The test pipeline can run in parallel; the deploy pipeline serialises. The cost of serialisation is on the deploy path, not the test path.

Cross-course references

• This course, Part LXII-02 (EnvLock) covers the concurrency group mechanism that produces the queue.
• This course, Part LXII-03 (StateLock) covers the target lock that complements the runner-level queue.
• This course, Part LXII-06 (Discipline) synthesises the decision framework: when to serialise, when to allow concurrency, when to throttle.

Quiz