Controlled Apply: Saved Plans, Approvals, and Concurrency

The apply is the only stage in the pipeline that mutates state. Everything before it is preparation; everything after it is bookkeeping. The job of the apply stage is to execute exactly what was reviewed, once, with the right credentials, in the right order, with the right output captured for the audit trail.

This lesson covers the three locks that protect an apply, how to ensure only one apply runs at a time, how to surface the apply output, what to do when the apply fails partway through, and why -target has no safe place in a production CI pipeline.

Three locks, three purposes

A Terraform apply in CI is held in place by three independent locks. Each protects a different race. Each can be the one that fails when things go wrong.

1. The state backend lock

The state backend (S3 and DynamoDB, GCS, Azure Storage, etc.) serialises operations against the same state file. When a runner calls terraform apply, the backend writes a lock record. When the apply finishes, the lock is released. A second runner that tries to operate against the same state gets Error acquiring the state lock.

acquire-state-lock    [DynamoDB row]
   |
   v
read state
   |
   v
plan (if not saved) / execute (if saved)
   |
   v
write state
   |
   v
release-state-lock

The state lock is a per-state lock, not a per-workspace or per-environment lock by itself. Two applies against different state files (different backends, different keys) do not contend. Two applies against the same state file do.

2. The saved-plan lock

The saved plan file is bound to the configuration and state at the moment of plan. If either changes, the saved plan is rejected at apply time:

Error: Saved plan is no longer up to date

The given plan is no longer up to date. The plan was created
against a state that has since been modified, so it cannot be
applied without potentially affecting the wrong resources.

The saved-plan lock is a semantic lock. It is not a mutex; nothing prevents a second runner from initiating an apply against the same state. The state backend lock does that. The saved-plan lock prevents the apply from executing a plan that no longer matches reality.

3. The pipeline concurrency lock

GitHub Actions, GitLab CI, Jenkins, and similar tools have a job-level concurrency concept. In GitHub Actions:

jobs:
  apply:
    concurrency: terraform-apply-${{ matrix.env }}

This lock is a per-runner lock. It prevents two applies from starting at all on the same environment, even before either reaches the state backend. The benefit is that only one runner is consuming the AWS STS session token; the losing runner waits or exits cleanly instead of failing with a state lock error.

The three locks together:

Concurrency lock    |    prevents two runners from starting
                    |
State backend lock  |    prevents two applies from mutating the same state
                    |
Saved-plan lock     |    prevents an apply from executing a stale plan

PR plan versus main apply

A Terraform pipeline has two apply paths: the PR-side plan (never applies) and the main-side apply (always applies the saved plan).

PR opened                           main updated
   |                                   |
   v                                   v
lint, validate, tflint, tfsec      lint, validate, tflint, tfsec
plan (with plan-only creds)        plan (with plan creds)
comment on PR                      upload plan artifact
                                   wait for approval
                                   apply (with apply creds)

The plan-side IAM role should be as narrow as the cloud provider allows. On AWS, use a role that can Describe*, List*, and read the resources it manages but cannot create, update, or delete them. The apply-side IAM role is wider because it has to do the work. The split is the production control.

A useful pattern for AWS: the plan role can run terraform plan against the real cloud (so the plan is accurate) because AWS supports dry-run IAM for most Describe* and List* calls. Where the cloud does not support dry-run, fall back to plan against a sanitised copy or against a null provider.

Surfacing apply output

The apply output is the only honest record of what the apply did. Three places need the output.

1. The job log

GitHub Actions captures stdout by default. The apply output appears in the job log automatically:

aws_instance.web: Creating...
aws_instance.web: Still creating... [10s elapsed]
aws_instance.web: Creation complete after 25s [id=i-0abc123def456789]
aws_s3_bucket.logs: Creating...
aws_s3_bucket.logs: Creation complete after 4s [id=runbook-logs]
aws_security_group.web: Creating...
aws_security_group.web: Creation complete after 8s [id=sg-0abc123def456789]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

A apply that succeeds without output is suspicious. Use -no-color to keep the log readable in CI. The flag does not change behaviour; it removes ANSI colour codes from the log.

2. The structured artifact

The JSON output is the machine-readable record:

terraform apply -json -input=false tfplan > apply.json 2>&1

Or use the TF_LOG environment variable for debug-level logging when an incident requires it:

TF_LOG=INFO terraform apply -input=false tfplan
# 2026-08-13T11:42:07.123Z [INFO]  provider: configuring client for aws provider
# 2026-08-13T11:42:08.456Z [INFO]  backend: applying plan
# 2026-08-13T11:42:09.789Z [INFO]  aws_instance.web: creating...

Capture the apply JSON as a job artifact with the same retention as the plan artifact.

3. The notification

The on-call engineer is paged on apply success and apply failure. A simple Slack notification:

- name: Notify on apply
  if: always()
  run: |
    status="${{ job.status }}"
    curl -fsSL -X POST "$SLACK_WEBHOOK" \
      --data-urlencode "payload={\"text\": \"Terraform apply on production: $status. Run ${{ github.run_id }}.\"}"

The notification is the production control for the post-apply monitoring. Without it, a silent apply that broke something at 03:00 is not visible until a customer notices.

Partial apply: when the apply fails mid-run

A apply that errors after some resources have been created is the worst-case production failure. The state file reflects a mix of new and old resources; the cloud reflects the same mix but possibly with provider-side inconsistencies.

aws_instance.web: Creation complete after 25s
aws_s3_bucket.logs: Creating...
aws_s3_bucket.logs: Error: AccessDenied: User ... is not authorised to perform s3:CreateBucket
aws_security_group.web: Still pending...

The state now records aws_instance.web as created. The aws_s3_bucket.logs resource is in an undefined state. The aws_security_group.web may or may not exist depending on where the apply aborted.

The recovery:

1. Stop the pipeline. Do not retry the job. Retrying against the same state with the same plan is unlikely to succeed; the failure was on the cloud side.
2. Read the state. terraform state list and terraform state show aws_s3_bucket.logs show what the state thinks exists.
3. Read the cloud. The AWS console, aws s3 ls, or the equivalent on the affected cloud shows what the cloud actually has.
4. Reconcile. Either:
   • Refresh the state (terraform apply -refresh-only) and then plan again. The plan will propose to create the missing resources.
   • Manually delete the partially-created resources from the cloud and run a fresh apply.
5. Investigate the root cause. The AccessDenied is a signal that the apply role lacks a permission, or that a service control policy blocked the call. Fix the policy before re-running.

The recovery runbook in the runbook collection covers the detailed steps.

-target in CI: why it has no safe place

-target is a flag that limits a plan or apply to specific resources. It exists to scope a fix to a single resource without re-planning the entire estate. In a local terminal, on a developer machine, with a clear scope, it is a useful debug tool.

In CI, it is the wrong tool. Three reasons:

1. The plan hides the dependency graph. The plan output shows only the targeted resources. The reviewer cannot see what other resources the apply will depend on or what side effects the apply will have. A -target apply can succeed while leaving the dependency graph in a state the reviewer never saw.

2. The saved plan is not the whole plan. If the configuration has resources that the target exclude, those resources still exist in the state. The saved plan binds to the state, not to the target. A -target saved plan applied in CI is a partial apply by design; see the partial-apply failure mode above.

3. It bypasses the pipeline’s production control. The -target flag was added to Terraform for ad-hoc operator fixes. In CI, the operator is the pipeline. The pipeline does not need -target because the pipeline plans the whole configuration.

If a change is so large that the full plan is too slow, the fix is to split the configuration into smaller state files, not to add -target to the CI apply.

A reference apply job

apply:
  needs: plan
  if: github.ref == 'refs/heads/main'
  runs-on: ubuntu-latest
  permissions:
    id-token: write
    contents: read
  # Serialise applies against the same state across all branches.
  concurrency: terraform-apply-${{ matrix.env }}
  strategy:
    fail-fast: false
    matrix:
      env: [staging, production]
  environment: apply-${{ matrix.env }}   # manual approval gate
  steps:
    - uses: actions/checkout@v4
    - uses: hashicorp/setup-terraform@v3
      with:
        terraform_version: 1.9.x
    - uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ secrets[format('AWS_APPLY_ROLE_{0}', matrix.env)] }}
        aws-region: eu-west-2
    - uses: actions/download-artifact@v4
      with:
        name: tfplan-${{ matrix.env }}
    # Re-init against the production backend; do NOT reuse the
    # init directory from the plan stage.
    - run: terraform init
    # The apply executes the saved plan; no re-computation.
    - run: terraform apply -input=false -no-color tfplan | tee apply.log
    # Capture the structured output for the audit trail.
    - run: terraform show -json tfplan > plan.json
    - uses: actions/upload-artifact@v4
      with:
        name: tfapply-${{ matrix.env }}
        path: |
          apply.log
          plan.json
    - name: Notify on apply
      if: always()
      run: |
        curl -fsSL -X POST "$SLACK_WEBHOOK" \
          --data-urlencode "payload={\"text\": \"Apply on ${{ matrix.env }}: ${{ job.status }}. Run ${{ github.run_id }}.\"}"

Three properties this job has:

1. One init per stage. The apply re-runs init against the production backend. Reusing the init directory from the plan stage risks a different backend or different provider versions.
2. The apply uses the saved plan. No re-computation. The apply is the plan that was reviewed.
3. The apply log is an artifact. The output survives the job and is queryable in the artifact store.

Validation commands

Confirm the apply stage is wired correctly:

# 1. Confirm the saved plan is current and the state is healthy.
terraform plan -detailed-exitcode
# exit 0 = no changes (steady state)
# exit 2 = drift or new plan

# 2. Confirm the lock can be acquired and released cleanly.
# The plan-side `init` already did this; the apply-side
# `init` does it again.

# 3. Confirm the apply output is captured.
terraform apply -input=false -no-color tfplan | tee apply.log
ls -la apply.log

A apply that succeeds without output, or that exits before flushing the log, is a CI bug, not a Terraform behaviour.

Production failure modes

1. Two applies raced for the state lock. The state backend rejected one with Error acquiring the state lock. The concurrency group was missing. The fix is concurrency: terraform-apply-${{ matrix.env }} on the apply job.

2. Apply succeeded but -target was used. The apply touched only the targeted resources. The dependency graph was not reviewed. A side-effect resource elsewhere in the state is now inconsistent. The fix is to remove -target and split the state instead.

3. Apply failed mid-run with AccessDenied. The apply role lacks a permission. The state is partially updated. The fix is to refresh the state, fix the IAM role, and re-apply. See the partial-apply section above.

4. Saved plan rejected at apply time. The state changed between the plan and the apply. Another runner wrote to the state, or a manual change drifted the resource. The fix is to re-plan, review the new plan, and apply the new saved plan.

5. Apply succeeded but the cloud drifted within seconds. A local-exec or null_resource made a side-effect call that did not match the resource state. The state says the resource exists in shape A; the cloud says shape B. The next plan will show a replace or a long diff. The fix is to remove local-exec from production state.

6. Apply output was not captured. The job ran; the log was lost when the runner was destroyed. The audit trail for the apply is gone. The fix is terraform apply ... | tee apply.log and uploading the log as a job artifact.

What comes next

The next lesson covers the plan artifact itself: how to serialise a plan to a file, how to ship the file from the plan stage to the apply stage, what the file contains, and why a plan file is itself a sensitive artefact.

Verification

Run the apply job against a throwaway AWS account. Confirm the job waits for approval, runs the apply with the saved plan, captures the output to an artifact, and notifies the Slack channel. Then run a second apply job against the same environment from a different branch and confirm the concurrency group serialises them. Destroy the test resources when finished.