CI/CD Observability and Failures

A Terraform CI pipeline that runs without observability is a pipeline that fails silently. The first indication of a broken plan, a stuck lock, or a drifted resource is the incident, not the alert. The pipeline does not have a UI; the only way to know it is healthy is to measure it.

This lesson covers what to measure in a Terraform CI pipeline, how to push the measurements to Prometheus or OpenTelemetry, what to alert on, and the operational cost of silent apply drift.

What observability means here

Observability for a Terraform CI pipeline is not the same as observability for an application. The pipeline is not a long-running service; it is a sequence of jobs. The metrics are about the pipeline itself (plan duration, apply duration, failure rate) and about the state of the estate (drift, lock contention, plan/apply success).

The three categories that matter:

1. Pipeline performance. Plan and apply duration. Plan output size. Lock acquisition time. Init duration.
2. Pipeline correctness. Plan success rate. Apply success rate. Saved-plan rejections (state drift between plan and apply). Drift detection failures.
3. Estate state. Resources in state. Resources out of state (drift). State lock contention. Provider errors.

For each category, the metric, the alert, and the runbook should exist together. A metric without an alert is a number on a dashboard nobody reads. An alert without a runbook is a page that nobody knows how to handle.

The metrics that matter

The minimum viable metric set for a Terraform CI pipeline:

terraform_plan_duration_seconds        # histogram, labels: env, ref
terraform_apply_duration_seconds       # histogram, labels: env, ref
terraform_plan_total                   # counter,   labels: env, status
terraform_apply_total                  # counter,   labels: env, status
terraform_state_lock_wait_seconds      # histogram, labels: env
terraform_state_lock_contention_total  # counter,   labels: env
terraform_drift_resources_total        # gauge,     labels: env, resource_type
terraform_init_duration_seconds        # histogram, labels: env
terraform_provider_errors_total        # counter,   labels: env, provider

Two of these need explanation.

terraform_plan_duration_seconds

The wall-clock time from terraform plan start to plan end. The histogram is bucketed by environment (staging, production, etc.) and by ref (branch or PR). The 50th, 90th, and 99th percentiles are the operational signals.

# 90th percentile plan duration in staging
histogram_quantile(0.90,
  sum by (le) (
    rate(terraform_plan_duration_seconds_bucket{env="staging"}[5m])
  )
)

A 90th percentile that is growing week-over-week is the signal that the configuration is getting too large for the current state partitioning. The fix is to split the state.

terraform_drift_resources_total

The number of resources that drifted since the last apply. This is a gauge, not a counter, because the value can go up and down as drift is detected and remediated.

# Current drift count by resource type in production
sum by (resource_type) (
  terraform_drift_resources_total{env="production"}
)

A gauge that trends upward over weeks is the signal that the estate is being modified outside Terraform. The fix is to audit the manual change path; the production control is to prevent the manual change in the first place (SCP, IAM boundary, automation policy).

Pushing metrics to Prometheus

The standard pattern is to instrument the CI job to emit metrics at the end of each run. The GitHub Actions example:

- name: Emit metrics
  if: always()
  run: |
    cat <<EOF > metrics.txt
    # HELP terraform_plan_total Total number of terraform plan runs
    # TYPE terraform_plan_total counter
    terraform_plan_total{env="${{ matrix.env }}",status="${{ job.status }}"} 1
    # HELP terraform_plan_duration_seconds Duration of terraform plan
    # TYPE terraform_plan_duration_seconds gauge
    terraform_plan_duration_seconds{env="${{ matrix.env }}"} $PLAN_DURATION
    EOF
    curl -fsSL -X POST "$PUSHGATEWAY_URL" \
      --data-binary @metrics.txt

The Prometheus Pushgateway is the right endpoint for short- lived batch jobs. The runner pushes the metrics at the end of the job; the Pushgateway stores them; Prometheus scrapes the Pushgateway on its regular interval.

A more durable pattern uses the OpenTelemetry Collector. The runner sends OTLP metrics; the Collector routes them to Prometheus, to a vendor, or to a long-term store:

- name: Send OTel metrics
  if: always()
  uses: axel-op/otel-collector-action@v1
  with:
    otlp-endpoint: ${{ secrets.OTEL_ENDPOINT }}
    metrics: |
      terraform_plan_total{env="${{ matrix.env }}",status="${{ job.status }}"} 1
      terraform_plan_duration_seconds{env="${{ matrix.env }}"} $PLAN_DURATION

The OTel path is the modern default. The Pushgateway path is the legacy default. Both work; pick one.

Alerts that catch real failures

A metric without an alert is a number nobody reads. Five alerts cover the most common production failures.

1. Plan duration regression

# 90th percentile plan duration in production exceeds 10 minutes.
histogram_quantile(0.90,
  sum by (le) (rate(terraform_plan_duration_seconds_bucket{env="production"}[1h]))
) > 600

A 10-minute plan is the boundary at which the PR-side feedback loop breaks. The fix is to split the configuration before the plan gets worse.

2. Lock acquisition failure

# Lock contention in the last hour.
increase(terraform_state_lock_contention_total[1h]) > 0

A lock acquisition failure is a state race. The fix is the concurrency group on the apply job; if the metric is non-zero after that, the state backend is the bottleneck.

3. Drift growth

# Drift count in production increased by more than 10 in the last hour.
delta(terraform_drift_resources_total{env="production"}[1h]) > 10

A 10-resource drift event is a bulk manual change. The fix is to find who made the change and prevent the next one.

4. Apply failure rate

# Apply failure rate over the last 24 hours is above 10%.
sum(rate(terraform_apply_total{status="failure"}[24h])) /
sum(rate(terraform_apply_total[24h])) > 0.1

A 10% apply failure rate is a signal that the configuration or the credentials are wrong. The fix is to investigate the most recent failures; the runbook for partial apply is the starting point.

5. Saved-plan rejection

# Saved plan was rejected because the state changed between plan and apply.
increase(terraform_saved_plan_rejections_total[1h]) > 0

A saved-plan rejection is the saved-plan lock firing. The fix is to re-plan and re-apply; the alert exists to catch the case where the rejection is the new normal, not a one-off.

Drift detection as an observability control

Drift detection is the observability control for the estate. The apply pipeline catches configuration drift (the configuration changed from the last plan). Drift detection catches operational drift (the cloud changed from the last apply).

name: terraform-drift

on:
  schedule:
    - cron: '0 */6 * * *'   # every 6 hours

jobs:
  drift:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    strategy:
      matrix:
        env: [staging, production]
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets[format('AWS_DRIFT_ROLE_{0}', matrix.env)] }}
          aws-region: eu-west-2
      - run: terraform init
      # plan with -detailed-exitcode; exit 2 means drift detected.
      - run: terraform plan -detailed-exitcode -input=false
        id: plan
        continue-on-error: true
      - name: Report drift
        if: steps.plan.outcome == 'failure'
        run: |
          drift_count=$(terraform show -json tfplan | jq '.resource_changes | length')
          echo "Drift detected: $drift_count resources"
          cat <<EOF | curl -fsSL -X POST "$PUSHGATEWAY_URL" --data-binary @-
          # HELP terraform_drift_resources_total Resources drifted from state
          # TYPE terraform_drift_resources_total gauge
          terraform_drift_resources_total{env="${{ matrix.env }}"} $drift_count
          EOF
          curl -fsSL -X POST "$SLACK_WEBHOOK" \
            --data-urlencode "payload={\"text\": \"Drift in ${{ matrix.env }}: $drift_count resources. Run ${{ github.run_id }}.\"}"

Three properties of this drift job:

1. It runs on a schedule, not on every PR. Drift is an operational property; it is not a code property. The configuration did not change. The cloud did.
2. It uses a read-only role. The job detects drift; it does not fix it. Fixing drift is the apply pipeline’s job, gated by review and approval.
3. It emits a metric and a notification. The metric feeds the drift-growth alert; the notification pages the on-call engineer to investigate the change.

The cost of silent apply drift

Silent apply drift is the failure mode where the apply succeeded, the log says “Apply complete!”, and the cloud does not match the state. The state file is the source of truth according to Terraform; the cloud is the source of truth according to the application. The two have diverged, and the apply log does not show it.

Three common causes:

1. local-exec made a side-effect call. The null_resource ran a shell command that updated the cloud, but the resource attribute in the state file does not reflect the update. The next plan sees the state; the cloud does not match.
2. The provider partially succeeded. The AWS provider updated a resource but did not update all of its attributes. The state file says “updated”; the cloud says “partially updated”. The next plan sees the state; the cloud is inconsistent.
3. A manual change happened between the plan and the apply. The plan was correct; the apply ran; in the seconds between the plan and the apply, someone edited the resource in the console. The apply then overwrote the manual change. The state file is consistent with the apply; the cloud is not.

The cost of silent drift:

• The next plan shows no changes (the state thinks the resource is correct).
• The cloud is in an unexpected state.
• An incident is triggered when the application fails because of the drift.
• The recovery requires reading the state and the cloud, reconciling them manually, and re-applying.

The fix is observability. The drift-detection job catches the silent change within six hours. The metric and the alert surface the drift. The runbook is the recovery procedure.

What the pipeline does not measure

The pipeline does not measure:

• Provider reliability. A provider that returns transient errors is not visible in the pipeline metrics. The fix is to monitor the provider API directly with terraform_provider_errors_total and to correlate with the cloud provider’s status page.
• State backend reliability. A DynamoDB table that is throttling is not visible in the pipeline metrics. The fix is to monitor the backend service directly (DynamoDB metrics, S3 metrics).
• Cost. The apply creates resources; the bill shows up later. The fix is to tag resources with the environment and to use a cost-allocation tool.

Each of these is a different control. The pipeline metrics cover the pipeline; the cloud-provider metrics cover the provider; the billing tool covers the cost.

Validation commands

Confirm the observability stack is wired correctly:

# 1. Pushgateway is reachable from the runner.
curl -fsS "$PUSHGATEWAY_URL/metrics" | head

# 2. The drift job detected or did not detect drift.
terraform plan -detailed-exitcode
# exit 0 = no drift
# exit 2 = drift detected

# 3. The metric is present in Prometheus.
curl -fsS "$PROMETHEUS_URL/api/v1/query?query=terraform_drift_resources_total"
# {"status":"success","data":{"resultType":"vector","result":[{"metric":{...},"value":[...]}]}}

# 4. The alert is firing (or not) as expected.
curl -fsS "$ALERTMANAGER_URL/api/v1/alerts" | jq '.data[] | {alertname: .labels.alertname, state: .state.state}'

A metric that does not appear in Prometheus is a pipeline bug, not a Terraform behaviour. The push step must succeed.

Production failure modes

1. Drift grows undetected. The drift-detection job was disabled for cost reasons or the schedule was removed. The cloud drifted; the next apply overwrote the manual change; the incident is the first signal. The fix is to keep the drift job running on a schedule and to alert on terraform_drift_resources_total growth.

2. Plan duration slowly creeps up. The 90th percentile was 30 seconds six months ago; it is now 8 minutes. The PR feedback loop is broken; engineers stop reviewing plans. The fix is to split the state before the next incident.

3. Lock contention alert fires for the first time. Two applies raced; one lost. The fix is the concurrency group on the apply job. If the alert keeps firing, the state backend is the bottleneck and the state needs to be split.

4. Apply failure rate spikes after a Terraform upgrade. The 1.9.x upgrade changed a provider’s default behaviour or removed an argument. The fix is to roll back the upgrade, fix the configuration, and re-apply.

5. Saved-plan rejections become the norm. The state is being modified between every plan and every apply. The fix is to find the concurrent modification (another CI pipeline, a manual state edit, a drift remediation that is racing the apply) and serialise it.

6. Pipeline runs but emits no metrics. The push step fails silently. The dashboard shows no data; the alerts never fire; the pipeline is broken. The fix is to require the push step to fail the job on error, and to alert on the absence of metrics after a successful apply.

What comes next

This lesson closes the CI/CD module. The next module in the course is on incident response: what to do when the pipeline fails in production, how to recover state, how to recover from a partial apply, and how to write a postmortem.

Verification

Run the drift-detection job against a throwaway AWS account and confirm it emits terraform_drift_resources_total. Add a metric to the alert rule and confirm it fires. Run the pipeline against the same account with a long-running provider operation and confirm terraform_plan_duration_seconds captures the duration. Tear down the test resources when finished.