Auditing Terraform Actions

A Terraform apply at 03:00 produces evidence in four distinct places. The forensic question at 09:00 is not “did the apply happen” but “what API calls did the apply make, which workspace, which commit, which engineer, and which state read”. The answer requires four log sources, joined by timestamps and ARNs.

The four sources are:

1. CloudTrail (management events)
   Every API call made by the Terraform execution role,
   including the role assumption, the resource CRUD, and
   the GetSecretValue read.

2. CloudTrail (data events on the state bucket)
   Every s3:GetObject and s3:PutObject against the state
   bucket. Records which state file was read and which
   apply wrote it.

3. Terraform Cloud (or self-hosted) audit log
   The workspace-level event: who initiated the plan, who
   approved the apply, the plan diff, the apply timestamp,
   the workspace variables used.

4. GitHub audit log (or GitLab/Bitbucket equivalent)
   Who pushed the commit, who approved the PR, who
   merged, the commit SHA, the workflow file that
   triggered the run.

Git commit log (the canonical source)
   The full diff that the apply executed.

Each source is necessary. CloudTrail alone does not say which engineer initiated the run; the Terraform audit log alone does not say which API calls were issued; the GitHub audit log alone does not say whether the apply succeeded. The production audit joins all four by requestId (CloudTrail), run-id (Terraform Cloud), workflow_run_id (GitHub), and commit SHA (Git).

This lesson covers the four sources, the retention that satisfies production forensics, the alerts that catch misuse, and the discipline that protects the audit trail itself.

CloudTrail management events

CloudTrail records every management API call by default. The Terraform execution role’s API calls appear as AssumedRole followed by the resource CRUD (ec2:RunInstances, rds:CreateDBInstance, and so on).

resource "aws_cloudtrail" "main" {
  name                          = "runbook-trail"
  s3_bucket_name                = aws_s3_bucket.cloudtrail_logs.arn
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::prod-tfstate-eu-west-2/"]
    }
  }

  depends_on = [aws_s3_bucket_policy.cloudtrail_logs]
}

Three production details:

1. is_multi_region_trail = true — the trail captures every region, not just the home region. A Terraform apply that creates a resource in us-east-1 from a workspace in eu-west-2 is recorded.
2. enable_log_file_validation = true — CloudTrail signs each log file. The integrity of the audit trail is cryptographically verifiable; a tampered file fails validation.
3. data_resource — the S3 data events. Without this, CloudTrail records that the role made an S3 call but not which object was read or written. The state bucket specifically must be in the data resource list.

CloudTrail data events on the state bucket

The state bucket is the asset. Every read of the state file is a potential reconnaissance event; every write is a potential tampering event. CloudTrail data events on the state bucket produce the audit trail for state access.

The relevant events:

s3:GetObject    arn:aws:s3:::prod-tfstate-eu-west-2/tfstate/prod/terraform.tfstate
s3:PutObject    arn:aws:s3:::prod-tfstate-eu-west-2/tfstate/prod/terraform.tfstate
s3:DeleteObject arn:aws:s3:::prod-tfstate-eu-west-2/tfstate/prod/terraform.tfstate

Each event records the source IP, the role ARN, and the request ID. A GetObject from an unexpected source IP or role is the canonical signal of credential misuse. A PutObject that is not associated with a RunInstances or equivalent API call is the canonical signal of state tampering.

Terraform Cloud audit log

Terraform Cloud (and Enterprise) maintains a per-workspace audit log of every plan and apply. The log includes the initiating user, the plan diff, the apply timestamp, the workspace variables referenced, and the run ID.

The audit log can be streamed to an external SIEM:

resource "tfe_audit_trail" "main" {
  name        = "runbook-prod"
  destination_type = "AWS"
  # AWS CloudWatch Logs group
  cloudwatch_logs = {
      log_group_name = "runbook-tfe-audit"
      role_arn       = aws_iam_role.tfe_audit.arn
    }
}

For self-hosted Terraform (the OpenTofu or Terraform CLI path), the equivalent audit is the local Terraform log (TF_LOG=DEBUG captured to a central log store) plus the GitHub Actions job log.

GitHub audit log

The organisation audit log records every action taken against the repositories: who pushed, who merged, who approved, who changed a workflow file, who changed a secret. The events are joined to CloudTrail by the commit SHA and to Terraform Cloud by the workspace ID embedded in the workflow file.

The relevant events for a Terraform change:

repo.push                  commit SHA, pusher
pull_request.opened        PR number, opener
pull_request_review        approver, review state
pull_request.merge         merger, merge commit
workflow_run.completed     run ID, conclusion

A workflow_run.completed with conclusion = "success" joined to a CloudTrail RunInstances event with the same requestId (visible in the GitHub Actions log) is the production-grade forensic trail.

Retention: 90 days hot, 365+ days cold

CloudTrail retains events for 90 days in the trail management console. Forensic investigations often require more. The production pattern is:

1. Hot retention (90 days): CloudTrail management events, queryable via aws cloudtrail lookup-events.
2. Cold retention (365 days minimum, 7 years for some compliance regimes): the S3 log archive bucket, with versioning and Object Lock in compliance mode.

resource "s3_bucket_object_lock_configuration" "cloudtrail" {
  bucket = aws_s3_bucket.cloudtrail_logs.id

  rule {
    default_retention {
      mode = "COMPLIANCE"
      days = 365
    }
  }
}

Object Lock in compliance mode prevents deletion of any object version, by any principal, including the root user, until the retention period expires. The audit trail is tamper-evident by design.

The alerts: real-time signals of misuse

Five alerts catch most production incidents before they become incidents:

1. AKIA-prefixed access key in the Terraform role trail. OIDC is configured; an AKIA appears. The OIDC path is broken or bypassed.
2. iam:* events from a human user (not a role). A human is editing IAM in the console. The change is drift.
3. AssumeRoleWithWebIdentity from a repo not in the trust policy. A workflow outside the expected scope is attempting to assume the role.
4. Apply outside business hours. A terraform apply initiated at 03:00 from a commit not on the on-call schedule. Could be legitimate; should be verified.
5. State bucket GetObject from an unexpected source IP. The role is being used from a network outside the expected CI runner range.

Each alert is wired to a topic in SNS, a Slack channel, or a PagerDuty service. The alerts are the production control that catches misuse in minutes, not weeks.

How to validate the configuration

# READ-ONLY — confirm the trail is multi-region and active
aws cloudtrail describe-trails --trail-name-list runbook-trail

{
    "TrailList": [
        {
            "Name": "runbook-trail",
            "IsMultiRegionTrail": true,
            "LogFileValidationEnabled": true,
            "HasCustomEventSelectors": true,
            "IsOrganizationTrail": false
        }
    ]
}

Confirm IsMultiRegionTrail is true and LogFileValidationEnabled is true.

# READ-ONLY — confirm data events are recorded for the state bucket
aws cloudtrail get-event-selectors --trail-name runbook-trail

{
    "EventSelectors": [
        {
            "ReadWriteType": "All",
            "IncludeManagementEvents": true,
            "DataResources": [
                {
                    "Type": "AWS::S3::Object",
                    "Values": ["arn:aws:s3:::prod-tfstate-eu-west-2/"]
                }
            ]
        }
    ]
}

Confirm the state bucket ARN is in the DataResources list.

# READ-ONLY — confirm Object Lock is in compliance mode
aws s3api get-object-lock-configuration --bucket runbook-cloudtrail-logs

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": {
            "DefaultRetention": {
                "Mode": "COMPLIANCE",
                "Days": 365
            }
        }
    }
}

Confirm Mode is COMPLIANCE and Days is at least 365.

Restricting access to the audit trail

The audit trail must be readable by the security team and writeable by CloudTrail only. A common production mistake is to grant the Terraform execution role read access to the log bucket “for debugging”. The role can then delete its own trail.

The discipline:

1. Log bucket policy allows s3:PutObject from the CloudTrail service principal only.
2. Read access is granted to a separate audit-reader role assumed by humans via SSO. The Terraform execution role has no read access.
3. Object Lock prevents deletion even by the audit reader.
4. CloudTrail integrity validation is run nightly; a cron job verifies each log file’s signature.

The result: the role that Terraform assumes cannot read or delete its own audit trail. The forensic trail is preserved even if the role is compromised.

Production failure modes

Six failures account for most audit gaps:

1. Single-region trail. The trail captures only the home region. An apply that creates resources in us-east-1 is invisible. The fix is is_multi_region_trail = true.

2. Data events disabled on the state bucket. CloudTrail records that the role made an S3 call, but not which object. The forensic trail for state access is empty. The fix is the data_resource block in the trail configuration.

3. No Object Lock on the log archive. An attacker with write access to the log bucket can delete the log files. The forensic trail is empty. The fix is Object Lock in compliance mode, 365+ days.

4. Alerts on the wrong events. Alerts wired to RunInstances only; no alert on iam:* or GetSecretValue. The signals of misuse are invisible. The fix is the five-alert list above.

5. Audit-reader role too broad. The audit reader can read every log bucket in the account, including application logs that may contain PII. The fix is bucket-scoped read access.

6. Log file integrity validation never run. A tampered log file passes a casual review; the signature check would catch it. The fix is a nightly cron that runs aws cloudtrail validate-logs.

Security and performance implications

The performance cost of data events on the state bucket is the per-event cost (CloudTrail charges per data event). For a state bucket that sees a handful of applies per day, the cost is negligible. The security value is the forensic trail for state access.

The performance cost of integrity validation is one API call per log file per day. The cost is negligible; the value is the cryptographic proof that the audit trail has not been tampered with.

The security cost of a single-region trail is incident visibility. An attacker who creates resources in a region not covered by the trail is invisible until the resources are discovered. The fix is the multi-region trail.

What to do in production

The minimum audit stack for a production Terraform estate:

1. Multi-region CloudTrail with log file validation enabled and an S3 data resource selector on the state bucket.
2. Object Lock in compliance mode on the log archive bucket, 365+ days.
3. Terraform Cloud audit log streamed to the same SIEM or a separate, equally-protected store.
4. GitHub audit log enabled for the organisation; events joined to CloudTrail by requestId and commit SHA.
5. Five alerts: AKIA in role trail, console-side IAM edits, OIDC assumption from unexpected repo, apply outside business hours, state bucket read from unexpected IP.
6. Audit-reader role with bucket-scoped read access, assumed via SSO. No service role has read access.

Verification

Run aws cloudtrail describe-trails and confirm IsMultiRegionTrail and LogFileValidationEnabled are both true. Run aws cloudtrail get-event-selectors and confirm the state bucket ARN is in DataResources. Run aws s3api get-object-lock-configuration on the log bucket and confirm Mode is COMPLIANCE. Trigger a test alert (AKIA in trail, console IAM edit) and confirm the page reaches the on-call.