Production Guardrails as Code

Guardrails are the rules that stop a bad plan from being applied. They are not the rules that tell you what to do — those are conventions. Guardrails are the last line of defence between a merge and a production change. A team that has no guardrails relies on the operator’s memory; a team that has too many guardrails grinds to a halt. The job is to find the middle.

A production scenario

A platform team runs 40 production Terraform workspaces on Terraform Cloud. Two apply failures the previous quarter:

1. An engineer wrote provider "aws" { region = "us-east-1" } in a tfvars override for a sandbox. The plan passed review. The apply ran. The bill for the next month included a 6 TB EBS volume in us-east-1 that the rest of the company does not use, in a region the security team had de-listed.
2. A different engineer copy-pasted a module "vpc" \{ source = "terraform-aws-modules/vpc/aws" \} from a public registry module they had not vetted. The apply ran. The module defaulted to creating a public S3 bucket endpoint and a permissive network ACL.

Neither failure was a flaw in Terraform. Both were failures of control. Both would have been stopped by a one-rule guardrail before the apply.

What a guardrail is, and what it is not

A guardrail is a rule that runs against a Terraform plan (or, less commonly, against the configuration itself) and either admits or rejects the apply. It is not:

• A linter. tflint and the terraform validate step catch syntax errors and provider-version problems. They are not the enforcement layer for organisational policy. A lint pass does not stop the apply.
• A code review. A human review catches intent problems, not policy violations. Reviews get rubber-stamped under deadline.
• A code style rule. Naming conventions belong in CI as pre-commit hooks. Guardrails are about safety, not aesthetics.
• A module invariant. Hard constraints (no public S3, no 0.0.0.0/0 ingress) belong in the module interface. A policy that catches them at the plan layer is a backstop for when the module boundary is bypassed.

The four classes

Most production Terraform guardrails fall into four classes. The right severity differs by class.

Class                          Example                       Severity
---------------------------------------------------------------------
1. Security & compliance       No public S3, encryption      Hard deny
2. Cost & reliability          Allowed regions, instance     Hard deny
                               type whitelist
3. Tagging & ownership         Required tags on every        Hard deny
                               billable resource
4. Module whitelist            Only approved modules         Soft deny
                               from internal registry

The first three are non-negotiable. A missed required tag means the bill cannot be reconciled to a team; a missed encryption rule can mean a data exposure incident. The fourth is more nuanced: blocking an unapproved module is healthy, but the whitelist needs maintenance, and an immature whitelist creates friction. Most teams start the fourth class as a soft deny (warn, allow with override) and graduate to a hard deny once the catalogue of approved modules is large enough.

Severity model

A guardrail framework supports three severities. Pick the right one for the class.

Advisory. The plan is annotated; the apply still runs. Use for style conventions and new rules during a grace period.

Soft deny. The apply is blocked unless the operator acknowledges the violation by name and gives a reason. The acknowledgement is audited. Use for module-whitelist rules and cost-control rules during the migration window.

Hard deny. The apply is blocked unconditionally. No acknowledgement overrides it. Use for security and compliance rules where the cost of an exception is itself a production risk. Examples: no public S3 buckets, no 0.0.0.0/0 SSH ingress, no plaintext secrets in state.

Where guardrails run

The enforcement point depends on the workflow. The four common patterns, in order of strictness:

1. Terraform Cloud / Enterprise Sentinel policies. A run task or Sentinel policy. Run tasks execute after terraform plan. Sentinel runs against the plan structure. The apply is gated by the policy result.
2. CI pipeline policy step. A pre-merge job that runs conftest test against the rendered plan or the HCL. Used on PR workflows without Terraform Cloud.
3. Local pre-commit and terraform plan wrappers. Catch the obvious before push. Useful but not sufficient as the only layer — operators can --no-verify.
4. Admission control for Atlantis or similar. Some self-hosted runners expose a webhook hook where the rule set is enforced before the apply job starts.

The defence-in-depth posture is all four, in order. The first three are the safety net for the fourth.

Example: allowed regions (Hard deny)

A common first policy. The plan may not create resources in any region other than an approved list. Sentinel:

import "tfplan/v2" as tfplan
import "strings"

approved = ["eu-west-2", "eu-west-1", "us-east-1"]

main = rule {
  all tfplan.resource_changes as _, rc {
    rc.change.after and rc.change.after.region contains approved[_]
  }
}

The same policy in Rego for OPA / Conftest, evaluated against the plan JSON:

package terraform.regions

approved := {"eu-west-2", "eu-west-1", "us-east-1"}

deny[msg] {
  rc := input.resource_changes[_]
  region := rc.change.after.region
  not approved[region]
  msg := sprintf("region %q is not in the approved list", [region])
}

Run it:

conftest test plan.json --policy policy/

FAIL - plan.json - region "us-east-1" is not in the approved list
1 test, 0 passed, 1 failed

Example: required tags (Hard deny)

Every taggable resource must carry Owner, Environment, CostCentre. Rego against the plan:

package terraform.tags

required := {"Owner", "Environment", "CostCentre"}

deny[msg] {
  rc := input.resource_changes[_]
  rc.type == "aws_instance"
  not rc.change.after.tags.Owner
  msg := sprintf("%s is missing tag Owner", [rc.address])
}

The rule iterates only over resources whose tags apply. Provider APIs that ignore tags on certain resources (e.g. aws_iam_role does, aws_kms_key does not) need a different selector. Test the selector before assuming the rule is universal.

The exception process

A hard deny without an exception process will be bypassed. The correct process is a code change to the policy file.

git checkout -b exception/PROJ-4521-eu-central-1
# Edit policy/regions.rego
# Add eu-central-1 to the approved list under a comment
#   linking to ticket PROJ-4521
git commit -m "exception(PROJ-4521): permit eu-central-1 for data residency"
git push

The merge is a normal PR. The policy file’s git history becomes the audit log. The exception is self-documenting and self-removing — when the project that needed the exception ends, the next engineer reads the history and removes the line.

Cost of over-blocking

A team that has the wrong severity for a class pays for it. Common symptoms:

• #policy-bot churn. The same Slack thread appears every day. The same ten PRs are re-opened to add an exemption.
• Workarounds. Engineers split resources into multiple modules to skip the rule, or commit to a different repository. The control surface is no longer what the policy team thinks it is.
• Stale rule set. A rule blocks a resource type that has been renamed or deprecated. The team stops filing tickets because “policy is broken anyway”.
• Shadow applies. Engineers run terraform apply from a laptop with --no-validate to skip the CI step. The guardrail formally exists and informally does not.

The recovery is to classify the offending rule as the wrong severity, downgrade it to a soft deny, and run an amnesty for existing violations while a fix is delivered.

Failure modes specific to guardrails

Five failure modes to recognise in production:

1. Rule bypassed by refactor. A required-tag rule enumerates resource types (aws_instance, aws_db_instance). A new resource type (aws_rds_cluster) is added to the catalogue. The plan creates it without the tag. The rule passes because the resource type is not in the selector. Mitigation: selectors by tag pattern, not by type.
2. Plan-time vs apply-time drift. A rule checks the plan. If the apply fails and the resources are left in a half-state, the rule passed but the result is unsafe. Pair the rule with a post-apply check (drift detection, continuous policy runs).
3. Provider upgrade breaks selector. A provider schema change moves an attribute into a different path. The rule still passes because no plan fails, but the rule no longer binds to the resource. Mitigation: run policy tests in the CI for the policy repository itself.
4. Regional policy applied globally. A rule intended for the prod account is enabled across all accounts including the sandbox. Engineers in the sandbox cannot create the resources they need. Mitigation: scope per workspace or per account; review the scope list quarterly.
5. Exception list out of sync. A project ends but its exception stays. Over time the approved regions list contains regions that no business reason supports. Mitigation: exceptions expire. The git commit that adds the exception also has a calendar entry to revisit.

What comes next

Guardrails are the goal. The next two lessons are the tools that enforce them and the rules that make them up. A team that understands the goal before picking the tool will write shorter, more durable policies.

Verification

Confirm the policy framework is in place and behaves correctly.

conftest test plan.json --policy policy/

Expected output for a plan that complies with the rule set:

2 tests, 2 passed, 0 warnings, 0 failures

For a plan that violates the regions rule:

FAIL - plan.json - region "us-east-1" is not in the approved list
2 tests, 1 passed, 1 failure

For Terraform Cloud, verify in the UI:

Policy check: Soft mandatory
  Sentinel result: false
  Policy: aws-region-whitelist
  Message: region "us-east-1" is not in the approved list

The apply is blocked. The PR cannot be merged past the Speculative Plan stage. The audit log records the run task invocation with a timestamp and the operator identity.