Declarative versus Imperative Provisioning

A junior engineer writes a Terraform configuration for a new service. The configuration declares a security group first, then an EC2 instance, then a security group rule. The apply fails. The error says the rule cannot find the security group. The engineer is convinced Terraform is broken: the file is in the right order.

The file is in the right order for a human reader. Terraform does not read in order. Terraform builds a graph. The graph is correct; the mental model is wrong. The resulting blame is the most common production-day incident caused by imperative thinking in a declarative tool.

This lesson is the disambiguation.

Definitions, in one paragraph

A declarative system is one where you describe the desired end state and a tool computes the steps. An imperative system is one where you write the steps and a tool executes them. The trade-off is explicit control (imperative) versus automatic ordering, idempotency, and plan-ability (declarative).

Imperative scripts are full of control flow:

if ! aws ec2 describe-security-groups \
       --filters Name=group-name,Values=web-sg \
       --query 'SecurityGroups[0].GroupId' --output text \
       | grep -q '^sg-'; then
  aws ec2 create-security-group --group-name web-sg \
    --description "Web security group"
fi

The declarative equivalent declares the end state. The tool handles the “if it does not exist” branch automatically.

resource "aws_security_group" "web" {
  name        = "web-sg"
  description = "Web security group"
}

Same outcome. The declarative version is reviewable, idempotent, and plan-able. The imperative version is none of those things.

Why Terraform’s core model is declarative

Terraform exists to answer a single question well: what would change if I ran this now? That question is hard to ask against an imperative script. The script does not know what it did last run. It does not know what exists. It does what it is told.

A declarative tool can answer the question because it describes the state, not the steps. Concretely, Terraforms declarative nature is visible in three properties.

Plan before apply. terraform plan produces a precise diff. A script does not have an equivalent — it has a run, and a run either happened or it did not.

Idempotency by construction. Re-applying the same configuration is safe. The tool compares to state and real world, and emits a no-op if nothing has changed.

Graph-ordered execution. Terraform builds a directed acyclic graph from the configuration, walks it in dependency order, and applies changes in parallel where independent. The order in the .tf file is irrelevant. The graph is the truth.

The third property is the one that surprises imperative thinkers. A shell script runs top to bottom. A Terraform file is not a script. It is a set of declarations that Terraform parses into a graph.

The resource graph, in code

A tiny example. Two resources, one explicit dependency. The order in the file is wrong from a human reader’s perspective.

# File order is web rule first, then web SG.
# Terraform ignores file order.

resource "aws_security_group_rule" "web_ingress" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.web.id
}

resource "aws_security_group" "web" {
  name        = "web-sg"
  description = "Web security group"
}

Terraform reads both blocks. It sees that web_ingress refers to aws_security_group.web.id. It builds the graph:

aws_security_group.web
        │
        ▼
aws_security_group_rule.web_ingress

The graph is correct. The SG is created first, then the rule. The file order does not matter. This is the declarative model working as designed.

$ terraform plan
# aws_security_group.web will be created
# aws_security_group_rule.web_ingress will be created

The imperative escapes

Terraform is declarative in its core model, but the tool exposes two well-bounded imperative escapes. They exist because some problems are genuinely imperative. Both should be used sparingly, and both should be reviewed harder than the declarative core.

provisioner blocks. Run a shell command, locally or over SSH, after a resource is created or destroyed.

resource "aws_instance" "bootstrap" {
  ami           = "ami-0e1bed4f"
  instance_type = "t3.medium"

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nginx",
    ]
  }
}

The provisioner runs after the EC2 instance is created. The commands are imperative. The wrapping resource is declarative. Mixed in the same block. This is the part that surprises people most.

lifecycle meta-arguments. Declare behaviour that the default declarative model would not produce: create_before_destroy, prevent_destroy, ignore_changes.

resource "aws_instance" "web" {
  ami           = "ami-0e1bed4f"
  instance_type = "t3.medium"

  lifecycle {
    create_before_destroy = true
  }
}

The lifecycle block is declarative about imperative-shaped behaviour. It is the right place to encode ordering guarantees (create the new thing before destroying the old) without leaving the declarative model.

The cost of imperative thinking inside a declarative tool

Five patterns appear repeatedly when teams treat Terraform as a scripting language.

1. Order-dependent file authoring. “I put A before B in the file.” File order is cosmetic in Terraform. The graph is what runs. Teams that author top-to-bottom write files that survive until the first refactor, then break silently.
2. Hidden control flow in dependencies. A local-exec provisioner that queries AWS and writes a value to a file that another resource reads. The plan cannot see this. The failure mode is silence.
3. Manual sequencing. Engineers organise PRs to “apply in a specific order.” The CLI does not honour this. The plan proposes what the graph proposes; the apply does the same.
4. Idempotency assumed. A user_data script that is not idempotent creates duplicate resources on re-apply. The next engineer diagnoses the result as a Terraform bug.
5. State modifications outside the tool. Console changes made because the operator did not trust the plan. The next plan now shows drift — and the operator does not know if the drift is theirs or the tool’s.

The right mental model:

Imperative script:  steps  → state
Declarative tool:   end state, tool computes steps  → state

When you write a Terraform file, write the end state. Let Terraform compute the steps.

When the imperative escape is the right choice

There are cases where the imperative escape genuinely is the right tool. They are rare but real.

• Bootstrapping a single VM that cannot be configured by a configuration manager. The VM has no Ansible, no Puppet, and the cloud-init is not sufficient. A remote-exec is the smallest possible install path.
• Calling a vendor CLI that has no Terraform provider. The vendor sells a CLI. The provider does not exist. The local-exec after a resource creation is the bridge.
• One-shot migration of legacy data into a new system. A local-exec runs migrate.sh once; the result is captured in state; the next apply is a no-op.

In each case, the imperative piece is narrow, documented, idempotent at the boundary, and followed by a comment that explains why the declarative model was not enough. If a comment cannot be written, the imperative escape is being used where the declarative model would have worked.

How to validate the mental model

Three checks to confirm that a configuration is being read as declarative.

# READ-ONLY: confirm the graph order matches your intent.
terraform graph | head -20
# Directed graph; arrows show dependency order, not file order.

# READ-ONLY: count provisioners in the configuration.
grep -rn 'provisioner' *.tf | wc -l
# A non-zero count is acceptable; a large count is a debt.

# READ-ONLY: the plan should not show file-order surprises.
terraform plan

If terraform graph shows an order that does not match what you wanted, the configuration is missing an explicit dependency — typically a forgotten security_group_id = aws_security_group.web.id reference. Add the reference, not the reordering.

Production failure modes

1. remote-exec against the wrong instance. The provisioner runs against the new instance, the engineer intended the old one. The result is configuration on the wrong host. Reachable when lifecycle changes rebuild the resource.
2. local-exec reads state from disk. A local-exec reads a file that another resource has not yet written. The apply races; the next apply produces a different result. Reachable when the dependency is implicit rather than graph-encoded.
3. create_before_destroy forgotten. A lifecycle upgrade that needs ordered replacement omits create_before_destroy and a 30-minute outage ensues because the old resource is destroyed before the new one is ready.
4. Provisioner scripts left after a server rebuild. The user_data is the right place for first-boot config; the remote-exec is the wrong place for ongoing config. The failure appears as drift between servers.
5. State drift from imperative workarounds. The engineer used the console because the provisioner was too slow. The next plan shows drift that cannot be reconciled without losing the operator-introduced change.

Security implications

The imperative escapes have a different security profile from the declarative core.

• local-exec runs on the workstation with the engineer’s shell environment and AWS credentials. The output is shell-visible. A local-exec that prints secrets to stdout exposes them in CI logs.
• remote-exec runs on the target host over SSH. The connection inherits whatever SSH key the resource block uses. A shared SSH key across production hosts is a compromise amplifier.
• provisioner secrets are not redacted by Terraform. Inline values such as password = var.db_admin_password appear in the plan output if they are not marked sensitive.

The default is: minimise provisioner use, and when the use is genuine, the secret values are sensitive = true in the source attribute and the runner has least-privilege credentials.

Performance implications

The declarative core plans efficiently because the graph is cacheable. The imperative escapes do not.

• A local-exec on every resource creation serialises the apply. Parallelism is lost.
• A remote-exec over SSH has a per-invocation setup cost. Hundreds of these in a single apply turns a 30-second apply into a 30-minute apply.
• A lifecycle.create_before_destroy doubles peak resource count for the duration of the apply. In capacity-constrained environments, the apply can fail with capacity errors.

If the apply is slow, look for provisioners. The declarative core has parallel-by-default behaviour. The imperative escapes do not.

What comes next

The next lesson is the Terraform ecosystem — where Terraform sits among OpenTofu, Pulumi, Crossplane, and the vendor-managed services, and how to choose.

Verification

Five checks at PR review time confirm the configuration is being read declaratively.

# READ-ONLY: the graph should match intent, not file order.
terraform graph | head -20

# READ-ONLY: provisioners should be rare.
grep -rn 'provisioner' *.tf | wc -l

# READ-ONLY: are the explicit dependency references in place?
# A security group rule should reference its security group
# by attribute, not by name search.
grep -E 'security_group_id' *.tf

# READ-ONLY: lifecycle blocks should declare intent, not
# hide behaviour.
grep -rn 'lifecycle' *.tf

# READ-ONLY: the plan should be runnable in CI on every PR.
gh api repos/{owner}/{repo}/contents/.github/workflows \
  --jq '.[].name' | grep -i plan

If the plan produced by CI matches the plan produced by the apply-time runner (modulo expected drift), the configuration is declarative. If they diverge, the configuration has imperative escapes that the graph does not capture.