Terraform for Production Sysadmins — Final Assessment

This paper has two parts and one estate. Part A is 62 auto-scored questions, drawn from all twenty-nine parts of the course and grouped into the eleven competency domains listed further down. Part B is four scenario responses that you write out and mark against the rubric printed with each one. Both parts refer to the same estate, described below; nothing in the paper describes a different one.

Part A is closed-book. Part B is open-book and open-shell: what is being assessed there is the evidence you collect and the reasoning you show, not whether you can recite a command. Every Part B answer should contain:

• the symptom and its impact, stated in one sentence
• the evidence you collected, in the order you collected it
• the most likely root cause, with the evidence that supports it
• the remediation you applied, and why that one
• the validation that proves the estate is healthy again
• the rollback you kept ready in case the remediation failed

A strong answer names the address, the command and the artefact. An answer that says “restore the state” without naming which version, from which layer, and how the restore is verified is incomplete. An answer that names a command without saying what evidence it produced is incomplete in the other direction.

Scoring

Each Part B response is marked out of 12: five marks for the evidence, five for the remediation, two for the validation and the rollback taken together.

• Pass mark: 88 of 110, which is the 80% recorded in this page header.
• Part A floor: 44 of 62. Part B floor: 34 of 48.

Both floors and the total have to be met. The arithmetic is deliberate: a perfect Part A on its own is 62 marks and does not pass, and neither does a perfect Part B. You cannot recite your way through this paper and you cannot hand-wave your way through it either.

The estate under assessment

Every question that says “the estate” means this one.

You have inherited a single production Terraform estate at a company called Acme. Four engineers share it. It has been applied against AWS for about a year and manages roughly 380 resources: one VPC, nine subnets, an ECS cluster and its services, one RDS PostgreSQL instance, and the IAM that binds them together.

One Git repository holds one root module. There is no staging and no development environment; production is the only environment that has ever been applied. terraform workspace list shows default and nothing else.

The backend is S3. Bucket versioning is off. The backend block names no dynamodb_table, sets no encrypt and names no kms_key_id. There has never been a state backup and nobody has ever restored one.

Every engineer holds a long-lived IAM user access key with the AdministratorAccess policy attached, exported from their shell profile. The same key is stored as a repository secret and is what CI authenticates with.

CI is GitHub Actions. On every push to main it runs terraform init, then terraform plan, then terraform apply -auto-approve. The plan is not saved and the apply re-plans. There is no branch protection: any of the four engineers can push straight to main.

The repository has no required_version, no provider version constraints, and no committed .terraform.lock.hcl — the lock file is listed in .gitignore. Internal modules are sourced from GitHub with no ?ref= argument. There is no drift detection, no policy check, no terraform test, no tflint and no security scanner anywhere in the pipeline. The only monitoring is a Grafana dashboard of the application itself; nothing watches Terraform.

The estate is otherwise healthy. Every managed object is running and serving traffic. Nothing here requires you to believe that the estate is broken today — only that you can say what will break it, in what order, and what you would do about each.

Excerpt 1 — the terraform block and backend

terraform {
  # no required_version

  required_providers {
    aws = {
      source = "hashicorp/aws"
      # no version constraint
    }
  }

  backend "s3" {
    bucket = "acme-tfstate-prod"
    key    = "production/terraform.tfstate"
    region = "eu-west-2"
    # no dynamodb_table, no encrypt, no kms_key_id
  }
}

Excerpt 2 — the production database

variable "db_password" {
  type      = string
  sensitive = true
}

resource "aws_db_instance" "primary" {
  identifier          = "acme-prod"
  engine              = "postgres"
  engine_version      = "15.4"
  instance_class      = "db.t3.medium"
  allocated_storage   = 100
  username            = "acme_admin"
  password            = var.db_password
  skip_final_snapshot = true
  # no deletion_protection, no lifecycle block
}

Excerpt 3 — the subnets

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "public" {
  count = 3

  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]
}

Excerpt 4 — the application IAM role

resource "aws_iam_role_policy" "app" {
  name = "application-policy"
  role = aws_iam_role.app.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "*"
      Resource = "*"
    }]
  })
}

Excerpt 5 — an internal module call

module "network" {
  source = "git::https://github.com/acme/modules.git//network"

  vpc_cidr    = "10.0.0.0/16"
  environment = "production"
}

Excerpt 6 — the pipeline

# .github/workflows/terraform.yml
on:
  push:
    branches: [main]

jobs:
  apply:
    runs-on: ubuntu-latest
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: terraform init
      - run: terraform plan
      - run: terraform apply -auto-approve

Part A — the competency domains

The questions above this body are grouped by the domains below. Each domain opens with the reasoning a passing student should be able to produce without looking anything up.

1. Architecture, the workflow and what each command proves

Terraform is three moving parts and one artefact. The CLI parses the configuration and builds a graph; Core walks that graph and decides the actions; the provider plugins translate each action into API calls and hold the credentials while they do it. The artefact is the state, and it is the only record that maps a resource address in your configuration to an object in the world.

A passing student can say what each command proves and, more importantly, what it does not. terraform fmt proves canonical style and nothing else; -check exits 3 when a file would change. terraform validate parses, builds the graph and type-checks expressions, calling no API and reading no state, so it cannot prove that an apply will succeed. terraform init resolves providers and the backend. terraform plan in 1.9.x refreshes by default, interleaved per resource — refresh that resource, then compute its diff — which is why one plan can differ from another twenty minutes earlier with no Git change between them. terraform apply executes; with a saved plan it executes exactly what was reviewed and refuses if the state has moved underneath it. terraform destroy is apply with the inverse plan, and it is the only command in the workflow whose cost is measured in data loss rather than in minutes.

-parallelism caps concurrent resource operations during apply and defaults to 10. It is bounded by the slowest API rate limit in the configuration, never by the core count of the machine running it.

2. Installation, versioning and the reproducibility contract

Reproducibility in Terraform is two files and one habit. required_version in the terraform block is the contract with the operator; required_providers is the range of plugin versions the configuration is willing to accept; and .terraform.lock.hcl is the record of the exact versions and package hashes that were last resolved. The constraint alone is not reproducibility — ~> 5.0 admits hundreds of patch releases — so the lock file is committed and the .terraform/ directory is not.

The pessimistic operator is the one most often misread. ~> allows only the rightmost component you name to increment. ~> 1.9.0 names the patch, so it accepts 1.9.0 and later 1.9.x and refuses 1.10.0. ~> 1.9 names the minor, so the minor moves: it is the same constraint as >= 1.9.0, < 2.0.0 and it admits 1.10.0. Nothing in Terraform zero-fills a component you did not write; the number of components you write is the constraint you get.

The lock file records a version, the constraint in force when it was written, and package hashes. Hashes are per platform, so a team with macOS workstations and a Linux runner records all of them with terraform providers lock -platform=... rather than letting the runner rewrite the file. Core upgrades and provider upgrades have separate cadences and belong in separate pull requests, so that when the plan moves you know which change moved it.

3. HCL, variables, outputs and the configuration interface

HCL has primitives (string, number, bool, null), collections (list, set, map) and structural types (tuple, object). A list is ordered and indexable; a set is unordered and is not, which is exactly why a module that indexes its input with count.index cannot take a set without converting first. null is its own value: as a variable default it means “the caller decides”, as an argument value it means “do not send this argument”, and it is equal to neither "" nor [] nor 0.

Variable precedence is fixed and it surprises people in production. From highest to lowest: -var and -var-file on the command line, then *.auto.tfvars read in alphabetical order with the later file winning, then terraform.tfvars, then TF_VAR_ environment variables, then the default in the variable block. production.auto.tfvars sorts before staging.auto.tfvars, so staging wins — the filename is the source of truth, not the environment name. A stale TF_VAR_ export in a shell is one of the few ways to apply 80 resources into the wrong region and have the plan look correct while you do it.

sensitive = true is a redactor over the human-readable CLI stream. The value is still in the state, in the saved plan file, in terraform show -json, in terraform output -json and in TF_LOG=DEBUG output. It is necessary and it is not sufficient.

4. Resources, dependencies and the graph

Every managed resource has a CRUD lifecycle that the provider implements and the schema describes. Whether a changed argument produces ~ or -/+ is the provider schema decision, not yours: an attribute marked as forcing replacement has no Update path, so the only way to change it is to destroy and create. lifecycle is where you control the consequences — create_before_destroy reverses the order, prevent_destroy refuses any plan containing a destroy of that resource (including the destroy half of a replacement), and replace_triggered_by models a cascade.

Identity is the other half. count identity is positional, so inserting an element into the middle of the list it iterates shifts every instance after it and replaces each one. for_each identity is the map key, which is stable under insertion. Moving between them without destroying anything is what moved blocks are for: one block per instance, shipped in the same pull request as the refactor so a reviewer sees the rename and the configuration change in one diff.

Dependencies are ordinarily implicit: referencing aws_vpc.main.id creates the edge. depends_on is the explicit form for ordering that no reference expresses. A cycle is a structural defect in the configuration — Terraform rejects it during graph construction, before the planner runs, and it cannot be worked around with -target, -parallelism or state surgery. Two security groups that reference each other are fixed by moving the rules into separate rule resources so that neither group depends on the other.

5. State: the trust boundary

State is a JSON document with a fixed set of top-level fields. version is the format version, serial increments on every write and is the conflict-detection token, and lineage is a UUID fixed at creation that never changes — a mismatched lineage means “this is a different state”, not “this is an older state”. resources maps each address to the real object.

Losing that map does not delete anything, but it does make every object unmanaged: the configuration still names them, the state no longer does, and the next plan proposes to create a second copy of each. The same logic runs the state operations. terraform state rm removes the entry and leaves the object running, which is a hand-off tool and a duplicate-generator if you then apply without thinking. terraform state mv and moved blocks rename. terraform import and import blocks adopt an object that exists but is not tracked. Every one of them is preceded by a terraform state pull snapshot.

Security is separate from correctness. The state holds every attribute the provider returned, including passwords. SSE-KMS on the bucket means a stolen S3 credential without KMS access reads an encrypted blob; it does nothing about an authorised operator reading the state, which is an access-control problem. Locking is a different control again: the S3 backend takes its lock in a DynamoDB table whose partition key is a string named LockID.

Recovery is the last leg. RPO is the state change you are willing to lose and RTO is the time to be back online; both are business numbers, and the backup cadence must be tighter than the RPO with margin. Versioning on the bucket, cross-region replication, and a scheduled state pull are three layers, not one, and an untested restore is not a backup.

6. Modules, environments and estates at scale

A module is an interface: inputs with types and validation, outputs that form a contract, and an implementation that consumers do not read. Sources are pinned or they are not reproducible — a Git source with no ?ref= tracks the default branch, so two init runs can fetch different code from the same line of configuration. Releases follow semver: adding an optional variable is a minor, and anything that changes behaviour a consumer depends on — including a new required input — is a major, because the major bump is the consumer chance to review.

A state boundary is the line an apply cannot cross, and it is the unit of blast radius. The rule is that one state reads only the published outputs of another, through a terraform_remote_state data source, never the other state file directly. Two stacks must not manage the same address, share an IAM principal, or share a variables file.

Workspaces are not that boundary. A workspace is a named state within one backend configuration; every workspace in a backend shares the bucket, the lock table, the IAM principal and the provider configuration. They are right for per-engineer sandboxes and short-lived branches, where the blast radius is small and the state is disposable. They are wrong for production isolation, which needs a separate backend, separate credentials and a separate lock table — which in practice means separate directories and, at scale, separate accounts.

7. Plan review and drift reconciliation

The plan is the unit of review and the prefixes are the vocabulary: + create, - destroy, ~ update in place, -/+ destroy then create, +/- create then destroy (which is what create_before_destroy produces), and <= read a data resource whose arguments were not known until apply. A replacement is data loss until proven otherwise, and the # forces replacement comment against a single attribute is the most useful string in the whole output. The summary line counts actions; it says nothing about risk, so an in-place update that opens a security group to the internet reads as 1 to change and is still the most dangerous line on the page.

Drift is what the refresh found, and Terraform prints it in its own block above the proposed changes under a note that objects have changed outside Terraform. Detection is the cheap control: a scheduled terraform plan -refresh-only -detailed-exitcode where 0 means clean, 2 means drift and pages a human with the plan attached, and 1 means the detection itself is broken and pages immediately because you now know nothing.

Every drift finding is one of four shapes. Undesired: revert the world. Unrecorded intent: codify it in HCL and commit. Uninteresting: ignore_changes with a comment saying why. Unknown: treat it as undesired and investigate before adopting anything. An emergency console change is legitimate when it has an incident behind it, and the reconcile loop that follows it is terraform apply -refresh-only to absorb the change, then a follow-up pull request to codify it.

8. CI/CD, policy as code and testing

A production pipeline separates the plan stage from the apply stage and hands one artefact between them. terraform plan -out=tfplan writes the binary plan; terraform show -json tfplan renders the reviewable JSON; terraform apply tfplan executes exactly those actions. The saved plan carries the state hash from plan time, so an apply against a state that has since been written is refused rather than executing something nobody reviewed. An apply -auto-approve that re-plans has no such contract.

Three locks protect an apply and they are not interchangeable: the pipeline concurrency group stops two runners starting, the state backend lock stops two applies mutating one state, and the saved plan stops an apply executing a stale plan. Credentials are short-lived: OIDC federation exchanges a signed job token for an STS session, so there is no static secret to leak and the audit entry names the repository and ref rather than a key id. The plan role is read-only; the apply role is wider; they are not the same role.

Testing is layered and the cheap layer runs first. terraform test with mock_provider proves the configuration logic — validation, conditionals, outputs, graph shape — in under a second and proves nothing about whether the real provider will accept the arguments. Integration tests run the same framework against a disposable sandbox. Policy is the gate: static scanners read the HCL on disk and are fast feedback for the author, while plan-time policy reads the plan JSON with resolved values and is the only layer that decides whether the apply runs.

9. Security, credentials and the supply chain

The credential is not the secret; the policy attached to it is. A long-lived access key with AdministratorAccess is a master key that does not expire, and its blast radius is the whole account for as long as the key exists. The production answer is short-lived credentials from workload identity, an execution role scoped to the resources the configuration actually manages, a permissions boundary as the role ceiling, and an SCP as the organisation ceiling that denies an action even when the role policy allows it.

Secrets follow the same discipline. Fetch them from a secrets manager rather than passing them in as variables, but understand what that does and does not fix: the password is still an attribute of the database resource, so it is still in the state, and the state therefore still needs encryption, access control and audit. After an exposure the order is fixed — rotate first, because the exposed value must be assumed compromised, then fix the pipeline that exposed it.

The supply chain is providers and modules. State the full source address rather than relying on the default registry, pin the version range so a new major cannot be resolved without review, commit the lock file so init re-verifies the package hashes, and evaluate a third-party module the way you would evaluate any dependency you are about to give credentials to.

10. Upgrades, migrations and platform operations

An upgrade is a managed change with a rollback. For providers: edit the constraint, run terraform init -upgrade, then plan and read the diff — a provider release can change a default and turn a quiet estate into a hundred-line plan. For Core: read the upgrade guide, move patch releases quickly, move minors deliberately after staging, and treat a major as a project. OpenTofu is a separate tool in the same shell: it does not honour the Terraform lock file, and its breaking-change history diverges from 1.6 onward.

A backend migration is terraform init -migrate-state, and the work is in the preconditions rather than the command. The bucket, the lock table and the IAM policy must all exist first; the migration creates none of them. Adopting existing infrastructure is import blocks in configuration rather than one-off CLI imports, because the block is reviewable in the same pull request as the resource it adopts, and the step teams skip is copying the imported attributes back into the HCL.

Platform operations is where the boundary discipline shows. Terraform provisions the topology; Ansible configures what runs inside the operating system. A remote-exec provisioner that installs packages inside an instance block has no idempotency story, blocks the apply on a port being open, and leaves state with no record of what the host looks like. The clean hand-off is an inventory generated from Terraform outputs and consumed by a separate pipeline stage.

11. Troubleshooting, disaster recovery and the 3 AM test

An apply is a sequence of boundaries: configuration, graph, backend, provider, provisioner, lifecycle. The layer that printed the message is not necessarily the layer that caused the problem, so the method is to read the whole error, classify the layer, find the recent change, and only then choose a command. A partial apply is a normal possibility rather than corruption: some objects exist, the state records them, and the next step is a fresh plan from where you actually are.

State failures are three different problems with three different recoveries. A lock is contention — record the id, owner, operation and timestamp, find out whether the holder is alive, and reach for force-unlock only once it is confirmed gone. Corruption is a data-integrity problem and the recovery is a verified restore. A wrong address is a consistency problem between configuration and state, and the recovery is state mv, an import, or a state rm — chosen deliberately, after a snapshot.

Incident response adds the parts that are not technical. Break-glass is a pre-approved bypass that is scoped, time-bounded and recorded against an incident; the three hatches are -target, a manual state edit, and force-unlock, and an unrecorded bypass is not break-glass but shadow IT. Restoration precedes root cause when users are affected. The post-incident review produces an artefact, and the emergency change is not finished until it is back in the repository.

Part B — scenario responses

Write each response out. Mark it against the rubric under it: five marks for the evidence, five for the remediation, two for the validation and the rollback together. Forty-eight marks in total.

B1 — the plan proposes to destroy the production database

A pull request that was meant to remove a decommissioned worker pool also deleted the aws_db_instance.primary block. It was merged. CI has started and is at the plan step. The database is healthy and serving traffic. Excerpt 2 is the resource in question.

Evidence (5 marks). The configuration no longer declares the address; the state still holds it; the plan proposes a - on it; the real object is healthy. A full answer says where each of those four facts came from — the Git diff, terraform state list, the plan output, and a check against the database itself — and captures the plan artefact before anything is changed.

Remediation (5 marks). Stop the pipeline before the apply step. Restore the resource block from the previous commit and confirm the plan is empty. Then close the gap that let it happen: skip_final_snapshot = true becomes false with a final_snapshot_identifier, deletion_protection is enabled, and a lifecycle block adds prevent_destroy = true so that the next such diff fails loudly in review. The pipeline change is the real fix — a saved plan and an approval gate, so that no plan proposing a destroy reaches an apply without a human seeing it.

Validation and rollback (2 marks). Validation is an empty plan plus the database still answering. Rollback: if the apply already started, do not interrupt it — an interrupted apply leaves state and world disagreeing in a way you then have to reconcile. If the destroy completed, the recovery is the final snapshot, which the current configuration does not take, which is the finding.

B2 — a laptop holding an AdministratorAccess key is stolen

An engineer reports a stolen laptop at 22:40. Their shell profile exports a long-lived access key with AdministratorAccess. The key is also the one stored as the CI repository secret.

Evidence (5 marks). Which key id, from IAM rather than from memory. When it was last used, and from where, from the IAM credential report and CloudTrail. What it did in the window between the theft and the report. Whether the same key is in use anywhere else — and in this estate it is, because CI holds it too, which turns a single revocation into a change that breaks the pipeline.

Remediation (5 marks). Deactivate the key immediately rather than waiting for a rotation window, then delete it. Audit CloudTrail for the window. Re-issue the pipeline on OIDC federation so there is no static key to steal next time, and scope the execution role to what the configuration manages rather than to the account. State that the policy is the blast radius and that AdministratorAccess on four engineers is the finding, not the theft.

Validation and rollback (2 marks). Validation is the key showing as deleted in IAM, CloudTrail showing no further use, and a pipeline run that authenticates through the new path. Rollback is a deliberate decision not to restore the key: if the new path fails, the recovery is a fresh short-lived credential, never a reinstated one.

B3 — the state object is unreadable

An apply fails while loading the state. The object in the bucket parses as neither JSON nor anything else useful. The bucket has no versioning. The last copy anyone can find is a terraform state pull an engineer happened to run three days ago and left on their laptop. The estate has been applied twice since.

Evidence (5 marks). What exactly is unreadable, and at which layer — a truncated object, a wrong lineage, a serial older than expected. The serial and lineage of the three-day-old copy against what the last successful apply recorded. Which changes were applied in the intervening three days, from the pipeline history. What the real world currently contains, established from the provider rather than assumed.

Remediation (5 marks). Stop every pipeline that could apply. Take a copy of the damaged object before touching it. Restore the three-day-old state into a working directory, run a refresh-only plan against it, and read the drift: the objects created in the intervening applies are the ones the restored state does not know about, and they are imported rather than recreated. Only then re-plan normally. Name the reason versioning on the bucket would have made this a five-minute recovery.

Validation and rollback (2 marks). Validation is an empty plan after the imports and a state whose resource count matches what the world holds. Rollback is the copy of the damaged object plus a written decision point: if the reconciliation cannot be completed safely, the escalation is to an incident commander rather than to another apply.

B4 — the database password is in an archived CI log

A security review finds the value of var.db_password in a CI log from four months ago. The variable is marked sensitive = true. The log archive is searchable by everyone in the engineering organisation.

Evidence (5 marks). Where the value appears and which step emitted it. Whether it is also in the state — it is, because password is an attribute of aws_db_instance — and who can read the state bucket. Whether it appears in any plan artefact. How long the archive retains logs and who has queried them. Name why sensitive = true did not prevent this.

Remediation (5 marks). Rotate the password first; everything else is worthless until that is done. Then move the source of truth into a secrets manager and read it with a data source, understanding that this does not empty the state. Then encrypt the state with SSE-KMS and restrict who can decrypt. Then purge or restrict the log archive. Order matters and the marking rewards it.

Validation and rollback (2 marks). Validation is the application connecting with the new credential, the old value rejected by the database, and a plan that shows no pending change. Rollback is the previous credential held only until the new one is confirmed working, and then destroyed — not filed.

What a passing candidate looks like

They can read a plan and say which line is the dangerous one, and why the summary does not tell them. They can name what is in the state, what is not, and who can read it. They know which command changes the world, which changes the state, and which changes neither. They treat the estate above as a set of findings with an order, not as a list of things that are wrong, and for every finding they can state the evidence, the risk, the remediation and the validation.