Multi-Team Terraform Operations

The team boundary in a Terraform estate is the smallest unit that can be given its own state, its own credentials, and its own change window. Get the boundary wrong and every lesson in this part of the course fights against the org chart. Get it right and the rest of the architecture — accounts, regions, repos, modules — falls into place.

What “team” means in Terraform

Terraform itself has no concept of a team. It has credentials, workspaces, and state files. “Team” is an organisational concept that the platform maps onto Terraform primitives. The mapping is where the operational decisions live.

In Terraform Cloud or HCP Terraform the mapping is direct: a Team object owns permissions on a workspace and inherits a role. In OSS Terraform or OpenTofu the mapping is external: an IAM role in AWS, a group in Okta, a CODEOWNERS file in Git. The shape is the same — a named group with a set of capabilities applied to a named set of resources.

A correctly drawn boundary has three properties.

1. Blast radius. A change made by the team can only affect the resources the team owns.
2. Cadence. The team can apply on its own schedule without coordinating with other teams.
3. Auditability. From logs alone, you can answer “which team changed this resource last Tuesday at 14:32?”

If the current setup cannot answer those three questions for every resource, the boundary is wrong.

The three roles

Every team in a Terraform estate plays at most three roles. The roles map to capabilities, not to job titles.

                ┌──────────────────────────┐
                │  Admin (1-2 per estate)  │
                │  state, vars, IAM,       │
                │  registry, workspaces    │
                └─────────────┬────────────┘
                              │
                ┌─────────────┴────────────┐
                │  Operator (per on-call)  │
                │  plan + apply, read      │
                │  state in scope          │
                └─────────────┬────────────┘
                              │
                ┌─────────────┴────────────┐
                │  Developer (read-most)   │
                │  plan, read, comment     │
                │  cannot apply            │
                └──────────────────────────┘

Mapping teams onto Terraform state

The state file is the unit of ownership. The simplest mapping is “one state per team”. In practice that is rarely sufficient; most teams own more than one state because most teams own more than one environment.

# team-platform/identity/oidc/main.tf
# Owner: platform-team
# Reviewers: @platform-admins
# Apply window: business hours, Mon-Fri

terraform {
  backend "s3" {
    bucket         = "tf-state-platform-prod"
    key            = "platform/identity/oidc/terraform.tfstate"
    region         = "eu-west-2"
    dynamodb_table = "tf-locks-platform"
    encrypt        = true
  }
}

variable "github_org" {
  type        = string
  description = "GitHub organisation that OIDC will federate against."
}

variable "aws_account_id" {
  type        = string
  description = "AWS account that hosts the OIDC provider."
}

resource "aws_iam_openid_connect_provider" "github" {
  url             = "https://token.actions.githubusercontent.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["a031c22865a290e7e10e2c4a76b6a8c5d4e6f1e8"]
}

output "provider_arn" {
  value       = aws_iam_openid_connect_provider.github.arn
  description = "ARN of the GitHub OIDC provider; consumed by workload teams via remote-state."
}

The crucial line is the key in the backend block: platform/identity/oidc/terraform.tfstate. The first path segment identifies the team, the next identifies the stack, the leaf is the state. A team owns everything that begins with its prefix; another team’s prefix is not its concern.

In Terraform Cloud or HCP Terraform, the equivalent is the project plus workspace naming:

project:    platform-prod
workspace:  identity-oidc

Projects are the team boundary; workspaces are the stack boundary inside the team. Project-level permissions are how you give the whole project to a team; workspace-level overrides are how you carve out exceptions (for example, a shared observability workspace that the platform team and the workload teams both write to).

CODEOWNERS as the boundary enforcer

The team boundary in Git is a CODEOWNERS file. The boundary in Terraform is the state. The two must align, or there is a place where the boundary is unclear.

# .github/CODEOWNERS

# Default owners for everything
*                                                 @platform-admins

# Platform team — owns network, IAM, observability
/platform/networking/                             @network-team
/platform/iam/                                   @platform-team
/platform/observability/                         @observability-team

# Workload teams — own their own services
/workloads/orders/                               @orders-team
/workloads/payments/                             @payments-team

# Shared modules — owned by the platform, locked to a release tag
/modules/                                        @platform-team

A PR that touches /platform/networking/ requires a review from @network-team. A PR that touches /modules/ requires a review from @platform-team regardless of who opened it. The CODEOWNERS file is the part of the boundary you can see in the GitHub UI; the state ownership is the part that does the actual enforcement at apply time.

Variable scoping per team

Each team gets its own terraform.tfvars per environment. The file lives in the team’s repository (or in versioned object storage) and is consumed via a *.auto.tfvars file or an explicit -var-file= flag.

# team-orders/prod.auto.tfvars

environment   = "prod"
aws_region    = "eu-west-2"
service_name  = "orders-api"
desired_count = 6
image_tag     = "v2.41.0"

Sensitive values are not in this file. They are fetched from the secrets manager at apply time:

# team-orders/main.tf

variable "db_password" {
  type        = string
  description = "Resolved from AWS Secrets Manager; never set in tfvars."
  sensitive   = true
  default     = null
}

data "aws_secretsmanager_secret_version" "db" {
  secret_id = "orders/db/password"
}

locals {
  resolved_db_password = (
    var.db_password != null
    ? var.db_password
    : data.aws_secretsmanager_secret_version.db.secret_string
  )
}

How to validate the boundary

The boundary is valid when four things agree.

# READ-ONLY: every state key in the backend begins with the team prefix.
aws s3api list-objects-v2 \
  --bucket tf-state-platform-prod \
  --prefix platform/ \
  --query "Contents[].Key" \
  --output text

platform/identity/oidc/terraform.tfstate
platform/identity/groups/terraform.tfstate
platform/identity/roles/terraform.tfstate

Every key begins with platform/. The team owns everything under that prefix. Any state whose key begins with another team’s prefix is a misconfiguration that the bucket policy should reject.

# READ-ONLY: the CODEOWNERS file actually matches the backend layout.
gh api repos/ORG/REPO/contents/.github/CODEOWNERS --jq '.content' \
  | base64 -d

Each path in CODEOWNERS corresponds to a state prefix in the backend. If a prefix is missing from either side, the boundary has a hole.

# READ-ONLY: who can apply to the prod environment?
gh api \
  -H "Accept: application/vnd.github+json" \
  /repos/ORG/REPO/environments/prod \
  --jq '.reviewers[].login'

platform-team-lead
network-team-lead

The reviewers list is small, named, and on-call. Anyone not on the list cannot deploy.

# READ-ONLY: which IAM principals can run apply against the state bucket?
aws iam get-policy-version \
  --policy-arn arn:aws:iam::123456789012:policy/tf-state-platform-prod \
  --version-id v1 \
  --query 'PolicyDocument.Statement[].Action'

[
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject"
]

Only the platform team’s CI role and the on-call rotation principal appear in the policy’s principals. The developer role cannot s3:PutObject against the state bucket, which means a developer cannot terraform apply against production even if they have the credentials.

Production failure modes

These are the failure modes that show up in the field.

1. The team boundary is drawn at the wrong layer. Symptom: multiple teams can apply changes to the same resources because they share a state file. Cause: one state was created “because it was easier” and grew to include every team’s resources. Recovery: split the state with terraform state mv and terraform state rm into per-team states. Document the new prefixes in CODEOWNERS and the bucket policy.

2. A developer role has apply permission in production. Symptom: a terraform apply runs in production outside the change window. Cause: the role was granted “to unblock the team” and never tightened. Recovery: revoke the capability, audit the recent applies, and add the developer-role CI pipeline as the only path that can promote to production.

3. The .tfvars file contains a secret. Symptom: the secret appears in git log, in CI logs, in state snapshots, and possibly in a public mirror. Cause: a developer added a -var flag in a hurry and committed the result. Recovery: rotate the secret immediately, scrub the history with git filter-repo or BFG, and add a pre-commit hook (gitleaks, trufflehog) to catch the next one.

4. CODEOWNERS is set up but not enforced. Symptom: a PR merges without the required review. Cause: branch protection does not require CODEOWNERS review, or the rule is on the wrong branch. Recovery: enable “Require review from Code Owners” in branch protection settings and re-test with a dummy PR.

5. The team prefix in the state backend collides with another team. Symptom: a state file written by team A is overwritten by team B because both use the key prod/main.tfstate. Cause: the team prefix was not enforced at the bucket policy or at the IAM role. Recovery: rename the keys, lock the bucket policy to the correct prefixes, and add a CI check that fails on duplicate prefixes.

6. A team onboards, gets the wrong role by default. Symptom: a new engineer can terraform apply in production on day one. Cause: the onboarding script grants the default role without differentiating “developer” from “operator”. Recovery: rotate credentials, audit the access log, and gate the onboarding on a role-aware workflow that defaults to “developer”.

7. A team loses its only operator. Symptom: nobody can run terraform apply because the rotation is empty. Cause: the team relied on a single operator whose credentials expired or who left. Recovery: keep at least two named operators per team on the rotation at all times; rotate credentials on a fixed schedule; cross-train.

Security implications

• The Developer role cannot read secrets, but can read non-secret state. Do not put secrets in non-secret state. Use data blocks to fetch from the secrets manager.
• The Operator role can apply. Logging of apply events must be centralised and must include the IAM principal, the workspace, the commit SHA, and the plan ID. Without those four fields, an incident review cannot identify the change.
• The Admin role is the smallest, slowest-rotating set of credentials in the estate. Use break-glass procedures for short-lived admin access; do not give standing admin.

Performance implications

A per-team state is bounded by what one team can apply. The practical upper bound is around 500 resources; beyond that, plan becomes slow enough that developers stop running it locally. If a state grows past that, split it before performance becomes the reason the boundary is redrawn.

Production guidance

1. One team, one prefix. Every state key in the backend should begin with the team’s identifier. Make this a CI check.
2. One workspace per stack per environment. A team that owns three stacks in three environments has nine workspaces. Each workspace has its own variable set, its own schedule, its own run history.
3. CODEOWNERS mirrors the state prefix. The path in CODEOWNERS matches the prefix in the backend. If you have to write a custom rule to bridge the two, the boundary has diverged.
4. Developers plan, operators apply. The PR pipeline does the plan. Only the operator role can promote to apply. The on-call rotation is the source of truth for who has the operator role on a given day.
5. Break-glass admin is logged. A break-glass admin role exists, it is short-lived (15 minutes), it requires a second-person approval, and every action is logged to a separate, immutable audit trail.

Verification

The team boundary is verified when the four read-only commands above all return values consistent with the design. Specifically:

• Every state key in the backend begins with a known team prefix.
• Every path in CODEOWNERS corresponds to a known team prefix.
• The prod environment’s required reviewers list is small, named, and on-call.
• The IAM policy for the state bucket lists only the platform team’s CI role and the on-call rotation principal.

If any one of those four disagrees with the others, the boundary has a hole. Fix it before onboarding the next team.

What comes next

The next lesson is Multi-Account State Boundaries, which applies the team boundary to AWS accounts. The team boundary is who; the account boundary is where.