Lab: Reading Existing Infrastructure with Data Sources

Objective

By the end of this lab you will have built the two-stack arrangement that every organisation running Terraform ends up with — a platform stack that owns shared infrastructure and an application stack that reads it — and you will have broken the join between them four times on purpose, so that you recognise each failure from its error text rather than from a guess.

You will also have found a password in a state file that was declared sensitive = true two directories away, and you will know the exact jq query that finds the next one.

Architecture

Two independent Terraform configurations, each with its own state, joined by one data source.

platform/                              app/
  main.tf                                main.tf
  terraform.tfstate  <---------------+   terraform.tfstate
  network.conf       <-----------+   |
                                 |   |
                                 |   +-- data "terraform_remote_state" "platform"
                                 |          reads the published outputs
                                 |
                                 +------ data "local_file" "network"
                                            reads a file that was never published

The two joins are deliberately different in kind, and the difference is the lab:

terraform_remote_state reads what the producer chose to publish. It is a contract. When the producer changes it, the producer knows.
data "local_file" reads a path inside the producer’s directory that the producer never advertised. It is coupling. When the producer changes it, the producer has no idea anyone cared.

Read local_file throughout as a stand-in for a cloud object: an S3 bucket the platform team happens to write to, a config file on an NFS share, a Consul key. The mechanism is identical and so is the failure.

Requirements

A Linux or macOS workstation with shell access and a writable $HOME.
Terraform 1.9.x or later. Every output below was captured on Terraform v1.9.8 with hashicorp/local v2.9.0. The postcondition block in Task 8 needs 1.2 or later and strcontains needs 1.5 or later.
jq, for reading state and plan JSON.
Outbound HTTPS to registry.terraform.io for the first terraform init in each directory.
No cloud account and no credentials. The local backend used here stores the producer’s state as a plain file, which is what makes the remote-state join inspectable with cat.

Scenario

The platform team owns the network. The application team owns the application. They are separate repositories with separate states, which is the arrangement everybody agrees is correct.

In the six months since that decision, the application stack has acquired two dependencies on the platform stack. One is the intended one: a terraform_remote_state data source reading published outputs. The other arrived in a hotfix, reads a file directly out of the platform repository’s working directory, and nobody outside the application team knows it exists.

Today the platform team is tidying up. They rename an output and rename a file. Both changes are, from where they sit, entirely internal.

Tasks

Task 1: Scaffold both stacks

LAB="$HOME/tf-data-sources"
mkdir -p "$LAB/platform" "$LAB/app"
cd "$LAB/platform"

terraform version

Write platform/main.tf. This stack owns one file and publishes two outputs:

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

resource "local_file" "network" {
  filename = "${path.module}/network.conf"
  content  = <<-EOT
    subnet_a = 192.0.2.0/24
    subnet_b = 198.51.100.0/24
  EOT
}

output "subnet_ids" {
  description = "Published contract: stable identifiers other stacks may reference."
  value = {
    a = "subnet-a"
    b = "subnet-b"
  }
}

output "db_password" {
  description = "Published contract: the application database password."
  value       = "correct-horse-battery-staple"
  sensitive   = true
}

Apply it:

Configuration changeplatform/— Creates one file inside the lab directory and writes two outputs to state.

$ terraform init && terraform apply -auto-approve

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

db_password = <sensitive>
subnet_ids = {
  "a" = "subnet-a"
  "b" = "subnet-b"
}

The password renders as <sensitive>. Note what that is and is not — ask for the machine-readable form:

Read-only / Safeplatform/— The same outputs, in the form a CI job would consume.

$ terraform output -json | jq -c '.'

{"db_password":{"sensitive":true,"type":"string","value":"correct-horse-battery-staple"},"subnet_ids":{"sensitive":false,"type":["object",{"a":"string","b":"string"}],"value":{"a":"subnet-a","b":"subnet-b"}}}

sensitive = true suppresses the value in Terraform’s human-readable rendering. It does not encrypt it, does not withhold it from -json, and does not keep it out of the state file. Confirm the last point directly:

jq '.outputs.db_password' terraform.tfstate

{
  "value": "correct-horse-battery-staple",
  "type": "string",
  "sensitive": true
}

The flag is stored beside the value, not instead of it. Hold that thought until Task 5.

Task 2: Build the consumer

Write app/main.tf. It has one managed resource and two data sources — one reading the contract, one reaching around it:

terraform {
  required_version = ">= 1.9.0"

  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

data "terraform_remote_state" "platform" {
  backend = "local"

  config = {
    path = "${path.module}/../platform/terraform.tfstate"
  }
}

data "local_file" "network" {
  filename = "${path.module}/../platform/network.conf"
}

resource "local_file" "app_config" {
  filename = "${path.module}/app.conf"
  content  = <<-EOT
    subnet = ${data.terraform_remote_state.platform.outputs.subnet_ids["a"]}
    db_password = ${data.terraform_remote_state.platform.outputs.db_password}
    network_block = ${trimspace(data.local_file.network.content)}
  EOT
}

cd "$HOME/tf-data-sources/app"
terraform init
terraform plan

Look at the first four lines of the plan before anything else:

data.terraform_remote_state.platform: Reading...
data.terraform_remote_state.platform: Read complete after 0s
data.local_file.network: Reading...
data.local_file.network: Read complete after 0s

Both data sources were read during plan, before any diff was computed. That timing is the single most useful thing to know about data sources, and Task 6 is about the one case where it does not hold.

The plan body shows the values already substituted:

  # local_file.app_config will be created
  + resource "local_file" "app_config" {
      + content              = <<-EOT
            subnet = subnet-a
            db_password = correct-horse-battery-staple
            network_block = subnet_a = 192.0.2.0/24
            subnet_b = 198.51.100.0/24
        EOT

The password is in the plan output in cleartext. If this plan ran in CI, it is now in the job log.

Task 3: Apply, and find the data sources in state

terraform apply -auto-approve

The lesson says the provider’s read result is cached in state. Confirm it, because it is the fact that makes everything in Task 5 follow:

Read-only / Safeapp/— Every resource in the consumer state, with its mode.

$ jq -r '.resources[] | "\(.mode)  \(.type).\(.name)"' terraform.tfstate

data  local_file.network
data  terraform_remote_state.platform
managed  local_file.app_config

Three entries, two of them mode: data. A data source is not a transient lookup that evaporates after the run — it is a state entry with the provider’s answer written into it. terraform state list shows the same three with a data. prefix on two of them.

Task 4: Watch the consumer plan change when nobody changed the consumer

Edit platform/main.tf so the first subnet identifier reads "subnet-a-rebuilt", and apply the platform stack. Then, without touching a single line of app/main.tf:

Read-only / Safeapp/— No consumer file changed. The plan is not empty. Content hashes filtered for width.

$ terraform plan | grep -v 'content_'

data.terraform_remote_state.platform: Reading...
data.terraform_remote_state.platform: Read complete after 0s
data.local_file.network: Reading...
data.local_file.network: Read complete after 0s [id=bd365b5261a088eb52450034aaec44c4e6e53933]
local_file.app_config: Refreshing state... [id=f3f6ce8d7847d22a8d85ae7aa92e2d1ec66110ad]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # local_file.app_config must be replaced
-/+ resource "local_file" "app_config" {
      ~ content              = <<-EOT # forces replacement
          - subnet = subnet-a
          + subnet = subnet-a-rebuilt
            db_password = correct-horse-battery-staple
            network_block = subnet_a = 192.0.2.0/24
            subnet_b = 198.51.100.0/24
        EOT
      ~ id                   = "f3f6ce8d7847d22a8d85ae7aa92e2d1ec66110ad" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

This is what “the producer’s outputs are a contract” means in practice. The application team’s next plan — for an unrelated change, on a Tuesday — now contains a replacement they did not author and cannot explain from their own diff.

Now try to freeze it:

terraform plan -refresh=false

The plan is identical: Plan: 1 to add, 0 to change, 1 to destroy.

Task 5: Find the password

The producer marked db_password sensitive. Search the consumer’s state for it:

Read-only / Safeapp/— Every path in the consumer state whose value contains the secret.

$ jq -r 'paths(scalars) as $p | select(getpath($p)|tostring|test("correct-horse")) | ($p|join("."))' terraform.tfstate

resources.1.instances.0.attributes.outputs.value.db_password
resources.2.instances.0.attributes.content

Two hits, and they are different in kind:

resources.1 is the cached terraform_remote_state read. The whole outputs object was copied into the consumer’s state, sensitive members included.
resources.2 is the managed local_file, whose content attribute the password was interpolated into.

Now check whether the sensitivity mark made the journey. Add a probe output to app/main.tf:

output "echo_password" {
  value = data.terraform_remote_state.platform.outputs.db_password
}

An output that carries a sensitive value must itself declare sensitive = true, or Terraform refuses to plan. This one plans:

Read-only / Safeapp/— A plain output re-exporting a value the producer declared sensitive.

$ terraform plan

Changes to Outputs:
  + echo_password = "correct-horse-battery-staple"

Sensitivity does not survive terraform_remote_state

The producer’s state records "sensitive": true beside the value. The consumer, on Terraform 1.9.8 with the local backend, receives the value without the mark: it renders in plan output, it can be re-exported by an output block that never declares itself sensitive, and no warning is emitted at any point.

The consequences are worth stating flatly, because each one is a real incident pattern:

The value is in the consumer’s plan output, and therefore in the CI job log of a repository whose access list is not the producer’s access list.
The value is in the consumer’s state file, twice, in cleartext.
Any downstream consumer of echo_password gets it in cleartext too, and the chain can be arbitrarily long.

The defence is not a flag. It is not putting the secret in an output at all. Publish the address of the secret — a secrets-manager path, a Vault mount, a KMS key id — and let each consumer read it with its own credentials and its own audit trail. sensitive = true is a shoulder-surfing guard for terminal output, not an access control.

Delete the echo_password output before continuing.

Task 6: The read that cannot happen at plan time

Every read so far happened during plan, which is why every derived value was known before the apply started. That guarantee has one exception, and it is worth seeing.

Create a third directory with a data source whose argument depends on a resource that does not exist yet:

mkdir -p "$HOME/tf-data-sources/deferred"
cd "$HOME/tf-data-sources/deferred"

terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

resource "local_file" "seed" {
  filename = "${path.module}/seed.txt"
  content  = "seeded\n"
}

data "local_file" "seed" {
  filename = local_file.seed.filename
}

output "seed_content" {
  value = data.local_file.seed.content
}

terraform init
terraform plan

  # data.local_file.seed will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "local_file" "seed" {
      + content              = (known after apply)
      + content_base64       = (known after apply)
      + filename             = "./seed.txt"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + seed_content = (known after apply)

The <= symbol and the comment say it outright: this read is deferred to apply. Everything computed from it becomes (known after apply), and that unknown propagates. A for_each keyed on a deferred data source cannot be expanded at plan time and Terraform will tell you so; a resource count derived from one has the same problem.

The operational point: a data source that reads something Terraform is also creating in the same run buys you an unreviewable plan. If a value matters enough to review, it must be readable before the apply — which means the thing producing it belongs in an earlier stack, not the same one.

Task 7: Break the contract

Back in the platform stack, rename the published output from subnet_ids to subnet_identifiers and apply. This is a rename of a name the platform team invented, inside their own repository. Then plan the consumer:

Read-only / Safeapp/— The consumer after the producer renamed one output.

$ terraform plan

Planning failed. Terraform encountered an error while generating this plan.


Error: Unsupported attribute

  on main.tf line 27, in resource "local_file" "app_config":
  27:     subnet = ${data.terraform_remote_state.platform.outputs.subnet_ids["a"]}
    ├────────────────
    │ data.terraform_remote_state.platform.outputs is object with 2 attributes

This object does not have an attribute named "subnet_ids".

This is the good failure. It happens at plan, it names the file and line, it says what the object does contain, and nothing was applied. Restore the output name before continuing.

Task 8: Break the coupling, then guard it

Now the other join. In the platform stack, rename the managed file from network.conf to net.conf and apply. Nothing about that change mentions the application team.

Read-only / Safeapp/— The consumer after the producer renamed a file it never published.

$ terraform plan

Planning failed. Terraform encountered an error while generating this plan.


Error: Read local file data source error

  with data.local_file.network,
  on main.tf line 20, in data "local_file" "network":
  20: data "local_file" "network" {

The file at given path cannot be read.

+Original Error: open ./../platform/network.conf: no such file or directory

Also a plan-time failure, also clear — but notice the asymmetry. The producer’s rename of an output was visible to them as a change to a block whose entire purpose is to be consumed. The rename of a file was not. Nothing in the platform repository records that anyone reads network.conf, and nothing ever will.

Restore the filename, then defend the coupling you cannot remove today. Add a postcondition to the data source:

data "local_file" "network" {
  filename = "${path.module}/../platform/network.conf"

  lifecycle {
    postcondition {
      condition     = strcontains(self.content, "subnet_a")
      error_message = "network.conf no longer declares subnet_a. The platform stack changed the shape of a file this stack reads directly."
    }
  }
}

A missing file is caught by the provider. A file whose shape changed is not — it reads fine and produces nonsense downstream. Prove the guard works: in the platform stack, change subnet_a to net_a inside the file content and apply, then plan the consumer:

Read-only / Safeapp/— The postcondition catches a shape change the provider read happily.

$ terraform plan

Error: Resource postcondition failed

  on main.tf line 25, in data "local_file" "network":
  25:       condition     = strcontains(self.content, "subnet_a")
    ├────────────────
    │ self.content is "net_a = 192.0.2.0/24\nsubnet_b = 198.51.100.0/24\n"

network.conf no longer declares subnet_a. The platform stack changed the
shape of a file this stack reads directly.

Your sentence, at plan time, naming the stack at fault. Without it the apply succeeds, app.conf is written with an empty subnet block, and the failure surfaces later somewhere with no connection to Terraform at all.

Restore subnet_a in the platform stack before the validation step.

Mapping this to a real change window

The two joins need different change processes, and treating them the same is how the second one bites.

The published contract. Producer outputs are an API. Give them the same courtesy: additive changes any time; a rename is a deprecation, so publish both names for one release cycle and delete the old one after consumers confirm. Before any output change, find the consumers — grep -rl terraform_remote_state across the estate, then read the config blocks for your state key. That grep is the whole discovery mechanism; there is no registry.

The coupling. A path read directly out of another team’s infrastructure has no contract and no notification. Two options, in order of preference: get the producer to publish the value as an output and delete the direct read, or — if that cannot happen this quarter — add the postcondition from Task 8 so the breakage lands in your plan with your error message. Record the coupling somewhere the producer will see it.

The window itself. A consumer plan can change with no consumer commit behind it, so “the plan was clean yesterday” is not evidence. Re-plan inside the window, immediately before applying, and use a saved plan file so the values you reviewed are the values that apply.

Validation

jq -r '.resources[] | "\(.mode) \(.type)"' app/terraform.tfstate lists three entries, two of them data.
The jq path query from Task 5 returns two paths in the consumer state that contain the producer’s password.
terraform plan in app/ reports No changes. Your infrastructure matches the configuration. once the platform stack is back to its original output names and file content.
You can state, from the error text alone, which of the two joins broke in Task 7 and which in Task 8.
The deferred/ plan shows will be read during apply and seed_content = (known after apply).
Removing the postcondition and re-running the Task 8 shape change produces a successful apply with a wrong app.conf — confirm that, then put the postcondition back. It is the difference the guard makes.

Expected Outcome

tf-data-sources/
├── platform/
│   ├── main.tf                two outputs, one managed file
│   ├── network.conf
│   └── terraform.tfstate      db_password stored with "sensitive": true
├── app/
│   ├── main.tf                two data sources, one postcondition
│   ├── app.conf
│   └── terraform.tfstate      db_password stored twice, in cleartext
└── deferred/
    ├── main.tf
    └── seed.txt

You have a working two-stack topology, the evidence that a data source lives in state, the evidence that a sensitive output does not stay sensitive across the boundary, and three error messages you will recognise on sight.

Troubleshooting

Error: Unsupported attribute on an output you are sure exists. The producer’s state has not been applied since the output was added. terraform_remote_state reads the state file, not the configuration — an output that exists only in the producer’s .tf files is invisible until the producer applies. Check with jq '.outputs | keys' platform/terraform.tfstate.

The consumer reads an empty outputs object. The path in the config block is wrong, and reading a state file that does not exist is not an error — you get an empty result and a confusing attribute failure downstream. path is resolved relative to the process working directory unless you anchor it with ${path.module}, which is why every example here does.

terraform destroy on the producer succeeds while consumers are live. Nothing links the two states, so Terraform cannot warn you. This is the structural cost of splitting stacks, and the only defence is ordering: destroy consumers first. There is no equivalent of a foreign key here.

A plan is clean locally and fails in CI. The data source read different values, because it read at plan time in a different environment — a different state file, different credentials, a different region. Compare terraform show -json | jq '.values.root_module.resources' between the two rather than comparing the configuration, which is identical by definition.

postcondition reports “Invalid function argument” on strcontains. That function needs Terraform 1.5 or later. On an older CLI use length(regexall("subnet_a", self.content)) > 0.

self is not recognised inside the condition. self is only available in postcondition, not in precondition. A precondition runs before the read and therefore cannot refer to its result; use it to check the inputs, and a postcondition to check the answer.

Cleanup

Everything this lab created lives under one directory. Nothing outside it was written, no credential was used, and no network service was contacted after terraform init.

Destroy in dependency order — consumers before producers. Terraform will not enforce this for you, and doing it the other way round leaves the consumer’s next plan failing on a state file that no longer exists:

Destructiveapp/ then platform/— Removes the files both stacks created. Lab data only.

$ terraform -chdir="$HOME/tf-data-sources/app" destroy -auto-approve

local_file.app_config: Destroying... [id=0e32bde77649b964b331fb5808a6d8b0606d5503]
local_file.app_config: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.

Destructiveplatform/— Removes the producer's managed file. Lab data only.

$ terraform -chdir="$HOME/tf-data-sources/platform" destroy -auto-approve

local_file.network: Destroying... [id=bd365b5261a088eb52450034aaec44c4e6e53933]
local_file.network: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.

terraform -chdir="$HOME/tf-data-sources/deferred" destroy -auto-approve

ls -la "$HOME/tf-data-sources"
rm -rf "$HOME/tf-data-sources"

Under the hood

terraform_remote_state is a data source in a builtin provider, terraform.io/builtin/terraform, which is why it needs no required_providers entry and no download. Its read opens the backend named in backend, fetches the state, and returns the state’s outputs map as one object attribute. It reads the whole state, not the outputs it was asked for — the filtering happens in your expression, afterwards.

That is the mechanism behind everything in Task 5. The consumer’s state entry for the data source holds the complete outputs object, because that is the value the read returned. Terraform’s sensitivity marks live in the plan and state metadata, alongside values, rather than travelling with the bytes; the producer’s mark is recorded in the producer’s state file and is not part of what the read returns. Nothing reconstructs it on the far side.

The read timing follows from the graph. Data sources are graph nodes like anything else. If every argument is known before the apply — literals, variables, other data sources — Terraform evaluates the node during plan and the result is a known value that the diff can use. If any argument depends on a managed resource with a pending change, the node cannot be evaluated yet, so Terraform defers it and marks every value derived from it unknown. -refresh=false does not enter into it: that flag governs whether Terraform re-reads managed resources to detect drift, and data sources are not drift, they are inputs.

The postcondition block attaches to the node and is checked immediately after the read resolves — at plan time for a plan-time read, at apply time for a deferred one. self refers to the resolved object, which is why a postcondition can inspect content that did not exist when the plan began.

What You Learned

A data source is a state entry, not a lookup. jq found mode: data rows in the consumer’s state holding the provider’s answer, and one of them held the entire producer outputs object.
Data sources are read during plan, every plan, including under -refresh=false. That is why a consumer’s plan can change with no consumer commit behind it, and why a saved plan file — not a refresh flag — is what makes a plan reproducible.
sensitive = true did not cross the boundary. The password rendered in the consumer’s plan, was re-exported by a plain output block without complaint, and sits twice in cleartext in the consumer’s state. Publish the location of a secret, not the secret.
Two joins, two failure profiles. A renamed output failed with the attribute the producer removed and the object that remains. A renamed file failed with a path — and the producer had no way to know anyone was reading it.
A postcondition converts a silent shape change into your error message at plan time. For a coupling you cannot delete yet, that is the cheapest defence available.
A data source that depends on a pending resource is read during apply, and everything derived from it becomes (known after apply), which is the same as saying the plan cannot be reviewed.

Why this matters

Splitting Terraform into a platform stack and application stacks is the right decision and almost everybody makes it. What nobody writes down is that the split converts an in-graph dependency, which Terraform enforces and shows you in a plan, into a data source, which it does not.

After the split, the producer can rename an output and no tool tells them who breaks. The consumer’s plan can change overnight with no commit to point at. A secret declared sensitive in one repository arrives unmarked in another and lands in a CI log. None of these are bugs, and none of them will be fixed, because each one follows directly from the two states being genuinely independent — which is the property the split was for.

The compensating controls are unglamorous: treat outputs as a versioned interface, grep for consumers before changing one, never publish a secret value where you could publish its address, and put a postcondition on every read you do not control. This lab is where you find out what each of those failures actually looks like, while the cost of finding out is a directory you are about to delete.

Objectives

Prerequisites