Data Sources: Reading Without Managing

A team needs to deploy an EC2 instance into an existing VPC. The VPC was created by a different team, in a different Terraform project, months ago. The team does not want to import the VPC into their project; they just need its ID. The right tool is a data source. This lesson is the operational discipline of reading without managing.

What a data source is

A data source is a read-only reference to infrastructure that exists outside the current Terraform state. The provider’s Read API is called to fetch the values; the values are cached in state; the configuration references the cached values.

+----------------------------+
|   Existing infrastructure  |
|   (in provider API or      |
|    in another Terraform    |
|    project or in a manual  |
|    console)                |
+-------------+--------------+
              |
              | data "..." "..." { ... }
              v
+----------------------------+
|   Terraform state          |
|   (cached result)          |
+-------------+--------------+
              |
              | reference
              v
+----------------------------+
|   Managed resources        |
+----------------------------+

Three things distinguish a data source from a managed resource:

1. The block keyword is data, not resource.
2. The provider API called is Read, not Create / Update / Delete. Data sources do not manage; they observe.
3. The plan action is read, not create / update / delete.

A real example:

data "aws_ami" "ubuntu_2204" {
  most_recent = true
  owners      = ["099720109477"]  # Canonical

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-jammy-22.04-amd64-server-*"]
  }
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu_2204.id
  instance_type = "t3.small"
}

The data source block tells Terraform which AMI to find; the resource block consumes the result. The provider reads from the EC2 API; the result is cached in state.

Reading from a remote state

The terraform_remote_state data source reads outputs from another Terraform state file. It is the right tool for cross-project references where the producer is also Terraform-managed.

data "terraform_remote_state" "network" {
  backend = "s3"

  config = {
    bucket = "acme-tfstate-prod"
    key    = "network/terraform.tfstate"
    region = "eu-west-1"
  }
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  subnet_id     = data.terraform_remote_state.network.outputs.public_subnet_ids["a"]
  instance_type = "t3.small"
}

The producer exposes public_subnet_ids as an output block. The consumer reads it. The cross-project dependency is explicit.

count and for_each on data sources

Data sources accept the same meta-arguments as resources:

data "aws_ami" "by_role" {
  for_each = toset(["web", "worker", "db"])
  owners   = ["099720109477"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-jammy-22.04-amd64-server-${each.key}-*"]
  }
}

resource "aws_instance" "app" {
  for_each      = toset(["web", "worker"])
  ami           = data.aws_ami.by_role[each.key].id
  instance_type = "t3.small"
}

The data source is instantiated once per key. The result is a map indexed by key, referenced with the standard [key] syntax.

Failure modes in production

1. most_recent = true on a security-sensitive AMI. A team uses data "aws_ami" "ubuntu" with most_recent = true to pick the latest Ubuntu AMI. The next refresh finds a new AMI, the configuration changes from under them, the new AMI is a beta, the instance fails to boot. Symptom: terraform plan shows a replacement that the team did not intend. Recovery: pin the AMI by ID rather than relying on most_recent; use a tested AMI pipeline instead.

2. Data source returns no results. The filter excludes everything. The plan fails with Error: no results found for the given filter. Symptom: apply stops; the operator did not anticipate that the lookup could be empty. Recovery: broaden the filter, add a precondition block to validate the result.

3. IAM permissions insufficient. The Terraform role does not have ec2:DescribeImages. The data source read fails with an access denied error. Symptom: every apply fails on the data source. Recovery: expand the IAM policy to include the read permission for the data source’s API.

4. Drift in a remote state source. The producer changes an output. The consumer’s terraform_remote_state data source detects the change; the consumer’s plan shows differences. Symptom: an unrelated consumer plan shows changes every time the producer applies. Recovery: version the producer’s outputs, communicate changes to consumers, or use moved blocks to renumber without breaking references.

5. Stale data. The state caches the data source result. If the underlying infrastructure changes between refreshes, Terraform uses the cached value. Symptom: the plan looks clean, but the resource references a value that no longer exists. Recovery: run terraform plan -refresh-only to update the cache; verify the data source filter is still correct.

6. Cross-account data sources. A data source in account A reads from account B. The cross-account role is misconfigured. Symptom: the data source read fails with AccessDenied. Recovery: confirm the trust policy and the IAM role; test with aws sts assume-role first.

Security and performance

Security. Data sources can return sensitive values (database passwords, private keys, certificate ARNs). The values are stored in state in cleartext. Treat state as a sensitive artefact regardless of whether the data source is marked sensitive.

Performance. Every data source read is an API call to the provider. A plan with many data sources can be slow. Use for_each to batch reads where possible; cache the result in a local block if the data is used multiple times in the same configuration.

Promoting a data source to a resource

The rule is simple: if Terraform should manage the lifecycle of the infrastructure, it must be a resource, not a data source. Use terraform import to bring existing infrastructure under management.

terraform import aws_vpc.main vpc-0123456789abcdef0

The data source becomes a resource block, the import moves the existing infrastructure into state, and subsequent plans manage the lifecycle rather than reading it.

Use a data source when:

• The infrastructure is managed by a different system or team.
• You only need to reference the value once, and it does not change often.
• You are reading provider metadata (current AWS account ID, current caller identity).

Use a resource (and import) when:

• You are responsible for the lifecycle of the infrastructure.
• The configuration needs to set attributes, not just read them.
• The infrastructure should appear in drift detection and state locking.

What to do in production

• Pin AMIs and other lookup values to specific IDs where stability matters. Use most_recent only for non-production environments.
• Treat terraform_remote_state outputs as a contract. Version them.
• Audit IAM permissions for the Terraform role against every data source in use. The most common cause of failed applies is insufficient read scope.
• Use preconditions to validate data source results:

data "aws_ami" "ubuntu_2204" {
  most_recent = true
  owners      = ["099720109477"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-jammy-22.04-amd64-server-*"]
  }
}

lifecycle {
  postcondition {
    condition     = self.architecture == "x86_64"
    error_message = "The selected AMI must be x86_64."
  }
}

• Run terraform plan -refresh-only periodically to catch silent changes in data source results.

Verification

# 1. List data sources in the configuration
grep -rE '^data\s+"' .

# 2. Confirm the data source result is what the plan expects
terraform plan -out=tfplan
terraform show -json tfplan | jq '.resource_changes[] | select(.change.actions == ["read"]) | .address'

# 3. Refresh the data source cache and confirm no change
terraform plan -refresh-only

# 4. Inspect a data source value in state
terraform state list | grep '^data'
terraform state show 'data.aws_ami.ubuntu_2204'

# 5. Validate IAM permissions for the data source
aws sts get-caller-identity
aws ec2 describe-images --image-ids ami-0c55b159cbfafe1f0

A clean verification looks like:

$ terraform plan -refresh-only
No changes. Your infrastructure matches the configuration.

The data source cache matches reality. The next apply does not need to re-read.