Terratest and integration tests — Go-based testing of real infrastructure

Every gate covered so far - fmt, validate, tflint, tfsec, checkov - is static. None of them creates real infrastructure. None of them asks the cloud whether the configuration is actually correct. A configuration that passes every static check can still produce a security group that allows the wrong traffic, a DNS record that does not resolve, or an IAM role that cannot assume the policy attached to it. The integration test layer exists to catch exactly this class of mistake. Terratest is the canonical tool for that layer.

What Terratest does

Terratest is a Go library that wraps Terraform and the cloud-provider SDKs. A Terratest test calls terraform init, then terraform apply, asserts on the live infrastructure (queries the cloud, not the plan), and finally calls terraform destroy (almost always via defer) regardless of whether the assertions passed.

A minimal test, asserting that an S3 bucket has encryption enabled:

func TestS3BucketIsEncrypted(t *testing.T) {
    t.Parallel()

    opts := &terraform.Options{
        TerraformDir: "../modules/s3-bucket",
    }

    defer terraform.Destroy(t, opts)

    terraform.InitAndApply(t, opts)

    bucketID := terraform.Output(t, opts, "bucket_id")
    actual := aws.GetS3BucketEncryption(t, aws.DefaultRegion, bucketID)

    assert.True(t, actual, "S3 bucket must have encryption enabled")
}

The test creates a real S3 bucket, queries its encryption configuration with the AWS SDK, asserts that encryption is enabled, and destroys the bucket on the way out. It runs in minutes and costs cents. It catches a class of misconfiguration that no static scanner can catch, because the static scanner reads the configuration while the live cloud may behave differently.

The cost

Terratest is not free in any sense:

• Money. Each test creates real cloud resources. A CI run with fifty Terratest tests in parallel can produce a measurable bill.
• Time. A terraform apply against a non-trivial module takes minutes. A test suite of fifty tests takes an hour. CI minutes are not free.
• Flake. The live cloud is not deterministic. API throttling, transient network errors, and eventual consistency produce test failures that have nothing to do with the change being tested.
• State. A failed defer terraform.Destroy leaves real resources running.

The categories that justify Terratest: security-critical primitives where a misconfiguration is dangerous and not visible in the configuration; stateful resources where encryption-at-rest, replication, or backup settings are silently wrong; network resources where static analysis cannot reason about topology.

The categories that do not justify Terratest: a module with comprehensive static coverage whose configuration is the entire behaviour; anything that would require spinning up a non-trivial environment to test.

What to test

The Terratest corpus should be small and high-value:

• Encryption is enabled and configured correctly. The most common category of cloud misconfiguration; the static scanners catch some of it, the live API is the source of truth.
• IAM policies evaluate correctly. A policy that parses cleanly can still deny the principal it was meant to allow. Terratest calls the cloud’s policy simulator and verifies the result.
• DNS resolves to the right address. Static analysis cannot test DNS; Terratest can.
• Module outputs match what downstream modules expect. A module that outputs endpoint when downstream code expects url is caught the first time a dependent test runs.

What not to test: the cloud provider’s own correctness (a test that asserts “S3 returns the object I put” is testing AWS, not the Terraform module); anything that requires more than a few minutes of cloud time; anything that requires simulating user behaviour (HTTP requests, browser clicks) - Terratest is for infrastructure, not for application-level integration.

The defer-and-destroy pattern

The cleanup pattern is the most important Terratest habit:

opts := &terraform.Options{
    TerraformDir: "../modules/example",
}

defer terraform.Destroy(t, opts)

terraform.InitAndApply(t, opts)

The defer runs when the surrounding function returns - which is always, even on test failure, even on panic. The destroy step is what guarantees that no real resources outlive the test. Skipping the defer is the most common Terratest mistake and the most expensive. The defer is paired with retry-and-timeout logic for the destroy step itself, because a destroy that fails halfway (an RDS instance that is still deleting, a VPC that has a leftover ENI) leaves resources behind. Terratest’s terraform.Destroy retries automatically with exponential backoff; relying on that retry is part of the discipline.

Production discipline

1. Terratest runs on a schedule, not on every PR. The cost is too high for the per-PR gate; nightly or pre-merge-to-default-branch is the right cadence.
2. Every test has defer terraform.Destroy as its first cleanup statement. Tests that leak resources are budget leaks.
3. The test corpus is small and high-value. Fifty cheap tests beat five expensive ones.
4. t.Parallel() is used for independent tests. Sequential runs that could be parallel waste CI minutes.

Cross-course references

• Terraform for Production Sysadmins - Part XXI (ModulePatterns) covers the module-level patterns that Terratest exercises.
• This course, Part XLIX (InfrastructureCI) - lesson git-cicd-gitops-xlix-05-test-and-validate is the general framing of the test-and-validate stage; this lesson is the Terraform-specific instantiation.
• This course, Part XLII (Secrets) - the Terratest runner needs cloud credentials; OIDC federation is the production way to provide them without long-lived keys.
• Linux for Production Sysadmins - Part XXXVIII (CostControls) covers billing alerts and quotas, which apply directly to a Terratest CI runner.

Quiz