Disposable integration tests — ephemeral environments and what they catch

The unit and module test layer catches contracts, defaults, and idempotency — the classes of mistake the configuration declares. It cannot catch the classes of mistake that depend on the cloud’s behaviour: whether a policy evaluates to allow, whether a DNS record resolves, whether a security group rule order produces the intended traffic. The disposable integration layer exists to catch exactly those classes, by asking the live cloud or cluster what happens.

What “disposable” means

A disposable integration test creates real resources in a real environment, asserts on the result, and tears the resources down. The environment is ephemeral: it is provisioned for the test, used for the duration of the test, and destroyed at the end. The test never leaves persistent state in the cloud or cluster. The blast radius of a test that fails to clean up is bounded by the test’s resource scope; the test’s discipline is to bound that scope tightly and to clean up reliably.

The three canonical implementations:

• Terratest for Terraform. Applies a Terraform module to a real cloud account, asserts on the live resources via the cloud SDK, tears them down with defer terraform.Destroy.
• Molecule with a cloud driver for Ansible. Provisions a cloud VM (EC2, GCE, Azure), converges the role against it, verifies the system state, tears the VM down. Slower and more expensive than the docker driver.
• kubectl apply --dry-run=server for Kubernetes. Sends the manifest to the API server’s admission control chain without persisting it; reports what the server would have accepted.

Each implementation asks the live system, not the file. Each is more expensive than the layers below it. Each is the only layer that catches what only the live system can answer.

Terratest

A Terratest test runs terraform init and terraform apply, asserts on the live cloud state, and tears the resources down. The test lives in Go and uses the cloud SDK (AWS, GCP, Azure) to query the live state. The canonical pattern:

func TestS3BucketEncryption(t *testing.T) {
    t.Parallel()

    opts := &terraform.Options{
        TerraformDir: "../modules/s3-bucket",
    }

    defer terraform.Destroy(t, opts)
    terraform.InitAndApply(t, opts)

    bucketID := terraform.Output(t, opts, "bucket_id")
    actual := aws.GetS3BucketEncryption(t, aws.DefaultRegion, bucketID)

    assert.True(t, actual, "S3 bucket must have encryption enabled")
}

The defer is the difference between a test that cleans up and a test that leaves real resources running. The cleanup must run on every code path — including panic — which is why the defer is the first statement after the test options are constructed. A test suite without defer terraform.Destroy is a budget leak.

Terratest’s value over terraform test:

• It asks the cloud’s API, not the local planner. A test that asserts the bucket has encryption at the live API will catch a misconfiguration that the local planner might miss.
• It exercises provider-specific behaviour that the local planner does not simulate (eventual consistency, region-specific defaults, IAM policy evaluation).
• It catches the gap between “the plan said X” and “the cloud did Y”.

Terratest’s cost:

• Money. Real resources in a real account. A suite of fifty tests in parallel can produce a measurable bill.
• Time. terraform apply takes minutes for non-trivial modules.
• Flake. The live cloud is not deterministic. API throttling, transient errors, eventual consistency produce false failures.
• State. A test that fails to destroy leaves real resources running.

The discipline is to keep the corpus small and high-value: tests for the resources whose misconfiguration is dangerous and not visible from the configuration alone. Security-critical primitives (IAM policies, encryption at rest, replication), stateful resources, network resources.

kubectl apply —dry-run=server

The Kubernetes analogue of Terratest is kubectl apply --dry-run=server. The flag sends the manifest to the API server, which runs the manifest through the full admission control chain (mutating webhooks, validating webhooks, schema validation, OpenAPI schema validation) and reports the result. The server-side dry-run does not persist the resource; it returns what would have happened if the apply had run for real.

kubectl apply --dry-run=server -f manifest.yaml

The output is the resource the API server would have accepted, with any errors the admission chain raised. A webhook that rejects the manifest for missing labels, a schema that disallows a deprecated field, an OpenAPI validation that rejects an unknown property — all surface in the dry-run output.

What kubectl apply --dry-run=server catches:

• Webhook rejections that the client cannot see.
• Admission policy violations (Kyverno, OPA Gatekeeper) that the YAML linter does not know about.
• Server-side schema drift: a manifest that is valid against the offline OpenAPI schema but rejected by the cluster’s actual schema (custom resource definitions that have evolved, admission webhooks that mutate the schema).
• Resource quotas and limit ranges: a manifest that exceeds the namespace’s quota is rejected at admission.

What it does not catch:

• Runtime behaviour: whether the pod actually schedules, whether the image pulls, whether the readiness probe passes.
• Network behaviour: whether a Service actually routes to the pods.

The dry-run is a server-side gate that catches the admission-time failures. It is cheaper than an apply (no resource is created) and faster than a real integration test. It belongs on the per-PR pipeline for any team that ships Kubernetes manifests.

Molecule with a cloud driver

Molecule supports a cloud driver — EC2, GCE, Azure — that provisions a real VM, runs the lifecycle against the VM over the real network, and tears the VM down. The cost is dollars per scenario (the VM is billable by the minute), and the wall-clock time is minutes (the VM takes minutes to provision and to converge a non-trivial role).

The cloud driver is appropriate for:

• Roles that target a specific VM image that no container can match (a hardened AMI, a Windows VM, a RHEL image with SELinux enforcing).
• Roles whose behaviour depends on cloud-init, instance metadata, or other cloud-specific subsystems.
• Roles that the team cannot reproduce in a container.

The cloud driver is not appropriate for roles that work fine in a container; the docker driver is cheaper and faster.

flowchart TB
    A["Change"] --> B[Disposable integration layer]
    B --> C["Terratest - real cloud account"]
    B --> D["Molecule cloud driver - real VM"]
    B --> E["kubectl apply --dry-run=server - real cluster"]
    C --> F[Live cloud SDK assertion]
    D --> G[VM lifecycle, role converge, verify]
    E --> H[Server-side admission chain]

The cost

The disposable integration layer is the most expensive layer that runs automatically. The order-of-magnitude cost:

• Terratest per test: minutes of wall-clock, cents to dollars of cloud cost. A suite of fifty tests in parallel: an hour of wall-clock, dollars of cloud cost.
• Molecule cloud driver per scenario: tens of minutes, dollars. A suite of five scenarios: hours, tens of dollars.
• kubectl apply --dry-run=server per run: seconds, zero cloud cost. A suite of a hundred manifests: minutes.

The layer does not run on every PR. The discipline is to run it on a schedule (nightly) or on pre-merge to the default branch. A per-PR Terratest run is a budget event; a scheduled Terratest run is a CI gate.

What the layer catches

The disposable integration layer is the only layer that catches:

• Whether the cloud accepts the planned resource. A plan that parses cleanly can still be rejected by the cloud’s API.
• Whether the IAM policy evaluates correctly. The cloud’s policy simulator is the source of truth.
• Whether the DNS record resolves. Static analysis cannot test DNS; only the live resolver can.
• Whether the security group rules allow the intended traffic. Rule order, deny precedence, and protocol-specific behaviour are not visible from the configuration alone.
• Whether the manifest passes the cluster’s admission webhooks. Client-side dry-run cannot see what the server-side chain rejects.

The layer is not scoped to:

• Real-user behaviour in production. A disposable integration test that runs against a real account still runs against a synthetic scenario.
• Long-horizon drift. The test is ephemeral; it does not catch the drift that accumulates over months.

Production discipline

1. Every Terratest test begins with defer terraform.Destroy. Cleanup is the first statement, not an afterthought.
2. The disposable integration corpus is small and high-value. A test for every resource is over-testing; a test for security-critical primitives, stateful resources, and network resources is right-sized.
3. Disposable integration runs on a schedule, not on every PR. The cost is too high for per-PR cadence.
4. Disposable integration runs in a dedicated test account with billing alerts. The production account is not the place to run fifty RDS instances per day.
5. kubectl apply --dry-run=server is the per-PR Kubernetes gate. It catches admission rejections without persisting resources.

Cross-course references

• Terraform for Production Sysadmins - Part XXI (ModulePatterns) is the layer where Terratest-exercised modules live.
• Kubernetes for Production Sysadmins - Part XXXIII (AdmissionControl) is the production counterpart of kubectl apply --dry-run=server.
• This course, Part L (TerraformCI) - lesson git-cicd-gitops-l-05-terratest-and-integration-tests is the Terratest deep dive.
• This course, Part LII (KubernetesCI) - lesson git-cicd-gitops-lii-02-manifest-validation-kubeconform is the static counterpart of --dry-run=server.

Quiz