Test and validate — terratest, Molecule, kyverno tests and the cost of testing

The test stage is the fourth gate in the pipeline. Where the lint stage asks “is the configuration correct?”, the test stage asks “does the configuration do what it claims to do?”. The two questions sound similar but they are not. A configuration can be lint-clean and still wire the wrong modules together; a configuration can pass security scanning and still produce a load balancer that does not listen on the intended port. Tests are the stage that catches the bugs the static gates cannot.

What tests prove

A test in this context is a program that creates resources or simulated resources, exercises them, and asserts that the results match expectations. The categories are:

• Unit tests. A unit test exercises a single module or role in isolation. It does not contact the real cloud; it uses a mock provider or a local backend. It is fast, deterministic, and cheap.
• Integration tests. An integration test exercises a module against the real cloud, or against a local simulation that closely matches the cloud. It is slow, may be flaky, and costs real money.
• Conformance tests. A conformance test asserts that the configuration matches a policy. In Kubernetes the policy is enforced by an admission controller; the test is the assertion that the policy rejects the configuration. In Terraform the policy is a Sentinel or OPA rule; the test is the assertion that the rule produces the expected verdict.

terraform validate

The cheapest test in the Terraform pipeline is terraform validate. It parses every .tf file, builds the configuration graph, and checks internal consistency: every reference resolves, every required argument is present, every type matches. terraform validate does not contact the cloud and does not check behaviour; it checks structure. The validate step belongs to the test stage, not the lint stage, because it builds the configuration graph rather than walking individual rules.

Canonical test frameworks

For Terraform the canonical test framework is Terratest. Terratest is a Go library that writes infrastructure tests as Go programs; the test creates real Terraform-managed infrastructure, waits for it to converge, runs assertions against the live resources, and then tears the resources down. Terratest is an integration test framework in the strict sense: every test run costs real cloud spend.

For Ansible the canonical test framework is Molecule. Molecule spins up a container (typically Docker), applies the role or playbook to the container, and runs idempotency and convergence assertions against the container. Molecule tests are faster than Terratest because the resource under test is a container rather than cloud infrastructure, but they are slower than pure unit tests because each scenario is a fresh container.

For Kubernetes the canonical test framework is the Kyverno test framework for policy, and Conftest or gomega/ginkgo for resource assertions. Kyverno ships with a kyverno test command that runs a predefined set of resources against a policy and asserts which resources are admitted and which are rejected.

flowchart LR
    A[Test runner] --> B[Module under test]
    B --> C[Convergence check]
    B --> D[Assertion check]
    B --> E[Teardown]
    C --> F[Pass or fail]
    D --> F
    E --> F

The cost of tests

Tests are the most expensive stage in the pipeline after apply, and the cost grows with the kind of test:

• terraform validate — seconds. Free. Runs on every commit.
• Terratest unit tests with the mock provider — seconds to tens of seconds. Free. Can run on every commit.
• Terratest integration tests against a real cloud — minutes per test. Costs cloud spend on every run. Typically gated to run only on pull requests that touch the modules under test, and only on a schedule or a manual trigger.
• Molecule scenarios — tens of seconds per scenario for simple roles, minutes for complex ones. Free but slow.
• Kyverno policy tests — seconds. Free. Can run on every commit.

The production discipline is to match the cost of the test to the cost of the bug. A bug in a null_resource that does nothing in production is not worth a Terratest integration test. A bug in a module that provisions a production database is worth an integration test that costs three minutes and a few cents of cloud spend on every run.

What tests cannot prove

Even a comprehensive test suite cannot prove:

• Performance. A test that asserts a load balancer exists does not assert that it handles production traffic. Load testing is a different discipline.
• Security under attack. A test that asserts an IAM policy is restrictive does not assert that the policy is correct under adversarial conditions. Security testing is a different discipline.
• Cross-module behaviour at scale. A test that exercises a single module does not exercise the composition of many modules across an account or a region. Integration testing at scale is closer to staging than to CI.

Production discipline

Three rules govern the test stage:

1. Tests are committed alongside the code they test. A test in a separate repository is a test that drifts out of sync. The test, the fixture, the assertion, and the module under test all live in the same commit.
2. The test stage is reproducible. A test that runs locally but not in CI, or in CI but not locally, is a test that the team cannot debug. The CI image and the contributor’s environment use the same tool versions.
3. The test stage is gated by cost. An integration test that costs money does not run on every commit to main; it runs on pull requests that touch the affected modules, and it is documented so reviewers know what they are approving.

Cross-course references

• Terraform for Production Sysadmins — Part XXVII (Testing) covers Terratest in depth, including patterns for parallel test execution and teardown guarantees.
• Ansible for Production Sysadmins — Part XXXVIII (Testing) covers Molecule scenarios, the role-versus- playbook boundary, and idempotency assertions.
• Kubernetes for Production Sysadmins — Part XXXIV (PolicyTesting) covers Kyverno test patterns.

Quiz