The testing pyramid for IaC — five layers and what each one costs

Every infrastructure pipeline you have built in this course — fmt, validate, tflint, tfsec, ansible-lint, kubeconform, molecule, terratest, kubectl apply --dry-run=server — sits somewhere on a five-layer testing pyramid. The layers are ordered by cost and ordered inversely by frequency: the cheapest layer runs on every keystroke, the most expensive layer runs once before a production change. The discipline is to know which layer catches which class of mistake and to spend the most time on the layer that runs the most.

The five layers

The pyramid, from cheapest and broadest at the base to most expensive and narrowest at the top:

1. Static checks. terraform fmt, terraform validate, yamllint, ansible-lint, kubeconform, tflint, schema validation. Pure-file analysis: no cloud calls, no containers, no execution. Runs in milliseconds. Catches formatting, syntax, schema violations, missing FQCNs, undefined references.
2. Policy. tfsec, checkov, conftest (OPA/Rego), kyverno (for Kubernetes manifests). Rules over static checks. Runs in seconds. Catches policy violations that are not syntax errors: an S3 bucket without encryption, a security group open to 0.0.0.0/0, a privileged container, a missing resource limit.
3. Unit and module tests. terraform test (1.6+), molecule test with a single converge against an ephemeral host, Go-based unit tests for module internals. Runs in seconds to minutes. Catches contract violations between modules, default-value mistakes, role idempotency on a clean target.
4. Disposable integration tests. Terratest against a real cloud account, Molecule with a cloud driver, kubectl apply --dry-run=server against a real cluster. Runs in minutes. Catches the gap between what the plan says and what the cloud does: a policy that does not evaluate, a DNS record that does not resolve, a network ACL that blocks the intended traffic.
5. Staging and production validation. Smoke tests against a deployed environment, canary analysis, drift detection, post-deploy verification. Runs once per change. Catches the mistakes that survived every cheaper layer — usually because the cheaper layers could not see them.

flowchart TB
    A["Layer 1: Static checks - terraform fmt / ansible-lint / kubeconform"] --> B["Layer 2: Policy - tfsec / checkov / conftest"]
    B --> C["Layer 3: Unit and module tests - terraform test / molecule"]
    C --> D["Layer 4: Disposable integration - Terratest / Molecule cloud driver"]
    D --> E["Layer 5: Staging and production validation - smoke / canary / drift"]

The pyramid is wide at the bottom because static checks are cheap and should be many; it is narrow at the top because integration tests are expensive and should be few. A pipeline that inverts the shape — many Terratest runs, few static checks — is a pipeline that is paying for things it could catch upstream.

The cost of each layer

The reason the pyramid is the right shape is in the table. The cheapest layer catches the cheapest bugs. A YAML formatting mistake should never reach a Terratest run; a Terratest run costs a dollar per execution and a YAML formatting mistake is free to catch at the fmt gate. The pyramid pushes mistakes to the cheapest layer that can catch them, which is the layer where the cost of finding them is lowest.

What to test where

The mapping from a class of mistake to the layer that catches it:

• Formatting and style. Static. Never let it reach any other layer.
• Schema and syntax. Static. terraform validate, kubeconform, ansible-playbook --syntax-check. The parser knows.
• Policy violations that are syntactically valid. Policy layer. An S3 bucket that parses cleanly and is missing encryption.
• Module contracts and idempotency. Unit and module layer. terraform test for input/output contracts, Molecule for role idempotency.
• Live cloud behaviour. Disposable integration. The only layer that asks the cloud.
• Real-user behaviour in production. Staging and production validation. The only layer that asks production.

Why the pyramid shape is the right shape

The shape is not arbitrary. The base of the pyramid is wide because static checks are cheap and should be exhaustive: every commit, every file, every keystroke. The top of the pyramid is narrow because integration tests are expensive and should be selective: only the changes where a live test is justified by the cost of the bug it might catch. A pipeline with the shape inverted — wide at the top, narrow at the base — is a pipeline that spends dollars to catch bugs that milliseconds could catch.

The shape is also resilient. A failure at the base — a fmt failure — short-circuits the pipeline before the expensive layers run. The pyramid is designed so that the cheapest layer fails first and the most expensive layer runs last; that is the same property the application testing pyramid has had for two decades.

Production discipline

1. Every change passes every layer below the layer that catches its class of mistake. A change that has not passed terraform fmt should never reach tfsec; a change that has not passed tfsec should never reach Terratest.
2. The expensive layers are scheduled, not per-PR. Terratest runs on a cadence, not on every commit.
3. A failure at any layer is a hard gate. The exit code of the cheapest layer is the only exit code that matters for that layer.
4. The pyramid is documented. The pipeline-as-code shows the layers; the runbook names what each layer catches.

Cross-course references

• Terraform for Production Sysadmins - Part XXI (ModulePatterns) is the layer where module contracts live; the LIV-03 lesson covers them.
• Ansible for Production Sysadmins - Part XXVI (Testing) covers Molecule; the LIV-03 lesson is the pipeline integration.
• This course, Parts L (TerraformCI), LI (AnsibleCI), LII (KubernetesCI) - the earlier parts cover the static and policy layers; this part frames how they compose.
• Kubernetes for Production Sysadmins - Part XXXIII (AdmissionControl) covers Kyverno and OPA, which are the policy layer for Kubernetes manifests.

Quiz