Staging and production validation — the final layer and what it should catch

By the time a change reaches the staging and production validation layer, it has passed four cheaper gates: static, policy, unit and module, and disposable integration. The remaining layer exists to catch what those four could not, and to confirm that the change behaves correctly in the environment where real users will hit it. It is the most expensive layer, the slowest layer, and the last line of defence before a mistake reaches production.

What the layer does

The staging and production validation layer performs four kinds of check, each scoped to a different question:

• Smoke tests. The system is up, the entry point responds, the database is reachable, the dependencies are present. A fast, broad check that the deploy worked at all.
• Canary analysis. A subset of real traffic is routed to the new version; metrics (error rate, latency, saturation) are compared to the baseline. A change that degrades the user experience is rolled back before the full rollout.
• Drift detection. The live system is compared to the declared state. Resources that the configuration does not declare are flagged; declared resources that are missing are flagged. Catches out-of-band changes made by humans or by other automation.
• Rollback readiness. The previous version is still runnable, the database migrations are reversible, the configuration can be reverted. Confirms the safety net is in place before the change is irreversible.

flowchart LR
    A["Deploy to staging"] --> B[Smoke test]
    B --> C[Canary - 1% traffic]
    C --> D[Canary - 25% traffic]
    D --> E[Full rollout]
    E --> F[Drift detection on production]
    F --> G[Rollback readiness verified]

Each step is a separate gate. The smoke test runs against staging. The canary runs in production with a small slice of traffic. The drift detection runs continuously against the declared state. The rollback readiness is asserted before the change is promoted to the full rollout.

What this layer should catch

The staging and production validation layer is scoped to catch what only the real environment can see. Three categories:

• Runtime behaviour under real load. A change that passes every static, policy, unit-and-module, and disposable integration gate can still fail under production load: a memory leak that only manifests after an hour, a database query that is fast on a 100-row table and slow on a 10-million-row table, a connection pool that exhausts under concurrency. The cheaper layers cannot simulate production load; the staging layer can.
• Interactions with state that is only in production. A configuration change that interacts with a year-old dataset, with a feature flag toggled two years ago, with a manual out-of-band fix that no one remembers. The disposable integration tests cannot reproduce a year’s worth of state; the staging environment with production-like data can.
• The gap between “the configuration applies” and “the system serves traffic”. A deploy that completes successfully can still produce a system that does not serve: a pod that is running but not ready, a service that is registered but not routable, a DNS record that is created but does not resolve. The smoke test catches the deploy-but-does-not-serve case.

The layer is also a feedback loop. Every mistake it catches should be expressible as a gate lower in the pyramid. A smoke test that fails because a misconfigured timeout is too short is feedback that the timeout should be validated at the unit-and-module layer. A canary that fails because a memory leak only manifests at production concurrency is feedback that the disposable integration layer should run a load test.

What this layer should not catch

The layer is deliberately scoped to a narrow class of mistake. The classes it deliberately leaves to cheaper layers:

• Syntactic and schema errors. A staging environment that catches a YAML indentation error is a staging environment that has lost the pyramid. The static layer catches this in milliseconds.
• Policy violations. An S3 bucket without encryption should never reach staging; the policy layer catches it for free.
• Module contract mistakes. A default value that produces a misconfigured resource should never reach staging; terraform test catches it.
• Cloud-side acceptance issues. A policy that does not evaluate correctly should be caught by Terratest, not by a production smoke test.

The discipline is to keep the staging/production layer focused on the mistakes only it can catch. Every failure that propagates from a cheaper layer is a failure of pipeline discipline; the fix is to add the cheaper gate, not to keep catching the mistake at staging.

Smoke tests in practice

A smoke test is a small, fast, broad check that the deployed system is healthy. For a web service, the canonical smoke test is:

curl -fsS https://service.example.com/healthz

For a Kubernetes deployment, the canonical smoke test is:

kubectl wait --for=condition=ready pod -l app=service -n production --timeout=120s
kubectl get endpoints service -n production

The test fails if the readiness probe does not pass within the timeout, or if the service has no endpoints (a deploy that completed but did not actually register a routable pod). The cost is seconds; the value is the catch of a deploy-but-does-not-serve failure.

Canary analysis

A canary analysis routes a small slice of production traffic to the new version and compares metrics to the baseline. The canonical implementation in Kubernetes is a service mesh (Istio, Linkerd) or a deployment strategy with traffic splitting. The metrics to watch:

• Error rate. The new version should not produce more 5xx responses than the baseline. A regression is grounds for an immediate rollback.
• Latency. The new version should not produce a tail latency (p95, p99) materially worse than the baseline.
• Saturation. The new version should not produce a CPU, memory, or I/O saturation materially worse than the baseline.

The canary window is short — five minutes is common — and the rollback is automatic if the metrics cross the threshold. The cost is the engineering investment in the service mesh and the metric pipelines; the value is the catch of a class of mistake that only the production traffic pattern can see.

Drift detection

Drift detection compares the live system to the declared state and reports the difference. The canonical implementations:

• terraform plan against the live state. Reports resources that exist but are not in the configuration, and configuration that expects resources that no longer exist.
• Kubernetes manifests compared to the cluster state. Tools like kubectl diff or ArgoCD’s drift detection report the divergence.
• Ansible playbooks run with --check --diff against the live inventory. Reports the changes the playbook would make.

Drift detection runs on a schedule — daily, hourly, or continuous, depending on the cost of the comparison. The output is a report of what changed out-of-band; the response is to commit the change (if it was intentional) or to revert it (if it was not).

Production discipline

1. Smoke tests run on every deploy. A deploy without a smoke test is a deploy without verification.
2. Canary analysis is automatic, not manual. The threshold is configured; the rollback fires without human intervention.
3. Drift detection runs on a schedule. A weekly drift report is the floor; continuous drift detection is the production target.
4. Rollback readiness is asserted before promotion, not after the failure. The safety net is in place before the change is irreversible.
5. Every staging/production failure is fed back to the cheaper layers. A failure that the fmt gate could have caught is feedback to add the fmt gate to the per-PR pipeline.

Cross-course references

• Kubernetes for Production Sysadmins - Part XXIV (Canary) covers canary strategies in depth.
• Terraform for Production Sysadmins - Part XXXV (Drift) covers drift detection and remediation.
• This course, Part XLIX (InfrastructureCI) - lesson git-cicd-gitops-xlix-05-test-and-validate is the general framing of test-and-validate; this lesson is the production-specific instantiation.
• Linux for Production Sysadmins - Part XXXVIII (CostControls) covers the cost of running staging environments; the discipline of right-sizing the staging environment is part of the production framing.

Quiz