Test and staging environments — closer to production and the data question

Staging is the environment that exists to find the bugs production would find, before production finds them. The discipline of staging is that it mirrors production in topology - same services, same configuration shape, same dependencies, same orchestration - and differs from production in data. The difference is not laziness; the difference is the data question, which is the central discipline of this lesson.

What staging is for

Three properties define the staging environment:

• Topology parity with production. The same number of replicas, the same database engine, the same message broker, the same network topology, the same observability stack. A staging environment that uses SQLite where production uses Postgres is not a staging environment; it is a development environment with a slower name.
• Realistic enough to expose integration bugs. The failure modes staging is meant to catch are the ones that only appear when services talk to each other across boundaries - serialisation mismatches, schema drift, network partitions, retries that mask errors in development but expose them in staging.
• Cheap enough to reset, expensive enough to reset carefully. A staging reset is more expensive than a development reset; it requires redeploying every service, repopulating data, and re-running integration tests. The reset is not free, but it is bounded.

gh workflow run deploy.yml --environment staging

The command targets the staging environment. The platform applies the staging identity, the staging protection rules, and the staging secrets. The deploy is recorded in the staging environment’s history.

flowchart LR
    A["Same digest"] --> B["Staging deploy"]
    B --> C["Integration tests"]
    B --> D["Smoke tests"]
    B --> E["Load tests"]
    C --> F{"Bugs found?"}
    D --> F
    E --> F
    F -- "yes" --> G["Fix and rebuild"]
    G --> A
    F -- "no" --> H["Promote to production"]

The diagram is the staging gate: the same digest that survived integration, smoke, and load tests is the digest that gets promoted. If the staging tests find a bug, the digest is rebuilt; the previous digest never reaches production.

The data question

Three options exist for staging data. Each is a deliberate tradeoff with a different failure mode:

• Synthetic data. Generated by a fixture script; no relationship to real production data. Fastest to produce, easiest to reset, but does not exercise production-realistic queries or volume. A staging environment with synthetic data catches serialisation bugs and integration bugs; it does not catch performance bugs that only appear at production volume.
• Sanitised production data. A copy of production data with PII removed, payment information redacted, and identifiers hashed. Most realistic; exposes the bugs that depend on data shape and volume; but the sanitisation pipeline is itself a system that can fail, and a sanitisation failure leaks production data into staging.
• Fresh data. An empty database, populated by walking through the application’s first-run flow. Slowest to produce; useful for migration tests; useless for integration tests that depend on existing data.

The choice is not “which is best”; the choice is “which failure mode can this team tolerate?”. A team building a reporting tool needs realistic volume; synthetic data misses every performance bug. A team building a payment system needs sanitised data; synthetic data misses every edge case in the payment schema. A team running a migration test needs fresh data; sanitised data masks the migration failures the test is meant to catch.

What is acceptable to break

The staging environment accepts a narrower range of failure modes than development:

• Short downtime. A staging deploy that takes the environment down for ten minutes is a deploy the team waits out. No customers are affected because no customers use staging.
• Test data corruption. A staging migration that corrupts the test database is a staging migration that is reset and re-run. The data is not production data; the corruption is contained.
• Failed load tests. A staging load test that exposes a performance bug is a load test that has done its job. The bug is fixed; the digest is rebuilt; the new digest is re-tested.

The staging environment does not accept:

• Production data leaks. A staging environment that receives unsanitised production data has had a boundary failure. The audit trail is consulted; the sanitisation pipeline is fixed; the staging database is wiped and rebuilt from a known-good sanitisation.
• Cross-environment identity use. A staging deploy that runs with the production identity has crossed the boundary by accident. The audit log is consulted; the identity that ran the deploy is revoked.
• Persistent state changes that affect future tests. A staging deploy that modifies a configuration used by every subsequent test has introduced a flake. The configuration is reverted; the test is marked flaky; the fix is in the test, not in the production configuration.

The staging-to-production boundary

The boundary between staging and production is the last rehearsal before the real deploy. The discipline of the boundary:

• The digest that survives staging is the digest that deploys to production. No rebuild between staging and production; no tag re-write; no patch deploy that changes the bytes.
• The production identity is distinct from the staging identity. The deploy that promotes from staging to production runs as the production identity; the staging identity cannot write to production.
• The promotion is recorded in both histories. The staging environment records the promotion; the production environment records the deploy. The two records can be cross-referenced by digest.

Production discipline

1. Staging mirrors production in topology. Same services, same configuration shape, same dependencies.
2. Staging diverges from production in data. The data choice is a deliberate tradeoff between realism and leak risk.
3. The digest survives staging, not a rebuild. The promotion invariant applies at the staging boundary too.
4. The staging identity cannot write to production. Cross-environment identity use is a policy bug.

Cross-course references

• This course, Part LVI-02 (Development environment) establishes the speed-and-isolation tradeoff this lesson extends with topology parity.
• This course, Part XLV-03 (Promotion) covers the digest invariant that the staging boundary enforces.
• Terraform for Production Sysadmins — Part XV covers the data-pipeline pattern for sanitised staging data.
• PostgreSQL for Production Sysadmins — Part XII covers the database-refresh patterns that staging resets depend on.

Quiz