Production environment — the boundary and what production means in this course

Production is not “the most important server” or “the environment with the most users” or “the cluster the on-call rotation watches”. Production is the boundary where user trust, real data, real consequences, and real money meet the deployment pipeline. The discipline of this course is that production is a discrete boundary, not a point on a spectrum. The properties that change at the boundary are not degrees more than staging; they are categorically different.

What production means

Four properties define the production environment:

• Identity. The production identity is the only identity that can write to production resources. No developer laptop, no development environment, no shared CI credential can write to production. The identity is scoped, time-limited, and auditable.
• Blast radius. The worst-case impact of a bad production deploy is total service failure, customer-facing data loss, and regulatory exposure. The blast radius is not a multiplier of the staging blast radius; it is a different category.
• Observability. The production observability stack is the one the on-call rotation watches. Alerts on production metrics page real engineers at real hours; the noise floor is held to zero.
• Change discipline. Production changes follow a documented process: signed commits, reviewed pull requests, plan artefacts, required reviewers, wait timers, branch restrictions, audit logs. The discipline is enforced at the platform layer, not in the workflow.

gh environment create production
gh workflow run deploy.yml --environment production

The first command registers the production environment with the platform; the second command targets a workflow at it. The platform applies the production protection rules - required reviewers, wait timers, branch restrictions - before the deploy job starts.

flowchart TD
    A["Deploy requested"] --> B{"Production environment?"}
    B -- "yes" --> C["Apply protection rules"]
    C --> D{"Required reviewers approved?"}
    D -- "no" --> E["Job held"]
    E --> D
    D -- "yes" --> F{"Wait timer elapsed?"}
    F -- "no" --> G["Job held"]
    G --> F
    F -- "yes" --> H["Deploy with production identity"]
    H --> I["Audit log recorded"]
    B -- "no" --> J["Apply with environment identity"]

The diagram is the production boundary. Every deploy to production passes through the protection rules; the identity used is the production identity; the audit log records the deploy with the digest, the commit, the reviewers, and the timestamp.

The boundary is not a spectrum

A common confusion is to treat “production” as a point on a spectrum from “least important” to “most important”:

• Development = 1
• Staging = 5
• Production = 10

The framing is wrong. Production is not “ten times more careful than development”; production is a different kind of careful. A staging deploy that fails does not affect customers. A production deploy that fails affects customers. The difference is qualitative, not quantitative.

The properties that change at the boundary:

• A failed staging deploy loses the team’s time. A failed production deploy loses customer trust. The cost of failure is not the same in kind.
• A staging identity that is compromised can be revoked. A production identity that is compromised is a regulatory event. The remediation is not the same in kind.
• A staging observability gap is a missed test. A production observability gap is a missed incident. The consequence is not the same in kind.

What “prod-like” means

“Prod-like” is a term teams use for environments that mimic production in some respects but are not production. The term is useful when the mimicking is honest: a load test environment that mirrors production topology but receives synthetic traffic is prod-like; a development environment that shares a database with production is not prod-like, it is production under a misleading name.

The spectrum of prod-likeness:

• Fully prod-like. Production data, production traffic, production users. This is production; calling it prod-like is a euphemism that hides the boundary.
• Topology-prod-like. Production topology, synthetic data, synthetic traffic. Useful for load tests; not a substitute for production validation.
• Configuration-prod-like. Production configuration shape, but different scale or different data. Useful for configuration validation; not a substitute for production topology.
• Not prod-like. A development environment with a production-coloured banner. Misleading; should be labelled development.

The spectrum is real; the discipline is to be honest about where on the spectrum a given environment sits. Calling a development environment “prod-like” because it runs the same code does not make it production. The boundary is the boundary.

Production discipline at the boundary

The discipline that applies at the production boundary:

1. The production identity is the only identity that writes to production. Cross-environment identity use is a policy bug. The platform enforces this; the workflow does not.
2. Production changes follow the documented change process. No exceptions for hotfixes; the hotfix process is a faster version of the standard process, not a bypass of it.
3. The production observability stack is a separate stack. Alerts that fire in production do not fire in staging; alerts that fire in staging do not pollute the production on-call rotation.
4. Production data never appears outside production. Sanitised production data may appear in staging under controlled conditions; real production data does not.
5. The audit log is permanent. The audit log is the record the regulator reads; the audit log is the record the post-incident review reads; the audit log is the record the next engineer reads. The retention is measured in years, not weeks.

Production discipline

1. Production is a discrete boundary, not a point on a spectrum. The properties that change at the boundary are categorically different.
2. The production identity is the only identity that writes to production. Cross-environment identity use is a policy bug.
3. “Prod-like” is an honest description of the environment’s parity, not a synonym for production.
4. The audit log is permanent. Retention is measured in years.

Cross-course references

• This course, Part LVI-01 (Environments) establishes the boundary model this lesson applies to production.
• This course, Part LV-06 (Decision framework) covers the five criteria that determine whether a team is ready to remove the production gate.
• Linux for Production Sysadmins — Part XXXVI (ChangeWindows) covers the change-control discipline that production boundaries enforce.
• Kubernetes for Production Sysadmins — Part XXXV (GitOps) covers the GitOps controller’s read of production as a special cluster.

Quiz