Development environment — fast feedback and the failure modes that are acceptable

The development environment is the inner loop of the engineering team. It exists so that an engineer who has just written a line of code can see that line of code run against a representative system within seconds. The discipline of development is that it is cheap to break, fast to reset, and impossible to confuse with any environment that affects real users. Failure modes that would be incidents in production — data loss, downtime, leaked secrets — are acceptable experimentation costs in development.

What the development environment is for

Three properties define the development environment:

• Fast feedback. The round-trip from “I committed a change” to “I saw the change run” is measured in seconds, not minutes. A development environment with a ten-minute feedback loop is not a development environment; it is a staging environment with a faster name.
• Cheap to reset. A broken development environment is a development environment that is rebuilt from scratch. The cost of a reset is minutes, not days. The data in a reset environment is data no one needed to keep.
• Isolated from consequences. A development deploy cannot affect production data, production users, or production observability. The boundary between development and everything-else is the boundary that makes the speed possible.

gh workflow run deploy.yml --environment development

The command targets the development environment. The platform applies the development environment’s protection rules, deploys with the development identity, and records the deployment in the development environment’s history. The production environment is untouched.

flowchart LR
    A["Engineer pushes commit"] --> B["CI runs unit tests"]
    B --> C["Deploy to development"]
    C --> D["Engineer observes result"]
    D -->|"feedback in seconds"| E["Next change"]
    D -->|"broken"| F["Reset environment"]
    F --> E

The diagram is the inner loop: commit, test, deploy, observe. The loop is fast because the cost of a broken deploy is small — the environment is reset, the engineer fixes the bug, the loop continues.

What is acceptable to break

The development environment is the only environment in this course where the following failure modes are acceptable:

• Data loss. Wiping the development database is a reset, not an incident. The data was test data, the schema is in version control, the rebuild is automated.
• Service downtime. Taking the development environment down for ten minutes is a coffee break, not an outage. No customers are affected because no customers exist.
• Restarted pods. A development deploy that crashes and restarts fifteen times is a development deploy that needs fixing, not a production incident that needs paging.
• Leaked test secrets. A development secret store may contain fake API keys, dummy database credentials, and placeholder tokens. These are test fixtures; they are not the production credentials.
• Broken builds. A failed development deploy is a failing test, not an alert. The failure is local to the engineer who pushed the change.

The list is not a license to be careless in development. The list is the boundary that separates careless behaviour from consequential behaviour. A development deploy that wipes the production database is not on the list; a development deploy that wipes the development database is. The boundary is what matters.

What is not acceptable

Three failure modes are never acceptable in development, because they cross the boundary the environment exists to enforce:

• A development identity that can write to production. The IAM role attached to the development environment must have permissions scoped to development resources only. If a development deploy can modify a production resource, the boundary is broken.
• Production data in the development environment. Real customer data, real PII, real payment information must never appear in development. The boundary is enforced by the identity, by the network policy, and by the data pipeline that generates test fixtures.
• Confusion between development and production. A URL prefix, a banner, a colour, a label — anything that makes it visually obvious which environment an engineer is looking at. A change that ships to development believing it is production is a change that has crossed the boundary by accident.

The speed-and-isolation tradeoff

The development environment is fast because it is isolated. A development environment that shares a database with staging is slower (the staging tests fail when development breaks the schema). A development environment that shares a network with production is slower (the production alerts fire when development spikes the traffic). The isolation is what makes the speed possible; the speed is what makes the development environment useful.

Production discipline

1. The inner loop is the development environment’s job. Fast feedback is the primary requirement; every other property is a means to that end.
2. Reset is free in development. A development environment that takes more than ten minutes to reset is a development environment that needs automating.
3. Production data never appears in development. The boundary is enforced by identity and by data pipeline, not by trust.
4. The development identity cannot write to production. A cross-environment permission is a policy bug, not a feature.

Cross-course references

• This course, Part LVI-01 (Environments) establishes the boundary model this lesson narrows to the development case.
• This course, Part XLVIII-04 (Expressions and context) covers the workflow expressions that target the development environment by name.
• Ansible for Production Sysadmins — Part V (DevEnvironment) covers the analogous pattern for configuration management.
• Kubernetes for Production Sysadmins — Part VI covers namespace isolation, which is the cluster-level implementation of the development boundary.

Quiz