Ephemeral and preview environments — per-PR environments and the cost

An ephemeral preview environment is a deployment target created for a single pull request, scoped to that pull request, and destroyed when the pull request is closed. Every PR gets its own URL, its own database, its own deployment, and its own lifecycle. The benefit is that a reviewer can click a link and see the change running in a real environment, not in a screenshot, not in a code review, not in the reviewer’s imagination. The cost is cloud spend, identity sprawl, and lifecycle complexity. The discipline is to scope the pattern to the workloads that benefit, not to apply it as a default for every change.

What an ephemeral environment is

Three properties define an ephemeral preview environment:

• Per-PR identity. A unique environment name (often the PR number), a unique URL, a unique database name, a unique set of secrets. Two PRs do not share state; closing one PR does not affect another.
• Short-lived. The environment exists only while the PR is open. PR closed, PR merged, or PR stale for N days: the environment is destroyed. The resources are released; the cloud spend ends; the secrets are revoked.
• Production-like enough to be reviewable. The preview environment runs the same artifact as the PR branch would deploy to staging; the configuration is scoped to the preview; the data is synthetic or sanitised.

flowchart LR
    A["PR opened"] --> B["Create preview env"]
    B --> C["Deploy PR artifact"]
    C --> D["Unique URL"]
    D --> E["Reviewer clicks URL"]
    E --> F{"PR closed or merged?"}
    F -- "no" --> G["Continue updating"]
    G --> C
    F -- "yes" --> H["Destroy preview env"]
    H --> I["Release resources"]
    H --> J["Revoke secrets"]

The diagram is the preview lifecycle. PR opens, the environment spins up; PR closes or merges, the environment tears down. The whole lifecycle is automated; no engineer manually creates or destroys the environment.

gh workflow run preview.yml --ref feature/new-checkout

The command triggers a workflow that reads the PR branch, creates a preview environment named for the PR, deploys the branch’s artifact, and posts the preview URL as a PR comment.

The cost of ephemeral environments

Three costs apply. Each is real; each is bounded; each is worth measuring.

Cloud spend

Each preview environment consumes cloud resources: a Kubernetes namespace with compute, a managed database, a load balancer, a public IP. The cost per preview per day is small; the cost across hundreds of previews across hundreds of days is significant.

A team with twenty open PRs, each running a preview environment with a $5/day managed database and a $3/day load balancer, spends $160/day on previews alone. Open the same preview for a week of review and the cost is $560. Multiply by a team of fifty engineers with twenty PRs each, and the monthly bill for previews can rival the production bill.

The discipline is to bound the lifetime: a stale preview is a preview that has not received a commit in N days; the lifecycle automation destroys it. The discipline is also to use the smallest viable resources: a preview database does not need the production instance size.

Identity sprawl

Each preview environment needs its own identity: a unique OIDC subject, a unique IAM role, a unique set of secrets. The number of identities grows with the number of open PRs. A team with twenty open PRs has twenty preview identities in the cloud provider; a team with two hundred open PRs has two hundred.

The discipline is to scope identities narrowly: a preview identity that can read a single S3 bucket, write to a single namespace, and connect to a single database. The preview identity cannot reach production. The blast radius of a compromised preview is the preview’s resources, not the production environment.

Lifecycle complexity

Each preview environment must be created, updated, and destroyed. The creation step provisions resources; the update step redeploys on every push; the destruction step tears down resources, revokes secrets, and removes DNS records. The lifecycle automation is a system with its own bugs: a stuck destruction step leaks resources; a missing update step shows stale code; a forgotten DNS record points at a destroyed environment.

The discipline is to treat the lifecycle automation as production software: version-controlled, reviewed, tested, monitored. A lifecycle bug is a leak; a leak costs money; the cost is invisible until the bill arrives.

When ephemeral environments help

Three workloads benefit from preview environments:

• User-facing UI changes. A reviewer who can click through the new checkout flow, the new dashboard, the new error page is a reviewer who catches the bugs that only appear in the rendered UI. A code review cannot catch a misaligned button.
• API integration changes. A reviewer who can call the new endpoint with realistic data is a reviewer who catches the integration bugs that only appear at the network boundary. A code review cannot catch a serialisation mismatch.
• Cross-service changes. A reviewer who can see the new service talking to the existing services is a reviewer who catches the contract bugs. A code review cannot catch a downstream consumer’s broken parser.

When ephemeral environments hurt

Three workloads do not benefit:

• Pure infrastructure changes. A Terraform change that provisions a new IAM role has no preview URL; the preview would be the production plan output. The existing plan-on-PR pattern covers the review need without spinning up an environment.
• Cost-sensitive workloads. A preview environment that consumes $50/day of cloud resources for a one-day PR review is not worth the cost. The discipline is to measure the cost-per-PR against the bug-detection value; the ratio must be positive.
• Regulated workloads. A preview environment that contains sanitised production data, even briefly, is a preview environment that has data-leak risk. The discipline is to use synthetic data in previews and sanitised data only in staging.

Production discipline

1. Bound the lifetime. A stale preview is destroyed automatically; no manual cleanup.
2. Use the smallest viable resources. A preview database does not need the production instance size.
3. Scope the identity narrowly. A preview identity cannot reach production resources.
4. Treat the lifecycle automation as production software. Version-controlled, reviewed, tested, monitored.
5. Measure the cost. Cost per PR per day, average lifetime, total monthly spend.

Cross-course references

• This course, Part LVI-05 (Identity isolation) covers the per-environment identity model that previews multiply.
• This course, Part XLIX-06 (Plan and review) covers the plan-on-PR pattern that infrastructure changes use instead of previews.
• Kubernetes for Production Sysadmins — Part XXIV covers the namespace-as-preview pattern for cluster workloads.
• Terraform for Production Sysadmins — Part XXIX covers the workspace-per-PR pattern for infrastructure changes.

Quiz