Environment isolation — dev, staging, prod, and the promotion pipeline

Environment isolation is the most common cluster boundary. Dev, staging, and prod are the standard environments; the promotion pipeline moves workloads from Git through each. This lesson walks the environments, the promotion pipeline, the differences in configuration, data, and access, and the operational discipline.

The standard environments

flowchart LR
    A[Dev] --> B[Staging]
    B --> C[Prod]
    A1[Synthetic data] --> A
    B1[Sanitized data] --> B
    C1[Real data] --> C
    A2[Developers full access] --> A
    B2[Limited access] --> B
    C2[Restricted access] --> C

The three standard environments:

• Dev. Where developers experiment. Synthetic data, full developer access, smaller resources, fast feedback.
• Staging. A production-like environment for validation. Sanitized (anonymised) production data, limited access, similar resources to production.
• Prod. The real environment. Real data, restricted access, full resources.

Each environment is a separate boundary — soft (namespaces within a cluster) or hard (separate clusters) depending on the driver.

The promotion pipeline

flowchart LR
    A[Git commit] --> B[CI pipeline]
    B --> C[Dev cluster]
    C --> D[Staging cluster]
    D --> E[Prod cluster]
    C --> C1[Automated tests]
    D --> D1[Integration tests]
    E --> E1[Smoke tests]

The promotion pipeline:

1. A commit to Git triggers the CI pipeline.
2. CI builds the image, runs unit tests, pushes the image to a registry.
3. CI updates the GitOps repository (or triggers an Argo CD sync) to deploy the new image to dev.
4. Automated tests run against dev.
5. On success, the pipeline promotes to staging.
6. Integration tests run against staging.
7. On success, the pipeline promotes to prod (with manual approval or progressive delivery).

The pipeline is the operational discipline that prevents “works on dev, breaks on prod.”

The configuration differences

flowchart LR
    A[Dev] --> A1["1 replica, 100m CPU, 256Mi memory"]
    B[Staging] --> B1["3 replicas, 500m CPU, 512Mi memory"]
    C[Prod] --> C1["5 replicas, 1 CPU, 1Gi memory"]

Each environment has different resource limits. Dev is small (cost-optimised); staging is medium (production-like); prod is large (production-scale).

The configuration differences are typically managed via Kustomize overlays (covered in part CV) or Helm values:

# base/deployment.yaml
spec:
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 512Mi

# overlays/dev/deployment.yaml
spec:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 256Mi

# overlays/prod/deployment.yaml
spec:
  replicas: 5
  resources:
    requests:
      cpu: 1
      memory: 1Gi

The data differences

Dev data is generated by fixtures; staging data is a sanitised copy of production (with PII removed); prod data is the real production data.

The access differences

flowchart TD
    A[Dev] --> A1["Developers: cluster-admin"]
    B[Staging] --> B1["QA, release managers: namespace-admin"]
    C[Prod] --> C1["On-call SRE: limited namespace access"]

Each environment has different access:

• Dev. Developers have cluster-admin. They can deploy, debug, restart, delete.
• Staging. QA engineers and release managers have namespace-admin. They can deploy and debug within their namespace.
• Prod. On-call SREs have limited access (debug, read); deployments require approval through the pipeline.

The access differences are enforced by RBAC and admission webhooks.

The operational failure modes

Environment isolation fails for predictable reasons:

• Staging not production-like. Staging with smaller replicas, different CNI, or different ingress does not validate production behaviour.
• Synthetic data in staging. Staging with synthetic data misses real-world load patterns and data shape issues.
• Production access too permissive. Developers with cluster-admin in production can deploy untested code; the pipeline is bypassed.
• Promotion without validation. The pipeline promotes to staging without running integration tests; the workload reaches prod broken.
• Drift between environments. Dev, staging, and prod have drifted manifests; the same workload behaves differently in each.

Quiz

Production discipline

Environment isolation in production rests on five non-negotiable elements:

• Staging mirrors production configuration. Same replica counts, StorageClasses, CNI, ingress, monitoring. Differences invalidate the staging test.
• Promotion pipeline enforces validation. The pipeline must run automated tests in dev and integration tests in staging before promotion to prod.
• Production access is restricted. No cluster-admin for developers in prod. Deployments go through the pipeline.
• Sanitised data in staging. No raw production data in non-production environments.
• Audit staging-vs-prod drift quarterly. Configuration drift between environments invalidates the staging test.

Environment isolation is the foundation of safe deployment. The discipline is to make each environment deliberately and to audit the differences regularly.