Blue/Green — atomic Service swap between two complete environments

Blue/green is the classic “two complete environments, switch the router” release pattern. Kubernetes does not implement blue/green natively — there is no strategy.type: BlueGreen field in the Deployment API — but the building blocks are there: two Deployments, one Service, one selector swap. This lesson covers how to assemble the pattern correctly and the production hazards that come with it.

The shape of blue/green

Two Deployments, each with its own Pods, sharing a Service whose selector is the release identifier:

flowchart TB
    subgraph "Namespace: prod"
      S["Service: web<br/>selector: {track: stable}"]
      B["Deployment: web-blue<br/>version: v1<br/>label: {track: stable}"]
      G["Deployment: web-green<br/>version: v2<br/>label: {track: preview}"]
    end
    S -->|selector matches| B
    G -.->|no match| S

At any moment, the Service selector chooses one Deployment’s Pods as routable. The other Deployment’s Pods run but receive no traffic — they exist for warm-up, smoke tests, and rollback.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-blue
  labels:
    track: stable
    app: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
      track: stable
  template:
    metadata:
      labels:
        app: web
        track: stable
    spec:
      containers:
      - name: web
        image: web:1.27.2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-green
  labels:
    track: preview
    app: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
      track: preview
  template:
    metadata:
      labels:
        app: web
        track: preview
    spec:
      containers:
      - name: web
        image: web:1.28.0
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
    track: stable   # change this line to switch
  ports:
  - port: 80
    targetPort: 8080

The Deployment selectors (app: web, track: stable vs app: web, track: preview) are immutable. The Service selector (app: web, track: stable) is mutable — the operator flips track: stable to track: preview to switch, and back to roll back.

The swap is atomic at the Service layer

kubectl patch service web --type=merge -p '{"spec":{"selector":{"app":"web","track":"preview"}}}' updates the Service’s selector in a single API server write. After the change, the Endpoints controller observes the new selector and rewrites the Endpoints object within milliseconds.

flowchart LR
    A["Before:<br/>Service selector: track=stable"] --> B[kubectl patch]
    B --> C["After:<br/>Service selector: track=preview"]
    C --> D["Endpoints controller<br/>rewrites Endpoints"]
    D --> E["kube-proxy / CNI<br/>programs dataplane"]
    E --> F["Traffic shifts to green<br/>blue continues running"]

There is no in-flight overlap of “half-old half-new” traffic inside the same selector. The Service routes 100% to blue or 100% to green.

The flip side: blue’s Pods continue running after the swap, incurring compute cost. In a cost-sensitive environment, blue is scaled down after the swap completes — but only after the operator is confident the green version is healthy.

Why blue/green is not a single-controller primitive

Kubernetes’ built-in Deployment strategy only knows how to roll forward. It cannot:

• Hold the old version warm while the new version runs.
• Switch traffic to the new version with a single decision.
• Roll back by flipping a selector.

These capabilities are added by progressive-delivery controllers — Argo Rollouts, Flagger, Knative — which implement blue/green, canary, and traffic-shifting as CRD-driven controllers. Plain Kubernetes Deployments give you the raw material; the controller is your responsibility.

flowchart TB
    subgraph "Native Kubernetes"
        D1[Deployment controller]
        D2[Service controller]
        D3[Endpoints controller]
    end
    subgraph "Progressive delivery"
        AR["Argo Rollouts / Flagger"]
        SVC["Service + Ingress / Gateway"]
    end
    D1 --> AR
    AR --> SVC
    AR -.->|rollback| D1

If you operate without Argo Rollouts, you implement blue/green by hand: keep both Deployments, manage the selector swap, and clean up the old Deployment after the rollout. This is mechanical work but it is straightforward.

The state problem

Blue/green is an application-layer switch. The data layer is still shared between blue and green unless explicitly handled.

The same hazard applies to message queues, caches, and any other shared mutable state. Two versions writing to the same Redis key can corrupt it; two versions reading the same Kafka topic with different consumer-group semantics can re-deliver or skip events.

Rollback: re-point the selector

# Switch to green
kubectl patch service web --type=merge \
  -p '{"spec":{"selector":{"app":"web","track":"preview"}}}'

# Verify green is healthy
kubectl get pods -l track=preview -n prod
kubectl logs -l track=preview -n prod --tail=200

# Roll back to blue
kubectl patch service web --type=merge \
  -p '{"spec":{"selector":{"app":"web","track":"stable"}}}'

Because blue’s Pods never went away during the swap, the rollback is an Endpoints rewrite — milliseconds, no Pod restart, no image re-pull. The trade-off: blue’s Pods held the old code in memory for the duration of the green deployment; the rollback resumes in the same state.

sequenceDiagram
    participant OP as Operator
    participant API as API server
    participant EP as Endpoints
    participant BLUE as Blue Pods
    participant GREEN as Green Pods
    OP->>API: patch Service selector=preview
    API->>EP: Service updated
    EP-->>GREEN: 100% traffic
    Note over BLUE: still running, idle
    OP->>API: detect problem
    OP->>API: patch Service selector=stable
    API->>EP: Service updated
    EP-->>BLUE: 100% traffic again

The cost

Blue/green doubles the resource footprint during the rollout. For a 6-replica service, 12 Pods run. In a capacity-constrained cluster this is unacceptable. The mitigations:

• Scale blue to 0 replicas immediately after the swap (and bring it back only if rollback is needed). Trade-off: the rollback requires a cold-start of blue (image already on the node, but container creation is not instant).
• Use taints / node pools so blue and green run on separate hardware. Trade-off: the cluster must have enough nodes for both.
• Use a node autoscaler so the cluster temporarily grows. Trade-off: autoscaler scale-up is not instant; the first green Pod may take minutes to schedule.

Quiz

Production discipline

• The Service selector is your traffic switch. Guard changes to it with code review. An accidental swap is an outage or a hotfix in reverse.
• Schema compatibility is a precondition. Every release must prove, by automated test or migration, that two versions can coexist on the same data layer.
• Validate green before the swap. Use a separate temporary Service (web-preview) with a different selector to smoke-test green against synthetic traffic before flipping the production Service.
• Clean up blue after the swap. Scale it to 0 replicas once green is stable for one full business cycle. Exceptions: stateful workloads where the rollback requires warm blue.
• Audit every swap. kubectl get events --field-selector reason=ServiceUpdated or admission-webhook logging of Service selector changes is the only record that survives the rollout.

Blue/green is the right pattern when the cost of a bad release exceeds the cost of running two replicas. It is not free, and it does not solve the data-layer problem. The operator owns both.