high riskservice affecting~60 min

Runbook: Deploy a Production Workload

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the namespace exists with the correct RBAC, NetworkPolicy and ResourceQuota applied
· Confirm the container image is pinned to a digest in the production registry: crane digest registry.internal/app@sha256:<digest>
· Confirm the registry credentials are available to the cluster: kubectl -n <ns> get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .
· Confirm DNS resolves the registry from a node: dig +short registry.internal @10.96.0.10
· Confirm any backing services (databases, queues) the workload uses are reachable from the namespace via NetworkPolicy
· Confirm cluster capacity can absorb replicas + maxSurge extra Pods of the workload size
· Confirm a change ticket is open, peer-reviewed and the change window current

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Author the manifests in a Git repo; never apply from a workstation clipboard
2Render the manifests: kubectl apply -k overlays/prod --dry-run=server -o yaml and inspect the rendered object set
3Diff against the live state: kubectl diff -k overlays/prod and have a second engineer review the diff
4Apply ConfigMap and Secret first (these rarely change once deployed): kubectl apply -k overlays/prod/config
5Apply the Deployment, Service, PDB, NetworkPolicy and probes: kubectl apply -k overlays/prod/workload
6Wait for the rollout to converge: kubectl rollout status deployment/<name> -n <ns> --timeout=10m
7Verify all replicas Ready: kubectl get deploy,rs,pod -n <ns> -l app=<name>
8Apply the Ingress or Gateway: kubectl apply -k overlays/prod/ingress
9Verify the Ingress/Gateway has an address and the backend health is healthy: kubectl get ingress -n <ns> and the controller logs
10Smoke-test from outside the cluster (or from a known client): curl the public URL with the production header and verify a 200 response
11Validate error rate, latency and request volume against the pre-deploy baseline for at least 10 minutes
12Capture the post-deploy evidence in the change ticket: rollout revision, image digest, replica counts, smoke-test response

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get deploy/<name> -n <ns> reports Ready replicas equal to desired, with the new rollout revision
✓kubectl rollout history deploy/<name> -n <ns> shows the new revision with the correct CHANGE-CAUSE
✓kubectl get pods -n <ns> -l app=<name> -o jsonpath='{.items[*].status.containerStatuses[*].ready}' returns true for every Pod
✓kubectl get endpoints -n <ns> shows every Pod IP behind the Service
✓kubectl get ingress -n <ns> -o jsonpath='{.status.loadBalancer.ingress}' returns the production address
✓kubectl get pdb -n <ns> reports Current Healthy at or above Min Available
✓curl -fsS https://app.prod.example/healthz returns 200 with the new version string
✓Observability dashboards show the new ReplicaSet and no error spike
✓kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -20 shows no Warning events since rollout complete

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the rollout stalls or replicas become unhealthy, kubectl rollout undo deployment/<name> -n <ns> --to-revision=<previous> returns to the prior revision
↶If the Ingress or Gateway misroutes, revert the manifest in Git and kubectl apply -k overlays/prod/ingress
↶If the workload is fundamentally broken (bad config, wrong secret), kubectl delete deploy/<name> -n <ns> and re-apply from a known-good revision
↶If the rollout succeeds but the smoke test fails, treat the deployment as failed and roll back even if Ready reports success
↶Always record the failing revision number so the next deploy does not silently retry it

6 · Escalation

When the runbook isn't enough, contact:

· The Deployment rollout completes but the application fails readiness: readiness probe is wrong for the workload; fix the probe and re-roll
· The Service has no endpoints after apply: selector does not match Pod labels; do not patch the Service by hand, fix the manifest and re-apply
· The Ingress controller reports 503 for every backend: backend service port mismatch or NetworkPolicy is blocking the controller; fix policy first
· Cluster capacity errors during surge: another workload is using headroom that no one reserved; re-plan the rollout to smaller surge and coordinate with capacity owners
· PDB blocks eviction during validation: PDB is misconfigured for the workload; the manifest is the bug, not the cluster

A production deploy is not a kubectl apply. It is a chain of evidence: the manifest matches what was reviewed, the rollout converged, the ingress serves, the smoke test passes, and the dashboards do not regress. Every step in the chain is what the rollback will rely on.

1. The production manifest set

A production deployment is several manifests, not one. The minimum set the runbook expects:

Deployment with replicas, strategy: RollingUpdate with explicit maxSurge/maxUnavailable, probes, requests and limits
Service matching the workload labels
PodDisruptionBudget with a realistic minAvailable (or maxUnavailable)
NetworkPolicy allowing only the expected ingress and egress
Ingress or HTTPRoute routing the public hostname to the Service
ConfigMap and Secret (often in a separate overlay)
HorizontalPodAutoscaler if the workload is autoscaled

# overlays/prod/workload/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: prod
  labels:
    app: web
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  progressDeadlineSeconds: 600
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: web
      containers:
      - name: web
        image: registry.internal/app@sha256:5b1d...
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests:
            cpu: '200m"
            memory: '256Mi"
            ephemeral-storage: '1Gi"
          limits:
            cpu: '1"
            memory: '512Mi"
        readinessProbe:
          httpGet:
            path: /ready
            port: http
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /live
            port: http
          periodSeconds: 30
          failureThreshold: 3
        securityContext:
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]

# overlays/prod/workload/service.yaml
  name: web
  namespace: prod
  selector:
    app: web
  ports:
  - name: http
    port: 80
    targetPort: http

# overlays/prod/workload/pdb.yaml
  name: web
  namespace: prod
  minAvailable: 4
  selector:
    matchLabels:
      app: web

2. Pre-flight evidence

Read-only / SafePre-flight evidence


kubectl diff -k overlays/prod/ -o yaml | tee /tmp/diff.yaml
kubectl apply -k overlays/prod/ --dry-run=server -o yaml | head -100

3. Apply in order

The order matters: configuration objects must exist before workloads that reference them, and workloads must exist before the Ingress that points at them. If you apply in the wrong order, the controller will report transient errors that look like real failures.

Read-only / SafeApply in order


kubectl get cm,secret -n prod | head -20

Read-only / SafeApply in order


# Wait for the rollout
kubectl rollout status deployment/web -n prod --timeout=10m

# Confirm replicas
kubectl get deploy,rs,pod -n prod -l app=web
kubectl get endpoints -n prod web

Read-only / SafeApply in order


kubectl get ingress -n prod -o wide
# If using Gateway API
kubectl get httproute -n prod -o wide

4. Smoke test from outside

Read-only / SafeSmoke test from outside


INGRESS_IP=$(kubectl get ingress <name> -n <ns> -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
HOST=app.prod.example

curl -fsS -H "Host: $HOST" --resolve "$HOST:443:$INGRESS_IP" https://$HOST/healthz
curl -fsS -H "Host: $HOST" --resolve "$HOST:443:$INGRESS_IP" https://$HOST/ | head -c 200

# Confirm version marker
curl -fsS -H "Host: $HOST" --resolve "$HOST:443:$INGRESS_IP" https://$HOST/version

A 200 is necessary, not sufficient. Confirm the version marker matches the new digest.

5. Post-deploy validation

Read-only / SafePost-deploy validation


# Expect:
#   - Error rate < 0.1% (pre-deploy baseline)
#   - p95 latency < 300ms (pre-deploy baseline)
#   - Request volume stable or rising (no traffic drop)

# Confirm in-cluster
kubectl logs -n prod -l app=web --tail=200 | grep -cE '5[0-9]{2} '

kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Common pitfalls

Symptom	Cause	Action
Deployment rollout stalls at the first batch	Readiness probe wrong for the new container	`kubectl describe pod` and the previous runbook `kubernetes-rb-deployment-rollout`
Service has no endpoints	Selector mismatch	`kubectl get svc -o yaml` and `kubectl get pod --show-labels`
Ingress returns 503	Backend service port mismatch or NetworkPolicy blocking the controller	`kubectl describe ingress` and the ingress controller logs
HPA reports unknown metric	Metrics Server not installed or not reachable	`kubectl get apiservice v1beta1.metrics.k8s.io`
PDB reports `Disallowed pods` during validation	PDB minAvailable exceeds current healthy count	Lower the PDB or scale up before validating

A deploy that passes the smoke test but fails the dashboard check is not done. Roll back, gather evidence, and re-roll when the cause is understood.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. The production manifest set

2. Pre-flight evidence

3. Apply in order

4. Smoke test from outside

5. Post-deploy validation

Common pitfalls

References