Kustomize production discipline — render before apply, review, pin, sign

Kustomize production discipline is the operational rigour that turns a useful tool into production configuration. This lesson walks the safety nets, the anti-patterns, and the discipline.

The safety nets

flowchart LR
    A[Kustomize safety nets] --> B["kubectl kustomize: render before apply"]
    A --> C["PR review: review the rendered diff"]
    A --> D["Pin image tags: no latest"]
    A --> E["Test selectors: verify commonLabels"]
    A --> F["Argo CD: declarative GitOps"]
    A --> G["Staging test: apply to staging first"]

The safety nets:

• Render before apply. kubectl kustomize overlays/prod shows the rendered output. Review it before applying.
• PR review. The rendered diff is reviewed in the PR. The reviewer verifies the expected changes.
• Pin image tags. Never use :latest or no tag in production.
• Verify selectors. commonLabels with includeSelectors: true changes selectors; verify they still match Pods.
• Argo CD integration. Use Argo CD for GitOps-driven Kustomize deployments.
• Staging test. Always apply to staging first.

Render before apply

# Render the overlay
kubectl kustomize overlays/prod

# Diff against the current cluster
kubectl kustomize overlays/prod | kubectl diff -f -

# Apply only if the diff is what you expect
kubectl apply -k overlays/prod

The workflow:

1. Render the overlay with kubectl kustomize.
2. Diff against the current cluster state with kubectl diff.
3. Apply only if the diff is what you expect.

This is the discipline: render, diff, review, apply.

Pin image tags

images:
  - name: myapp
    newName: myapp
    newTag: "2.0.0"  # pinned, not latest
  - name: postgres
    newName: postgres
    newTag: "16.3"  # pinned, not latest

Always pin the image tag to a specific version. A floating tag (:latest or no tag) is unpredictable: the image may change without warning.

Verify selectors

# Before commonLabels
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp

# After commonLabels with includeSelectors: true
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
    env: prod  # added
spec:
  selector:
    matchLabels:
      app: myapp
      env: prod  # added — Pod template must also have this label
  template:
    metadata:
      labels:
        app: myapp
        env: prod  # added

The discipline: after adding commonLabels with includeSelectors: true, verify the selector still matches the Pod template. Mismatched selectors cause Deployments to fail to roll out new Pods.

Quiz

The operational discipline

Kustomize in production rests on five non-negotiable elements:

• Render before apply. Always kubectl kustomize first.
• Review the diff. The rendered diff is the operational signal.
• Pin image tags. No floating tags.
• Verify selectors. After commonLabels changes.
• Test in staging. Every overlay change goes through staging first.

Kustomize is production configuration. Treat it with the same rigour as any other production tool.