kubectl diff and server-side diff — preview before apply

The first rule of any apply workflow is “look before you leap.” kubectl diff is the look — it shows exactly what apply would change, field by field. This lesson covers the diff modes, when each is the right choice, and how to wire diff into the change-management pipeline.

kubectl diff — the local preview

kubectl diff -f manifest.yaml runs the same three-way merge that apply would, then prints the resulting change as a unified diff. It does not write to the cluster; it sends the merge result to a dry-run=server API request and compares what the API server would store against what it currently has.

kubectl diff -f deployment.yaml
kubectl diff -f manifests/
kubectl diff -f https://example.com/manifest.yaml

Output is a unified diff:

--- // LIVE Deployment/web (before)
+++ // LIVE Deployment/web (after)
@@ -8,7 +8,7 @@
     name: web
 spec:
-  replicas: 3
+  replicas: 5
   selector:
     matchLabels:
       app: web
   template:
@@ -22,7 +22,7 @@
       - name: nginx
-        image: nginx:1.27.1
+        image: nginx:1.27.2

The diff is colour-coded when the terminal supports it (--server-side=false is implicit). It is also the output that code-review tools consume; many GitOps controllers use the same diff format.

Client-side vs server-side dry-run

Two dry-run modes exist; they answer different questions:

kubectl apply -f manifest.yaml --dry-run=client      # local check
kubectl apply -f manifest.yaml --dry-run=server       # full pipeline, no persist
kubectl diff -f manifest.yaml                         # same as --dry-run=server

--dry-run=client runs entirely on the kubectl workstation. It does not contact the API server at all. It catches:

• Schema errors (typos, wrong field types)
• Merge conflicts that kubectl’s client-side logic can detect
• Static analysis errors (e.g., mismatched apiVersion for a kind)

--dry-run=server runs the full admission pipeline against the API server. It catches everything client catches plus:

• Admission webhook rejections (PodSecurity, OPA, etc.)
• Mutating webhook defaults
• ResourceQuota violations (often)
• Server-side schema strictness
• Webhook-allowed dry-run requests

Production discipline: always use --dry-run=server for the final CI gate. Client-side dry-run is fast and catches typos, but it cannot see what the admission controllers will do.

Diff in code review

The production workflow for manifest changes:

1. Engineer edits the manifest in a feature branch.
2. CI runs kubectl apply --dry-run=server -f manifests/ — fails if any manifest is invalid.
3. Engineer opens a PR. The reviewer runs kubectl diff -f manifests/ against a staging cluster.
4. Reviewer approves based on the diff.
5. Merge triggers a deploy to staging, then prod.

The diff in step 3 is the artefact the reviewer reads. It should show:

• Exactly the fields the engineer intended to change
• No accidental side effects (e.g., the engineer changed replicas while editing image)

If the diff shows unexpected changes, the engineer should fix the manifest before merging. This is the discipline that keeps changes scoped.

Diff in CI pipelines

The standard CI gate:

# .github/workflows/manifest-validate.yml
name: Validate manifests
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: azure/setup-kubectl@v4
    - name: Server-side dry-run
      run: |
        kubectl apply --dry-run=server -f manifests/
    - name: Diff against staging
      run: |
        kubectl diff -f manifests/ --context staging
      env:
        KUBECONFIG: ${{ secrets.STAGING_KUBECONFIG }}

Server-side dry-run is fast (no waiting for controllers); it catches schema and admission errors. The diff step requires credentials to the staging cluster; secrets management should restrict those credentials to read-only.

Diff with kubectl-neat

Default kubectl diff output includes server-defaulted fields (status, metadata.resourceVersion, managedFields, creationTimestamp). These fields change every apply and make the actual changes hard to see.

kubectl-neat (a krew plugin) strips these fields before diffing:

kubectl diff -f deployment.yaml | kubectl neat

Output is a clean diff of only the fields the operator changed. This is the right tool for code review.

When diff cannot tell the whole story

kubectl diff is powerful but it has blind spots:

• Generated fields. Some admission webhooks add fields based on inputs that are not visible to kubectl (e.g., a webhook that adds an annotation based on a label value). These fields appear in the diff as “removed” because the manifest does not declare them, but the live state has them from the previous webhook run.
• Status fields. kubectl diff does not diff status, but controllers write status. The diff is silent on status changes.
• Side effects of controllers. Applying a manifest that creates a Deployment causes the Deployment controller to create ReplicaSets and Pods. diff does not show those downstream creations.
• Resource quota. diff shows the change but does not show whether the change will fit inside the namespace’s ResourceQuota.

For these cases, you need more than diff. Production pipelines combine diff with:

• Admission webhook dry-run testing
• Pre-prod clusters with the same admission and quota configuration
• Manual review of controller-generated changes

Cross-course references

• The Terraform course part XVI-Terraform-Plan-Review covers plan output as a code-review artefact; kubectl diff is the same idea at the cluster level.
• The Ansible course part XXV-Ansible-CheckDiff covers --check mode; --dry-run=server is the same idea for Kubernetes.
• The GitOps course (CIII-Kubernetes-GitOps) covers ArgoCD’s diff view; it implements the same three-way merge.

Quiz

Production discipline

• Always run --dry-run=server in CI. Client-side dry-run is faster but cannot see what admission webhooks will do.
• Pipe diff through kubectl neat for code review. Strip defaulted fields so the diff shows only the change.
• Make staging’s quota and admission mirror production. Otherwise dry-run in staging does not predict production outcomes.
• Diff is necessary but not sufficient. A clean diff does not validate intent. Human review is required for changes that scale, delete, or re-architect.
• Test admission webhooks for dry-run correctness. Webhooks that ignore the dry-run flag will execute side effects during a CI dry-run, which can corrupt state.