kubectl apply — last-applied-configuration and three-way merge

kubectl apply is the workhorse of declarative cluster management. Where create writes an object once and replace overwrites it, apply merges the manifest with what is already in the cluster. This lesson dissects how that merge works, what it preserves and overwrites, and why apply is the right command for GitOps-driven workflows.

The three inputs to apply

kubectl apply -f manifest.yaml reads the manifest and sends a request to the API server. The request is shaped by the result of a three-way merge between:

1. The manifest — what you declared in the YAML.
2. The last-applied-configuration — what you declared in the previous apply (stored as an annotation on the live object).
3. The live state — what the API server currently has, including fields set by other tools, controllers, and admins.

flowchart LR
    M[Manifest] --> A[Three-way merge]
    L[Last-applied annotation] --> A
    S[Live state] --> A
    A --> R[PATCH request to API server]
    R --> API[API server]

The merge produces a partial object containing only the fields that need to change. The API server applies that partial object as a PATCH, leaving fields you did not declare untouched.

How the merge resolves each field

For every field in the manifest, the merge asks:

The rule:

• Fields you declared and did not change → no-op.
• Fields you changed since last apply → set to the new manifest value.
• Fields you removed since last apply → removed (because they were in last-applied but not in the new manifest).
• Fields you never declared → left alone, even if they were set by other tools or controllers.

This last rule is the key property. If a controller (e.g., a HorizontalPodAutoscaler) adds a field to the live object, and your manifest does not declare that field, the field is preserved across apply calls.

The last-applied-configuration annotation

After apply, kubectl stores the manifest’s spec (and a few other fields) as an annotation:

kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"web","namespace":"team-a-prod"},"spec":{"replicas":3,...}}

This annotation is what apply reads on the next invocation to compute the merge. It is the record of what you declared last time, and it is what makes apply idempotent and reversible.

To inspect the annotation:

kubectl get deployment web -o jsonpath='{.metadata.annotations.kubernetes\.kubernetes\.io/last-applied-configuration}'

Or more readably:

kubectl get deployment web -o yaml | yq '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"'

If you delete this annotation manually, the next apply will behave like a fresh create: it will overwrite every field declared in the manifest, but cannot tell which other fields to preserve. Never edit this annotation.

apply vs create vs replace

Three commands write objects to the API server. They differ in when and how:

kubectl create -f manifest.yaml           # POST; fails if object exists
kubectl replace -f manifest.yaml          # PUT; replaces whole object
kubectl apply -f manifest.yaml            # PATCH; merges manifest with live

create is for one-shot bootstrap (e.g., create a namespace once). replace is for the rare case where you want to overwrite a whole object and have the API server reject unintended merges. apply is the standard GitOps command: the manifest is the source of truth, the live state may have additions you want to preserve, and the merge reconciles the two.

Common apply flags

# Dry-run — show what apply would do without doing it
kubectl apply -f manifest.yaml --dry-run=client
kubectl apply -f manifest.yaml --dry-run=server

# Validate the manifest without applying
kubectl apply -f manifest.yaml --validate=true

# Prune — delete objects that are no longer in the manifests
kubectl apply -f manifests/ --prune -l app.kubernetes.io/part-of=checkout

# Force conflict resolution (server-side apply only)
kubectl apply -f manifest.yaml --force-conflicts

# Apply from a directory
kubectl apply -f manifests/

# Apply from a URL
kubectl apply -f https://example.com/manifest.yaml

--dry-run=client runs the kubectl-side merge logic without contacting the API server. --dry-run=server runs the full admission pipeline against the API server and returns the result, but does not persist. Both are essential for CI: a failed dry-run catches schema errors before they reach the cluster.

--prune is the inverse of apply: it deletes objects that match the label selector but are no longer in any of the applied manifests. Use it for “manifest directory is the universe” workflows. Prune requires a label selector; without it, kubectl refuses to run (to avoid mass deletion).

The merge in practice

A worked example:

# deployment.yaml
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.1
        resources:
          requests:
            cpu: 100m

Apply, then edit live state to add replicas: 5:

kubectl apply -f deployment.yaml
kubectl scale deployment web --replicas=5

The live Deployment now has replicas: 5. The last-applied-configuration has replicas: 3. If you run kubectl apply -f deployment.yaml again:

The Deployment stays at 5 replicas. Your manifest says 3 but the live state has been modified — apply does not overwrite the manual change.

To force the manifest’s value:

kubectl apply -f deployment.yaml
kubectl scale deployment web --replicas=3   # or edit the manifest to 5

The pattern: when you want apply to win, your manifest must reflect the desired value. When you want to keep a manual change, your manifest must omit the field.

Cross-course references

• The Ansible course part XXXVI-Ansible-Drift covers the same source-of-truth discipline: the manifest in Git is authoritative; the live state converges to it.
• The Terraform course part XVII-Terraform-Drift covers state-vs-config drift; apply’s three-way merge is the Kubernetes equivalent of Terraform’s reconciliation.
• The GitOps course (CIII-Kubernetes-GitOps) covers ArgoCD/Flux; those controllers implement the same merge logic at the cluster level.

Quiz

Production discipline

• Use apply as the default for GitOps workflows. The manifest is the source of truth; apply reconciles the live state.
• Use apply --dry-run=server in CI. Catches schema, admission, and validation errors before they reach the cluster.
• Use apply --prune -l <specific-selector> carefully. A broad selector can mass-delete objects created by other tools.
• Never edit the last-applied-configuration annotation. It is the merge’s source of truth; corrupting it produces silent drift.
• Reflect manual changes back in Git. If you kubectl scale or kubectl set image outside of GitOps, commit the change to the manifest afterwards.