Lab 20: Reconcile a Kubernetes application through Argo CD with `syncPolicy.automated`

Objective

By the end of this lab you will have authored the artefacts that configure Argo CD for automated reconciliation: an Application CR with syncPolicy.automated and selfHeal: true, tuned syncOptions for prune, replace, and namespace creation, a drift-test script that manually edits the cluster, the observation document that records the auto-reconciliation, and a failure-mode catalogue for automated sync.

The point of this lab is not the Application CR syntax — the lab in Lesson LXXIX-02 covered automated sync. The point is the behaviour: the difference between automated and selfHeal, the blast radius of prune: true, and the failure modes that automated sync introduces (the controller is now the source of truth, and a bug in the Git source is a bug in production).

Architecture

A single Application CR with syncPolicy.automated and selfHeal: true. The application controller polls the Git repository every 3 minutes (the default reconciliation interval) and applies any drift it detects. selfHeal ensures that drift from outside Git (manual kubectl edit) is also reconciled.

flowchart LR
    A["Git repo\napp-source/"] --> B["argocd-repo-server"]
    B --> C["argocd-application-controller"]
    C --> D["in-cluster\nsample-app"]
    D -- "manual edit (drift)" --> C
    C -- "reconcile" --> D

The controller polls Git, compares the desired state to the actual state, and applies any difference. The polling interval is configurable (syncPolicy.retry and the global timeout.reconciliation); the default is 3 minutes.

Requirements

• A kind cluster with Argo CD installed (Lab 19).
• The sample application from Lab 19, with the Application CR pointing at the Git repository.
• kubectl and argocd CLI authenticated against the cluster.

Scenario

A platform team has Argo CD installed (Lab 19) with a manual sync policy. They want to enable automated sync for the sample application so that any change to the Git repository is applied within 3 minutes (the default reconciliation interval). They also want to enable selfHeal so that any drift from manual cluster edits is reverted. The lab builds the configuration, drift-tests it, and documents the behaviour.

Tasks

Task 1 — Build the automated Application CR

LAB="$HOME/argocd-reconcile-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > application-automated.yaml <<'EOF'
# application-automated.yaml — the Application CR with
# syncPolicy.automated and selfHeal: true.
#
# The CR is the same as Lab 19's application.yaml, with two
# additions:
#
#   - syncPolicy.automated: { prune: true, selfHeal: true }
#   - syncOptions: CreateNamespace=true, PruneLast=true,
#     ApplyOutOfSyncOnly=true, ServerSideApply=true
#
# The syncOptions are tuned for the team's preferences:
#
#   - CreateNamespace=true: the target namespace is created if
#     it does not exist. Required for the team's pattern of
#     per-environment namespaces.
#   - PruneLast=true: prune happens *after* the apply. If the
#     apply fails, prune is skipped. Reduces the blast radius
#     of a bad sync.
#   - ApplyOutOfSyncOnly=true: only resources that are out of
#     sync are re-applied. Reduces reconciliation time on
#     large applications.
#   - ServerSideApply=true: use the API server's
#     server-side-apply, which is the safer default for
#     concurrent controllers.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/runbook-academy/argocd-reconcile-lab.git
    targetRevision: main
    path: app-source
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true       # delete resources removed from Git
      selfHeal: true    # revert manual cluster edits
      allowEmpty: false # do not delete all resources if Git is empty
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true
      - ServerSideApply=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
EOF

git add application-automated.yaml
git commit -m 'argocd: automated application CR with self-heal'

The CR has syncPolicy.automated.prune: true (Argo CD deletes resources removed from Git), selfHeal: true (Argo CD reverts manual cluster edits), and four syncOptions that tune the behaviour.

Task 2 — Build the sample app source

# check-shell-blocks: allow-invalid
cd "$HOME/argocd-reconcile-lab"

mkdir -p app-source

cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
EOF

cat > app-source/web-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
EOF

cd app-source
git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'
git add web-deployment.yaml web-service.yaml
git commit -m 'initial: web app manifests'
cd ..

git add app-source/
git commit -m 'argocd: sample app source for automated sync'

The sample app is the same as Lab 19. The lab’s reconciliation test will edit the cluster and observe Argo CD reverting the edit.

Task 3 — Author the drift-test script

# check-shell-blocks: allow-invalid
cd "$HOME/argocd-reconcile-lab"

cat > drift-test.sh <<'EOF'
#!/usr/bin/env bash
#
# drift-test.sh — manually edit the cluster to cause drift,
# then observe Argo CD auto-reconciling.
#
# The script performs three edits:
#
#   1. Scale the Deployment to 5 replicas (drift: desired=2,
#      actual=5).
#   2. Change the container image tag (drift: desired=1.27-alpine,
#      actual=:latest).
#   3. Add a label that is not in Git (drift: cluster has extra
#      label).
#
# After each edit, the script waits for Argo CD's reconciliation
# (default 3 minutes) and asserts the drift is reverted.
#
# Required: Argo CD installed and the sample application
# registered with syncPolicy.automated and selfHeal: true.

set -euo pipefail

APP_NAME="\${APP_NAME:-web}"
ARGOCD_NS="\${ARGOCD_NS:-argocd}"
TARGET_NS="\${TARGET_NS:-production}"
WAIT_FOR_RECONCILE="\${WAIT_FOR_RECONCILE:-200}"

# ─────────────────────────────────────────────────────────────────
# Test 1: scale the Deployment
# ─────────────────────────────────────────────────────────────────
echo "=== Test 1: scale drift ==="
kubectl scale deployment web -n "$TARGET_NS" --replicas=5
sleep 5
echo "before reconciliation:"
kubectl get deployment web -n "$TARGET_NS" -o jsonpath='{.spec.replicas}'

echo "waiting ${WAIT_FOR_RECONCILE}s for Argo CD to reconcile..."
sleep "$WAIT_FOR_RECONCILE"

echo "after reconciliation:"
kubectl get deployment web -n "$TARGET_NS" -o jsonpath='{.spec.replicas}'
echo ""

# ─────────────────────────────────────────────────────────────────
# Test 2: change the image tag
# ─────────────────────────────────────────────────────────────────
echo "=== Test 2: image tag drift ==="
kubectl set image deployment/web web=nginx:latest -n "$TARGET_NS"
sleep 5
echo "before reconciliation:"
kubectl get deployment web -n "$TARGET_NS" \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
echo ""

echo "waiting ${WAIT_FOR_RECONCILE}s for Argo CD to reconcile..."
sleep "$WAIT_FOR_RECONCILE"

echo "after reconciliation:"
kubectl get deployment web -n "$TARGET_NS" \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
echo ""

# ─────────────────────────────────────────────────────────────────
# Test 3: add a label
# ─────────────────────────────────────────────────────────────────
echo "=== Test 3: label drift ==="
kubectl label deployment web -n "$TARGET_NS" extra=drift
sleep 5
echo "before reconciliation:"
kubectl get deployment web -n "$TARGET_NS" \
  -o jsonpath='{.metadata.labels.extra}'

echo "waiting ${WAIT_FOR_RECONCILE}s for Argo CD to reconcile..."
sleep "$WAIT_FOR_RECONCILE"

echo "after reconciliation:"
kubectl get deployment web -n "$TARGET_NS" \
  -o jsonpath='{.metadata.labels.extra}'
echo "(empty = reconciled)"
EOF

chmod +x drift-test.sh

git add drift-test.sh
git commit -m 'argocd: drift test script'

The drift-test script performs three manual edits and observes Argo CD reverting each one. The default wait is 200 seconds (3 minutes + buffer) to match the default reconciliation interval.

Task 4 — Observe the reconciliation

# check-shell-blocks: allow-invalid
cd "$HOME/argocd-reconcile-lab"

cat > reconcile-observation.md <<'EOF'
# Reconcile observation

This document is the canonical record of what Argo CD does
when the cluster drifts from Git. The drift-test script is the
implementation; this document is the recorded observation.

## Test 1: replica count drift

=== before reconciliation === spec.replicas: 5 (kubectl scaled to 5)

=== after reconciliation (3 minutes) === spec.replicas: 2 (Argo CD reverted to Git’s value)

**What Argo CD did:**

1. Detected the diff between the Git state (`replicas: 2`) and
   the cluster state (`replicas: 5`).
2. Applied the Git state via server-side apply.
3. Reported the resource as `Synced: True, Healthy: True` after
   the reconciliation.

## Test 2: image tag drift

=== before reconciliation === spec.template.spec.containers[0].image: nginx:latest

=== after reconciliation (3 minutes) === spec.template.spec.containers[0].image: nginx:1.27-alpine

**What Argo CD did:**

1. Detected the image tag change.
2. Applied the Git's image tag (`nginx:1.27-alpine`).
3. Triggered a rolling update because the container spec
   changed. The Pods were restarted with the new image.

## Test 3: label drift

=== before reconciliation === metadata.labels.extra: drift

=== after reconciliation (3 minutes) === metadata.labels.extra: (empty; Argo CD removed the extra label)

**What Argo CD did:**

1. Detected the extra label.
2. Applied the Git's labels (which do not include `extra`).
3. The label was removed.

## Why this matters

The three tests demonstrate that `selfHeal: true` is
**aggressive**: any cluster state that does not match Git is
reverted. This is the intended behaviour for a GitOps system,
but it has implications:

- A developer who debugs by editing a Deployment via
  `kubectl edit` will see their edits reverted on the next
  reconciliation.
- An operator who performs an emergency change (for example,
  scaling up during an incident) will see the change reverted.
- An external controller that manages the same resource will
  fight with Argo CD.

## Mitigation

The team's discipline is:

1. **All changes go through Git.** A change that is not in Git
   is, by definition, drift.
2. **Emergency changes are exempted explicitly, temporarily.**
   Before an emergency `kubectl` change, pause automation with
   `argocd app set web --sync-policy none`; re-enable automated
   sync once the change has landed in Git. For a field another
   actor legitimately owns (for example, `spec.replicas` under
   an HPA), declare `spec.ignoreDifferences` on the Application
   CR so self-heal leaves that field alone. The team's runbook
   for emergencies includes both paths.
3. **External controllers are not used.** If a resource needs
   to be managed by a controller (for example, a HorizontalPodAutoscaler),
   the controller is declared in Git and Argo CD manages its
   lifecycle.

EOF

git add reconcile-observation.md
git commit -m 'argocd: reconcile observation document'

The observation document is what the team reads when they ask “what does selfHeal actually do?”. The three tests are the answer: Argo CD reverts any drift, regardless of source.

Task 5 — Document the sync options

# check-shell-blocks: allow-invalid
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-reconcile-lab"

cat > sync-options-table.md <<'EOF'
# Sync options reference

This document is the canonical reference for the `syncOptions`
that Argo CD applies during automated sync. The
`application-automated.yaml` is the implementation; this
document is the rationale.

## `CreateNamespace=true`

If the target namespace does not exist, Argo CD creates it
before applying the manifests. Required for per-environment
namespaces that are created on demand.

**Default:** `false`.

**Use when:** the team creates per-environment namespaces
(`production`, `staging`) on first deploy.

**Trade-off:** Argo CD's service account needs permission to
create namespaces. The team's RBAC gives Argo CD that
permission for the namespaces it manages, not for all
namespaces in the cluster.

## `PruneLast=true`

Prune (delete) resources that are no longer in Git *after*
the apply. If the apply fails, prune is skipped.

**Default:** `false` (prune happens before apply).

**Use when:** the team wants to avoid the case where a bad
sync deletes resources before the new ones are ready.

**Trade-off:** briefly, both the old and new resources exist
in the cluster. For most workloads this is fine; for resources
with side effects (PersistentVolumeClaims, Services with
external IPs), the overlap can cause issues.

## `ApplyOutOfSyncOnly=true`

Only resources that are out of sync are re-applied. Resources
that are already in sync are left alone.

**Default:** `false` (every resource is applied on every sync).

**Use when:** the application has many resources and the sync
time is dominated by re-applying in-sync resources.

**Trade-off:** if the apply of an in-sync resource has a
side-effect (for example, a `kubectl apply` of a Job), that
side-effect does not happen on subsequent syncs. The team's
discipline: jobs are not idempotent and should not be managed
by Argo CD.

## `ServerSideApply=true`

Use the Kubernetes API server's server-side apply, which is
the safer default for concurrent controllers.

**Default:** `false` (client-side apply via `kubectl apply`).

**Use when:** the team has other controllers managing the
same resources (for example, a HorizontalPodAutoscaler).

**Trade-off:** server-side apply requires the API server to
track field ownership. The team's discipline: declare field
ownership in the manifest's `metadata.managedFields` and let
Argo CD be the owner of the resources it manages.

## `PrunePropagationPolicy=foreground`

The prune (delete) uses foreground propagation: the
resource is marked for deletion, and the API server waits
for finalizers to complete before removing the resource.

**Default:** `background`.

**Use when:** the resources have finalizers (for example,
PersistentVolumeClaims with their underlying PVs).

**Trade-off:** foreground propagation is slower than
background. For most workloads, the difference is seconds.

## `Replace=true`

If a resource cannot be updated (for example, an immutable
field), Argo CD deletes and recreates it.

**Default:** `false` (Argo CD reports the failure).

**Use when:** the manifests include immutable fields that
change between versions (for example, `spec.clusterIP` on a
Service).

**Trade-off:** `Replace=true` is destructive. The resource is
deleted and recreated; for a Deployment, the Pods are
restarted. Use only when the alternative is a stuck sync.

## Summary table

| Option | Default | Effect | Trade-off |
|--------|---------|--------|-----------|
| `CreateNamespace=true` | false | create target namespace | RBAC permission |
| `PruneLast=true` | false | prune after apply | brief overlap |
| `ApplyOutOfSyncOnly=true` | false | only sync drifted | side-effects skipped |
| `ServerSideApply=true` | false | SSA via API server | ownership tracking |
| `PrunePropagationPolicy=foreground` | background | wait for finalizers | slower |
| `Replace=true` | false | delete+recreate if needed | destructive |

EOF

git add sync-options-table.md
git commit -m 'argocd: sync options reference'

The sync-options reference is what the engineer reads when they add a new option to application-automated.yaml. The table at the bottom is the summary; each section is the detail.

Task 6 — Document the failure modes

# check-shell-blocks: allow-invalid
# check-shell-blocks: allow-invalid
cd "$HOME/argocd-reconcile-lab"

cat > automated-sync-failure-modes.md <<'EOF'
# Failure modes: automated sync

This document is the canonical record of the common failures
the team has seen with `syncPolicy.automated` and `selfHeal:
true`. Each section includes the symptom, the cause, and the
fix.

## 1. Prune deletes a resource the team needs

**Symptom:** A Secret, ConfigMap, or PersistentVolumeClaim is
deleted after a sync.

**Cause:** The resource was created outside Git (for example,
via `kubectl apply` for an emergency). The prune sees the
resource as drift and deletes it.

**Fix:** Add the resource to Git. If the resource cannot be
in Git (for example, a dynamically-generated Secret), annotate
the resource itself with
`argocd.argoproj.io/sync-options: Prune=false` so prune skips
it, or set `prune: false` on the CR. For an emergency change
window, pause automation first with
`argocd app set web --sync-policy none` and re-enable it once
Git is reconciled. The team's discipline: every resource
that survives a sync must be in Git.

## 2. Replace fails on an immutable field

**Symptom:** The sync fails with
`Error: field is immutable`.

**Cause:** The manifest changes an immutable field (for
example, `spec.clusterIP` on a Service). The API server
rejects the update.

**Fix:** Either set `Replace=true` in `syncOptions` (destructive:
delete+recreate), or accept that the resource is immutable and
edit it via a separate manifest. The team uses `Replace=true`
only for resources that are intentionally recreated.

## 3. Self-heal fights with an external controller

**Symptom:** The resource is `OutOfSync` repeatedly, and the
cluster state alternates between two values.

**Cause:** An external controller (for example, a
HorizontalPodAutoscaler, a custom controller) and Argo CD both
manage the resource. Each reconciliation cycle overwrites the
other's changes.

**Fix:** Use `ServerSideApply=true` and declare Argo CD as the
owner of the conflicting fields. If the external controller
must own the field, exclude it from the manifest in Git
(for example, `replicas` is not in the Deployment; the HPA
manages it) and declare `spec.ignoreDifferences` on the
Application CR so self-heal leaves that field alone.

## 4. Sync interval is too long

**Symptom:** A change to Git takes minutes to apply.

**Cause:** The default reconciliation interval is 3 minutes;
plus the sync itself takes time. A team that needs faster
feedback sets the interval shorter.

**Fix:** Set `metadata.annotations.[argocd.argoproj.io/refresh]`
to `normal` (the default) or `hard` (forces a full re-sync).
For continuous reconciliation, set the cluster's
`timeout.reconciliation` to a shorter interval (for example,
30 seconds). The trade-off: shorter intervals increase the
controller's CPU usage.

## 5. Controller outage causes prolonged drift

**Symptom:** The application is `OutOfSync` for an extended
period (hours) after a Git change.

**Cause:** The `argocd-application-controller` pod is not
running (OOMKilled, evicted, or crashed). Without the
controller, no reconciliation happens.

**Fix:** Check the controller's status:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-application-controller kubectl -n argocd logs -l app.kubernetes.io/name=argocd-application-controller

The Events section and the logs point to the cause. Common
causes: OOMKilled (the controller needs more memory), evicted
(node pressure), image pull errors.

## 6. Git repository is unreachable

**Symptom:** The application is `OutOfSync` and
`argocd app get` reports
`Failed to load source repo: connection refused`.

**Cause:** The Git server is down, or Argo CD's credentials
have been revoked.

**Fix:** Verify the Git server is reachable from the cluster.
The team's discipline: monitor the Git server's uptime via
synthetic checks; alert on unreachable.

## 7. Sync window blocks a critical deploy

**Symptom:** A production deploy is queued but not applied.

**Cause:** A deny sync window on the application's AppProject
blocks the sync at the current time. The team has configured
a maintenance window that excludes the deploy time.

**Fix:** Sync windows live on the AppProject, not the
Application, so they are managed with `argocd proj windows`.
Either wait for the window to open, update or delete the
window on the project, or enable manual sync on the window
and sync by hand:

argocd proj windows list <PROJECT> argocd proj windows enable-manual-sync <PROJECT> <WINDOW_ID> argocd app sync <APP_NAME>

The team's discipline: emergency deploys bypass the window
via `argocd proj windows enable-manual-sync` (or `argocd proj
windows update`/`delete`) on the AppProject and are recorded
in the incident postmortem.

EOF

git add automated-sync-failure-modes.md
git commit -m 'argocd: failure modes for automated sync'

The failure-mode catalogue is the on-call reference. Each section is the answer to a specific failure; the document is organised by failure mode, with symptom-to-cause-to-fix at the top of each section.

Task 7 — Validate the YAML structure

cd "$HOME/argocd-reconcile-lab"

# Application CR parses and has the expected fields.
python3 -c "
import yaml
with open('application-automated.yaml') as f:
    doc = yaml.safe_load(f)
app = doc
sp = app['spec']['syncPolicy']
print('automated.prune:', sp['automated']['prune'])
print('automated.selfHeal:', sp['automated']['selfHeal'])
print('automated.allowEmpty:', sp['automated']['allowEmpty'])
print('syncOptions:', sp['syncOptions'])
"

# drift-test script syntax.
bash -n drift-test.sh && echo "drift-test.sh: syntax ok"

# Verify the manifests parse.
python3 -c "
import yaml
with open('app-source/web-deployment.yaml') as f:
    doc = yaml.safe_load(f)
print('kind:', doc['kind'])
print('replicas:', doc['spec']['replicas'])
"

Expected output (excerpt):

automated.prune: True
automated.selfHeal: True
automated.allowEmpty: False
syncOptions: ['CreateNamespace=true', 'PruneLast=true',
              'ApplyOutOfSyncOnly=true', 'ServerSideApply=true']
drift-test.sh: syntax ok
kind: Deployment
replicas: 2

The Application CR has prune: true, selfHeal: true, and four sync options. The drift-test script passes syntax check. The Deployment manifest has replicas: 2.

Task 8 — Capture the deliverables

cd "$HOME/argocd-reconcile-lab"

cp application-automated.yaml     "$HOME/application-automated.yaml"
cp drift-test.sh                  "$HOME/drift-test.sh"
cp reconcile-observation.md       "$HOME/reconcile-observation.md"
cp sync-options-table.md          "$HOME/sync-options-table.md"
cp automated-sync-failure-modes.md "$HOME/automated-sync-failure-modes.md"
mkdir -p "$HOME/app-source"
cp app-source/web-deployment.yaml "$HOME/app-source/web-deployment.yaml"
cp app-source/web-service.yaml    "$HOME/app-source/web-service.yaml"

ls -l "$HOME"/application-automated.yaml \
       "$HOME"/drift-test.sh \
       "$HOME"/reconcile-observation.md \
       "$HOME"/sync-options-table.md \
       "$HOME"/automated-sync-failure-modes.md \
       "$HOME"/app-source/

The deliverables are the five files plus the app-source/ directory in $HOME.

Validation

• application-automated.yaml parses as valid YAML and has syncPolicy.automated.prune: true, syncPolicy.automated.selfHeal: true, and syncPolicy.automated.allowEmpty: false.
• drift-test.sh is executable, has set -euo pipefail, and performs three drift tests.
• reconcile-observation.md documents all three drift tests and the reconciliation behaviour.
• sync-options-table.md documents all six sync options.
• automated-sync-failure-modes.md documents all seven failure modes.

Expected Outcome

An Application CR configured for automated reconciliation with self-heal, a drift-test script that proves the reconciliation works, and the documentation that makes the behaviour reviewable.

$HOME/argocd-reconcile-lab/
├── application-automated.yaml            # the Application CR
├── drift-test.sh                         # the drift test
├── reconcile-observation.md              # the observation
├── sync-options-table.md                 # the option reference
├── automated-sync-failure-modes.md       # the failure catalogue
└── app-source/                           # the Git source
    ├── web-deployment.yaml
    └── web-service.yaml

The Application CR is the configuration; the drift-test script is the verification; the documents are the rationale.

Troubleshooting

The drift test does not revert. selfHeal is not enabled in the Application CR. Verify with argocd app get web -o yaml | grep -A 5 automated.

The drift test reverts but the rollout is slow. The Deployment uses RollingUpdate with a default surge of 25%; with 2 replicas, that is one new Pod at a time. For faster rollouts, increase the surge or use Recreate.

The sync fails with field is immutable. The manifest changes an immutable field. Either set Replace=true in syncOptions or accept the immutability and edit the resource via a separate manifest.

argocd-application-controller is OOMKilled. The controller needs more memory. The default is 1 GB; large clusters need 2-4 GB. Set the memory request and limit in the controller’s Deployment.

argocd app get web shows Unknown for sync status. The application controller cannot reach the Git repository. Check the credentials and the network.

A resource the team needs is deleted by prune. Add the resource to Git, or annotate it with argocd.argoproj.io/sync-options: Prune=false so prune skips it. The team’s discipline: every resource that survives a sync must be in Git.

Cleanup

LAB="$HOME/argocd-reconcile-lab"

mv "$LAB"/reconcile-observation.md "$LAB"/sync-options-table.md \
   "$LAB"/automated-sync-failure-modes.md \
   "$HOME"/ 2>/dev/null
mv "$LAB/application-automated.yaml" \
   "$HOME/application-automated.yaml" 2>/dev/null
mv "$LAB/drift-test.sh" "$HOME/drift-test.sh" 2>/dev/null
mv "$LAB/app-source" "$HOME/app-source" 2>/dev/null

rm -rf "$LAB"

find "$HOME" -maxdepth 1 -name 'argocd-reconcile-lab' -print
# expected: (no output)

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

• automated and selfHeal are different controls. automated triggers a sync when a new Git revision arrives; selfHeal additionally re-applies Git’s desired state when the live cluster drifts. Both converge the cluster toward Git — neither ever writes to Git. Without selfHeal, a manual cluster edit persists until the next Git change triggers a sync; with selfHeal, the next reconciliation reverts it.
• Prune is destructive and irreversible. A bug in Git that deletes a manifest will delete the in-cluster resource on the next sync. The team’s discipline: every production resource is in Git.
• allowEmpty: false is the safety net. An empty Git source does not delete all resources; Argo CD keeps the existing state. The flag is a safety net, not a permanent fix.
• ServerSideApply=true is the safer default. Concurrent controllers (HPA, custom controllers) do not fight Argo CD when SSA is enabled. Field ownership is tracked by the API server.
• Sync options have trade-offs. PruneLast=true reduces the blast radius of a bad sync but briefly overlaps old and new. ApplyOutOfSyncOnly=true skips side-effects on in-sync resources. The team reviews each option in code review.
• Self-heal has developer-experience implications. A developer who debugs via kubectl edit will see their edits reverted. The team documents this in the runbook.
• Controller outage is a single point of failure. Without the controller, no reconciliation happens. The team monitors the controller’s status and alerts on prolonged OutOfSync.