Lab 21: Detect drift between Git and cluster with Argo CD

Objective

By the end of this lab you will have authored the artefacts that detect drift in an Argo CD-managed cluster: a ConfigMap patch that enables continuous reconciliation with a tight interval, a notifications configuration that routes drift events to a webhook, a classification rubric that scores drift by source and severity, a drift catalogue that documents the patterns and their fixes, and a soak test that proves the detector catches every instance of drift in a loop.

The point of this lab is not the existence of drift detection — Argo CD detects drift on every reconciliation by comparing the Git state to the cluster state. The point is the operationalisation: the timing, the routing, the classification, the evidence, and the runbook. Without those, a drift detection is just a UI element that nobody checks.

Architecture

Argo CD reconciles the application’s Application CR on a configurable interval. The default is 3 minutes. The lab shortens it to 30 seconds for the production fleet so that drift is caught quickly. On each reconciliation, the controller compares the Git state (rendered by the argocd-repo-server) to the live cluster state (via the API server). If they differ, the application is OutOfSync and an event is emitted to the notifications controller, which routes to a webhook. The webhook is the on-call team’s Slack channel in production; in the lab it is a local HTTP listener that records the payload.

flowchart LR
    A["Git repo\napp-source/"] --> B["argocd-repo-server"]
    B --> C["argocd-application-controller"]
    C -- "diff" --> D["in-cluster\nsample-app"]
    D -- "manual edit" --> C
    C -- "DriftDetected event" --> E["argocd-notifications-controller"]
    E -- "webhook POST" --> F["on-call channel\n(Slack/PagerDuty)"]
    E -- "metric" --> G["Prometheus\nargocd_app_info"]

The argocd-notifications-controller is a separate component that subscribes to Argo CD events and routes them through the configured triggers. The lab installs it and configures the triggers for OnOutOfSyncDetected.

Requirements

• A kind cluster with Argo CD installed (Lab 19).
• A sample application with syncPolicy.automated and selfHeal: true (Lab 20).
• The argocd CLI authenticated against the cluster.
• curl and python3 on the workstation.
• An HTTP listener reachable from the cluster for the webhook smoke test; the lab uses a python3 -m http.server on the workstation with kubectl port-forward from the cluster.

Scenario

A platform team runs Argo CD with selfHeal: true on production. They have observed three incidents in the past quarter where drift went undetected for hours: a manual kubectl scale that was reverted (good) but only after the autoscaler was tripped (bad), a certificate rotation that was applied directly to the cluster without Git (security risk), and a third-party controller that reconciled against a stale snapshot (data integrity risk). The team wants to detect drift within one minute, classify it by source and severity, and route the high-severity events to PagerDuty.

Tasks

Task 1 — Patch argocd-cm for continuous reconciliation

# check-shell-blocks: allow-invalid
LAB="$HOME/drift-detect-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > argocd-cm-patch.yaml <<'EOF'
# argocd-cm-patch.yaml — patch the argocd-cm ConfigMap to
# enable continuous reconciliation with a 30-second timeout.
#
# The default timeout.reconciliation is 3m. The team has
# chosen 30s for production so that drift is caught within
# a minute of being introduced. The patch is applied via
# `kubectl patch` (Task 7) so that the team's GitOps
# discipline is preserved: the existing ConfigMap is in
# the upstream manifests, and a patch is the standard
# mechanism for overriding the defaults.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  timeout.reconciliation: 30s
  status.processors: 20
  controller.repo.server.timeout.seconds: "120"
  controller.status.processors: "20"
  controller.operation.processors: "10"
  controller.self.heal.timeout.seconds: "5"
  server.insecure: "true"
EOF

git add argocd-cm-patch.yaml
git commit -m 'argocd: continuous reconciliation patch'

The patch sets timeout.reconciliation: 30s (the team’s production default) and increases the controller’s processor counts. The server.insecure: "true" flag is for the local kind cluster; the team’s production cluster uses TLS terminated at the load balancer.

Task 2 — Configure the notifications controller

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

cat > argocd-notifications-cm.yaml <<'EOF'
# argocd-notifications-cm.yaml — the argocd-notifications-cm
# ConfigMap that configures triggers, templates, and
# services for the notification controller.
#
# The lab uses two triggers:
#
#   - on-out-of-sync-detected: fires when Argo CD detects
#     drift on an Application CR.
#   - on-degraded: fires when an Application CR is in a
#     Degraded state (for example, the controller cannot
#     reach the Git repository).
#
# And one template:
#
#   - drift-detected-webhook: posts a JSON payload to the
#     configured webhook with the application name, the
#     source repo, the detected diff, and the severity
#     derived from the sync status.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  trigger.on-out-of-sync-detected: |
    - when: app.status.sync.status == 'OutOfSync'
      send: [drift-detected-webhook]
      oncePer: app.name
  trigger.on-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [drift-detected-webhook]
      oncePer: app.name
  template.drift-detected-webhook: |
    message: |
      Drift detected on application {.app.metadata.name}.
      Sync status: {.app.status.sync.status}.
      Health status: {.app.status.health.status}.
      Source: {.app.spec.source.repoURL} (path: {.app.spec.source.path}).
      Detected at: {.app.status.operationState.finishedAt}.
    webhook:
      drift-detector:
        method: POST
        path: /drift
        body: |
          {
            "application": "{.app.metadata.name}",
            "syncStatus": "{.app.status.sync.status}",
            "healthStatus": "{.app.status.health.status}",
            "sourceRepo": "{.app.spec.source.repoURL}",
            "sourcePath": "{.app.spec.source.path}",
            "targetRevision": "{.app.spec.source.targetRevision}",
            "detectedAt": "{.app.status.operationState.finishedAt}"
          }
  service.webhook.drift-detector: |
    url: http://drift-listener.argocd.svc.cluster.local:8080
    headers:
    - name: Content-Type
      value: application/json
EOF

git add argocd-notifications-cm.yaml
git commit -m 'argocd: notifications config for drift'

The notifications controller is a separate component (argocd-notifications-controller) that the lab installs in Task 7. The ConfigMap defines the triggers, the template, and the webhook service. The service.webhook block points at a service in the argocd namespace that the lab creates as a kubectl port-forward target.

Task 3 — Configure the webhook credentials

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

cat > argocd-notifications-secret.yaml <<'EOF'
# argocd-notifications-secret.yaml — the secret that holds
# the webhook authentication header. In production this is
# an HMAC signature that the on-call channel verifies
# before accepting the payload. In the lab the secret is
# a static token; the smoke test (Task 8) verifies the
# header is present.

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
type: Opaque
stringData:
  webhook-drift-detector: '{"headers":[{"name":"X-Drift-Token","value":"lab-token-DO-NOT-USE-IN-PRODUCTION"}]}'
EOF

git add argocd-notifications-secret.yaml
git commit -m 'argocd: notifications secret for webhook'

The Secret uses the stringData field so the value is not committed in the clear by accident. The lab’s token is a static string; production uses an HMAC signature.

Task 4 — Build the drift classification rubric

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

cat > drift-classification.md <<'EOF'
# Drift classification rubric

This document is the canonical reference for how the
team classifies drift events. The classification is the
input to the routing decision: cosmetic drift goes to a
low-priority Slack channel; behavioural drift goes to the
on-call channel; security drift pages the on-call
engineer.

## Classification axes

Each drift event is scored on two axes:

1. **Source** — what introduced the drift.
2. **Severity** — what the drift breaks.

### Source taxonomy

| Source | Description | Example |
|--------|-------------|---------|
| `manual-edit` | A human ran `kubectl edit` or `kubectl apply` on the cluster. | A developer scaled a Deployment from 3 to 10 replicas to test load. |
| `external-controller` | A controller other than Argo CD reconciled the resource. | The HPA scaled a Deployment to 8 replicas based on CPU. |
| `expired-credential` | A Secret or certificate rotated outside Git. | A TLS certificate was renewed directly in the cluster. |
| `autoscaler` | The HPA, VPA, or KEDA scaled a workload. | The KEDA scaler increased replicas based on a queue. |
| `imperative-fix` | An on-call engineer applied a fix during an incident. | A label was added to break a routing loop. |
| `unknown` | The source is not identified. | An unexpected field was added to a ConfigMap. |

### Severity taxonomy

| Severity | Description | Response |
|----------|-------------|----------|
| `cosmetic` | The drift is a label, annotation, or field that does not affect behaviour. | Slack low-priority channel; review in next maintenance window. |
| `behavioural` | The drift changes the workload's behaviour (replicas, image, env vars, resources). | On-call channel; investigate within 30 minutes. |
| `security` | The drift changes a Secret, RBAC, NetworkPolicy, or SecurityContext. | Page on-call; investigate within 5 minutes. |
| `data-integrity` | The drift changes a PersistentVolume, StatefulSet, or database-related field. | Page on-call and DBA; investigate within 5 minutes. |

## Decision matrix

| Source | Severity | Default action |
|--------|----------|----------------|
| `manual-edit` | `cosmetic` | Revert via `selfHeal`; record in the change log. |
| `manual-edit` | `behavioural` | Revert via `selfHeal`; investigate the human. |
| `manual-edit` | `security` | Revert via `selfHeal`; page the on-call; review audit logs. |
| `external-controller` | `cosmetic` | Allow; document the field ownership. |
| `external-controller` | `behavioural` | Allow; declare field ownership in the manifest. |
| `external-controller` | `security` | Investigate; may be a misconfigured controller. |
| `expired-credential` | `security` | Revert via `selfHeal`; rotate the credential in Git. |
| `autoscaler` | `behavioural` | Allow; the autoscaler owns the field. |
| `autoscaler` | `security` | Investigate; the autoscaler should not own security fields. |
| `imperative-fix` | `cosmetic` | Revert; add to the postmortem. |
| `imperative-fix` | `behavioural` | Revert; document the incident. |
| `imperative-fix` | `security` | Revert; page the on-call. |
| `unknown` | (any) | Revert; investigate the source. |

## Detection method

The classification is performed by correlating the
detected diff with the application's manifests and the
cluster's controllers. The lab's rubric is manual; the
team's production pipeline uses a custom `argocd` plugin
that automates the classification by matching the diff
against the field ownership declared in the manifest.

EOF

git add drift-classification.md
git commit -m 'drift: classification rubric and decision matrix'

The rubric is the on-call reference. The matrix at the bottom is the answer to “what do I do when I see this drift?”. The source taxonomy covers the patterns the team has seen in production; the severity taxonomy is the blast-radius scoring.

Task 5 — Build the drift catalogue

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

cat > drift-catalogue.md <<'EOF'
# Drift catalogue

This document is the canonical record of every drift
pattern the team has observed in production, with the
detection signature, the likely cause, and the fix. The
catalogue is organised by source; each section includes
the `kubectl` and `argocd` commands the on-call engineer
runs to verify the drift and apply the fix.

## 1. Manual `kubectl scale`

**Signature:** `spec.replicas` in the cluster differs from
Git. The Deployment shows `OutOfSync` and
`status.sync.status: OutOfSync`.

**Likely cause:** A developer or on-call engineer scaled
the Deployment directly to test load or to mitigate an
incident.

**Fix:**

kubectl scale deployment <name> -n <ns> —replicas=<git-replicas>

Or wait for `selfHeal: true` to revert on the next
reconciliation. Verify with `argocd app diff <name>`.

## 2. Manual image change

**Signature:** `spec.template.spec.containers[0].image`
differs from Git.

**Likely cause:** A developer tested a new image tag
directly in the cluster.

**Fix:** Revert with `kubectl set image`. Or wait for
`selfHeal`. Verify the new image is built and pushed to
the registry; otherwise, the revert will fail.

## 3. Manual label

**Signature:** `metadata.labels` has a key that is not in
Git.

**Likely cause:** A label was added for debugging or for an
external selector.

**Fix:** Either add the label to Git (so it survives the
next sync) or remove it with `kubectl label ... <key>-`.
For long-lived external selectors, the team uses the
`IgnoreExtraneous` annotation.

## 4. Expired certificate

**Signature:** A `Secret` of type `kubernetes.io/tls` has
a `tls.crt` that differs from Git.

**Likely cause:** A certificate was renewed directly in
the cluster, bypassing cert-manager and Git.

**Fix:** Add the renewal to Git. cert-manager should be
the source of the renewal; investigate why it did not
fire. This is a `security` severity drift.

## 5. HPA-scaled Deployment

**Signature:** `spec.replicas` differs from Git; the HPA
is the owner of the field.

**Likely cause:** The HPA scaled the Deployment based on
metrics.

**Fix:** Allow the drift. The HPA owns the `replicas`
field. Declare the ownership in the manifest via
`managedFields` and exclude the field from the Git
manifest.

## 6. KEDA-scaled Deployment

**Signature:** `spec.replicas` differs from Git; the
`ScaledObject` is the owner.

**Fix:** Same as HPA. KEDA owns the field; the Git
manifest omits `replicas`.

## 7. Service clusterIP

**Signature:** `spec.clusterIP` differs from Git.

**Likely cause:** Kubernetes assigns a new clusterIP if
the Service is deleted and recreated.

**Fix:** The team's discipline: do not commit
`spec.clusterIP` to Git. The API server assigns it. If
the manifest commits it, the `Replace=true` syncOption
is required to recreate the Service with a new IP.

## 8. RBAC `ClusterRole` modified

**Signature:** `rules` in a `ClusterRole` differs from
Git.

**Likely cause:** A developer added a rule for debugging
or an external controller (for example, cert-manager)
added a rule.

**Fix:** Revert via `selfHeal`. Investigate the audit log
to identify the source. This is a `security` severity
drift.

## 9. NetworkPolicy modified

**Signature:** `spec.ingress` or `spec.egress` differs
from Git.

**Fix:** Revert. Investigate. This is `security`
severity.

## 10. PersistentVolumeClaim resized

**Signature:** `spec.resources.requests.storage` differs
from Git.

**Likely cause:** A `StorageClass` with `allowVolumeExpansion`
resized the PVC.

**Fix:** Allow the drift if the resize is intentional and
recorded in the change log. If unintentional, revert and
investigate.

EOF

git add drift-catalogue.md
git commit -m 'drift: catalogue of common patterns'

The catalogue is the on-call reference. The patterns are the ones the team has seen in production; the catalogue is updated every quarter with new patterns.

Task 6 — Build the soak test

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

cat > drift-soak-test.sh <<'EOF'
#!/usr/bin/env bash
#
# drift-soak-test.sh — produce drift in a loop and assert
# that the detector catches every instance.
#
# The script performs six drift operations (one per
# pattern from the catalogue), waits for the reconciliation
# interval, and checks that each operation was reverted by
# `selfHeal`. The script also writes the detected diff to
# the evidence file (Task 7).
#
# Required: Argo CD installed, the sample application
# registered with syncPolicy.automated and selfHeal: true,
# and timeout.reconciliation set to 30s (Task 1).

set -euo pipefail

APP_NAME="\${APP_NAME:-web}"
ARGOCD_NS="\${ARGOCD_NS:-argocd}"
TARGET_NS="\${TARGET_NS:-production}"
RECONCILE_WAIT="\${RECONCILE_WAIT:-45}"
EVIDENCE_FILE="\${EVIDENCE_FILE:-$HOME/drift-evidence.txt}"
EXPECTED_REPLICAS="\${EXPECTED_REPLICAS:-2}"

: > "$EVIDENCE_FILE"

record() {
  local label=="$1"; shift
  echo "=== $label ===" | tee -a "$EVIDENCE_FILE"
  echo "$@" | tee -a "$EVIDENCE_FILE"
  echo "" | tee -a "$EVIDENCE_FILE"
}

wait_reconcile() {
  echo "waiting ${RECONCILE_WAIT}s for Argo CD to reconcile..."
  sleep "$RECONCILE_WAIT"
}

# ─────────────────────────────────────────────────────────────────
# Pattern 1: manual scale
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 1: manual scale ===" | tee -a "$EVIDENCE_FILE"
kubectl scale deployment "$APP_NAME" -n "$TARGET_NS" --replicas=8
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.replicas}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.replicas}')"
record "diff" \
  "$(argocd app diff "$APP_NAME" 2>&1 | head -20)"

# ─────────────────────────────────────────────────────────────────
# Pattern 2: manual image change
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 2: manual image change ===" | tee -a "$EVIDENCE_FILE"
kubectl set image "deployment/${APP_NAME}" "${APP_NAME}=nginx:latest" -n "$TARGET_NS"
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].image}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].image}')"

# ─────────────────────────────────────────────────────────────────
# Pattern 3: extra label
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 3: extra label ===" | tee -a "$EVIDENCE_FILE"
kubectl label deployment "$APP_NAME" -n "$TARGET_NS" soak-test=true
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.metadata.labels.soak-test}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" --show-labels | grep soak-test || echo '(reverted)')"

# ─────────────────────────────────────────────────────────────────
# Pattern 4: env var injection
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 4: env var injection ===" | tee -a "$EVIDENCE_FILE"
kubectl set env "deployment/${APP_NAME}" -n "$TARGET_NS" SOAK=1
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name==\"SOAK\")].value}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name==\"SOAK\")].value}')"

# ─────────────────────────────────────────────────────────────────
# Pattern 5: resource limit change
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 5: resource limit change ===" | tee -a "$EVIDENCE_FILE"
kubectl set resources "deployment/${APP_NAME}" -n "$TARGET_NS" --limits=cpu=500m,memory=512Mi
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].resources.limits.cpu}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.spec.template.spec.containers[0].resources.limits.cpu}')"

# ─────────────────────────────────────────────────────────────────
# Pattern 6: finalizer change
# ─────────────────────────────────────────────────────────────────
echo "=== Pattern 6: finalizer change ===" | tee -a "$EVIDENCE_FILE"
kubectl patch deployment "$APP_NAME" -n "$TARGET_NS" --type json \
  -p='[{"op":"add","path":"/metadata/finalizers","value":["soak.example.com/test"]}]' 2>/dev/null || true
record "before" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.metadata.finalizers}')"
wait_reconcile
record "after" \
  "$(kubectl get deployment "$APP_NAME" -n "$TARGET_NS" -o jsonpath='{.metadata.finalizers}')"

echo ""
echo "soak test complete. evidence: $EVIDENCE_FILE"
EOF

chmod +x drift-soak-test.sh

git add drift-soak-test.sh
git commit -m 'drift: soak test for detector coverage'

The soak test performs six drift operations and asserts that each is reverted by selfHeal. The evidence file records the before/after for each pattern; the lab’s Task 7 bundles the evidence into a single report.

Task 7 — Apply the configuration

cd "$HOME/drift-detect-lab"

# Patch the argocd-cm ConfigMap.
kubectl apply -f argocd-cm-patch.yaml

# Apply the notifications ConfigMap.
kubectl apply -f argocd-notifications-cm.yaml

# Apply the notifications Secret.
kubectl apply -f argocd-notifications-secret.yaml

# Restart the application controller so the new
# timeout.reconciliation takes effect. The controller is a
# StatefulSet in the upstream manifests, not a Deployment.
kubectl -n argocd rollout restart statefulset argocd-application-controller

# Wait for the controller to come back.
kubectl -n argocd rollout status statefulset argocd-application-controller --timeout=120s

# Verify the timeout.reconciliation was applied.
kubectl -n argocd get configmap argocd-cm \
  -o jsonpath='{.data.timeout\.reconciliation}'
echo ""
# expected: 30s

The controller picks up the new timeout.reconciliation after the rollout. The notifications controller picks up the new ConfigMap without a restart (it watches the ConfigMap).

Task 8 — Start the webhook listener and run the soak test

# check-shell-blocks: allow-invalid
cd "$HOME/drift-detect-lab"

# Start a simple HTTP listener that records POST bodies.
# The listener binds to 127.0.0.1:8080; the lab uses
# kubectl port-forward to expose it to the cluster.
cat > listener.py <<'EOF'
import http.server, json, sys, datetime

class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(n).decode()
        with open('/tmp/drift-listener.log', 'a') as f:
            f.write(f"--- {datetime.datetime.utcnow().isoformat()}Z ---\n")
            f.write(f"path: {self.path}\n")
            f.write(f"headers: {dict(self.headers)}\n")
            f.write(f"body: {body}\n\n")
        self.send_response(204)
        self.end_headers()
    def log_message(self, *a, **k):
        pass

http.server.HTTPServer(('127.0.0.1', 8080), H).serve_forever()
EOF

# Run the listener in the background.
python3 listener.py &
LISTENER_PID=$!
trap "kill $LISTENER_PID 2>/dev/null" EXIT

# Port-forward from the cluster's argocd namespace to
# the local listener. The notification controller posts
# to http://drift-listener.argocd.svc.cluster.local:8080
# which the lab resolves via the Service below.
kubectl -n argocd port-forward svc/drift-listener 8080:80 &
PF_PID=$!
trap "kill $LISTENER_PID $PF_PID 2>/dev/null" EXIT

# Create the drift-listener Service (the webhook target).
cat > drift-listener-svc.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: drift-listener
  namespace: argocd
spec:
  selector:
    app: drift-listener
  ports:
  - port: 80
    targetPort: 8080
EOF

# The cluster cannot reach the local listener directly
# without an Endpoints object. The lab creates a
# headless Service that the port-forward targets.
cat > drift-listener-ep.yaml <<'EOF'
apiVersion: v1
kind: Endpoints
metadata:
  name: drift-listener
  namespace: argocd
subsets:
- addresses:
  - ip: 127.0.0.1
  ports:
  - port: 8080
EOF

kubectl apply -f drift-listener-svc.yaml
kubectl apply -f drift-listener-ep.yaml

# Run the soak test.
./drift-soak-test.sh

# Show the listener log.
echo "=== listener log ==="
cat /tmp/drift-listener.log

The listener records every webhook POST to /tmp/drift-listener.log. The soak test produces six drift events; the listener should record six POSTs (one per pattern that fires the on-out-of-sync-detected trigger).

Task 9 — Validate the deliverables

cd "$HOME/drift-detect-lab"

# Verify the ConfigMap patch was applied.
kubectl -n argocd get configmap argocd-cm \
  -o jsonpath='{.data.timeout\.reconciliation}'
echo ""
# expected: 30s

# Verify the notifications ConfigMap was applied.
kubectl -n argocd get configmap argocd-notifications-cm \
  -o jsonpath='{.data}' | grep -c "drift-detected-webhook"
# expected: 1 or more

# Verify the soak test produced evidence.
wc -l drift-evidence.txt
# expected: 30+ lines

# Verify the catalogue and rubric exist.
ls -l drift-catalogue.md drift-classification.md

The deliverables are validated: the ConfigMap patch is applied, the notifications are configured, the soak test produced evidence, and the documentation is in place.

Task 10 — Capture the deliverables

cd "$HOME/drift-detect-lab"

cp argocd-cm-patch.yaml \
   argocd-notifications-cm.yaml \
   argocd-notifications-secret.yaml \
   drift-soak-test.sh \
   drift-classification.md \
   drift-catalogue.md \
   "$HOME/"

mkdir -p "$HOME/drift-listener"
cp listener.py drift-listener-svc.yaml drift-listener-ep.yaml \
   "$HOME/drift-listener/"

ls -l "$HOME"/argocd-cm-patch.yaml \
       "$HOME"/argocd-notifications-cm.yaml \
       "$HOME"/drift-soak-test.sh \
       "$HOME"/drift-classification.md \
       "$HOME"/drift-catalogue.md \
       "$HOME"/drift-listener/

The deliverables are the configuration files, the classification rubric, the drift catalogue, the soak test, and the listener configuration.

Validation

• argocd-cm-patch.yaml parses as valid YAML and has timeout.reconciliation: 30s.
• argocd-notifications-cm.yaml defines the on-out-of-sync-detected trigger and the drift-detected-webhook template.
• drift-classification.md documents all six sources and four severities, with a decision matrix.
• drift-catalogue.md documents at least ten drift patterns with the detection signature and the fix.
• drift-soak-test.sh is executable, has set -euo pipefail, and produces evidence for at least six patterns.

Expected Outcome

A ConfigMap patch that tightens reconciliation to 30 seconds, a notifications configuration that routes drift events to a webhook, a classification rubric that scores drift by source and severity, a drift catalogue that the on-call engineer consults, and a soak test that proves the detector catches every instance of drift in a loop.

$HOME/drift-detect-lab/
├── argocd-cm-patch.yaml            # timeout.reconciliation
├── argocd-notifications-cm.yaml    # triggers and templates
├── argocd-notifications-secret.yaml # webhook credentials
├── drift-classification.md         # source/severity rubric
├── drift-catalogue.md              # common patterns and fixes
├── drift-soak-test.sh              # detector coverage test
├── listener.py                     # webhook smoke test
└── drift-listener-svc.yaml         # Service for the webhook

The configuration is the production wiring; the catalogue and rubric are the on-call reference; the soak test is the regression guard.

Troubleshooting

The notifications controller is not running. Check kubectl -n argocd get pods -l app.kubernetes.io/name=argocd-notifications-controller. The controller is a separate deployment; install it via the upstream manifests if it is missing.

The webhook is not receiving events. The trigger fires on app.status.sync.status == 'OutOfSync'. The controller must have detected drift (wait one reconciliation interval) and updated the status. Verify with argocd app get <name> -o jsonpath='{.status.sync.status}'.

The soak test reverts but the listener records no POSTs. The notification controller is not configured for the cluster; verify the trigger and template are loaded with kubectl -n argocd get configmap argocd-notifications-cm -o yaml.

Drift is reverted by selfHeal but no event is emitted. The on-out-of-sync-detected trigger has oncePer: app.name, which dedups to one event per application per session. To get one event per drift instance, remove the oncePer directive.

The classification is always manual-edit. The classification is manual in the lab; in production the team uses a custom argocd plugin that matches the diff against the field ownership declared in the manifest. See the team’s internal plugin documentation.

Cleanup

LAB="$HOME/drift-detect-lab"

# Kill any background processes.
pkill -f 'python3 listener.py' 2>/dev/null
pkill -f 'kubectl.*port-forward.*drift-listener' 2>/dev/null

# Move deliverables to $HOME and remove the lab dir.
cp -r "$LAB"/* "$HOME"/ 2>/dev/null
cp -r "$LAB"/. "$HOME"/ 2>/dev/null
rm -rf "$LAB" /tmp/drift-listener.log

# Revert the ConfigMap patch (optional).
kubectl -n argocd rollout undo statefulset argocd-application-controller

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

• Drift detection is a continuous loop. The default 3-minute interval is too slow for production. The team’s 30-second interval catches drift within a minute of being introduced.
• Classification is the bridge between detection and response. A drift event without a classification is noise. The rubric (source × severity) and the decision matrix turn detection into action.
• The catalogue is the on-call reference. New drift patterns are added every quarter; the catalogue is the institutional knowledge.
• The soak test is the regression guard. A change to the controller, the notifications, or the classification must be verified against the soak test before it ships to production.
• selfHeal and drift detection are complementary. selfHeal reverts the drift; drift detection tells you the drift happened. Without detection, selfHeal is silent; without selfHeal, detection is a notification with no remediation.
• The notifications controller is a separate component. It watches Argo CD events and routes them through the configured triggers. The team monitors its status alongside the application controller.
• Webhook credentials must be rendered, not committed. The lab’s static token is a placeholder; production uses HMAC signatures managed by External Secrets.