Drift detection and alerting — what metrics show drift and what the alerts look like

Detection and alerting are separate concerns. Detection is the controller comparing observed and desired state and recording a non-empty diff. Alerting is the operator-facing signal that surfaces the detection. A team that conflates the two — that alerts on every detected drift — pages on every transient mismatch and learns to ignore the pages. A team that separates the two pages only on drift that is sustained, attributable, and actionable.

flowchart LR
    D["Diff engine"] -->|"non-empty"| Det["Drift detected"]
    Det --> M["Metric: argocd_app_sync_status"]
    Det --> Age["Metric: drift age"]
    M --> Prom["Prometheus"]
    Age --> Prom
    Prom --> Rule["Alert rule"]
    Rule -->|"fire"| Page["PagerDuty"]
    Rule -->|"warn"| Slack["Slack channel"]
    Page --> Op["On-call"]
    Slack --> Dash["Dashboard"]

What metrics show drift

A production GitOps controller exposes metrics that, taken together, answer the four questions a responder needs: what, where, when, and how long.

• Sync status. Argo CD exposes argocd_app_info and argocd_app_sync_status. The labels include the application name, the namespace, and the sync status (Synced or OutOfSync). A count of OutOfSync Applications over time is the most basic drift indicator.
• Health status. argocd_app_health_status reports the health of the application as a whole. Drift can coexist with health (a Deployment may be Running and OutOfSync at the same time) and the metric surfaces both independently.
• Drift age. The Application’s status.operationState.finishedAt and the timestamp of the last observed state read together produce a drift-age metric. A 30-second-old drift is the controller’s current work; a 30-day-old drift is a piece of the declared state the cluster no longer matches.
• Resource count by drift status. A breakdown of resources by drift status — Synced, OutOfSync, Missing — is more informative than a single Application-level status. An Application with 50 resources, 49 Synced and 1 OutOfSync, is a different operational signal from an Application with all 50 OutOfSync.

kubectl -n argocd get application -o yaml

The Application object is the source of truth for these metrics; the Prometheus exporter reads the same fields. The production use is to scrape the metrics, aggregate by namespace and application, and surface the breakdown on a dashboard that an on-call engineer can read at 03:00.

The alert chain

The alert chain has four stages, and each stage has a different audience:

• Detect. The controller computes the diff and records the status. This stage is silent; no signal leaves the cluster.
• Surface. The controller exposes the status as a metric. The metric is queryable but not visible by default.
• Alert. A Prometheus rule watches the metric and fires when the threshold is crossed. The alert is a Slack message or a PagerDuty incident.
• Remediate. The on-call engineer responds — by investigating, by committing the change, or by suspending self-heal. The remediation is the only stage that produces change.

The production discipline is that the alert must be sustained, attributable, and actionable. A single OutOfSync tick is not an alert; sustained OutOfSync for 15 minutes is. A generic OutOfSync alert is not attributable; an alert that names the application, the namespace, and the drifted resource is. An alert with no documented remediation is not actionable; an alert that links to a runbook is.

Dashboards that answer the four questions

A production drift dashboard surfaces four panels, one per question:

• What. A table of OutOfSync Applications with the count of drifted resources per Application. The table is sortable by drift age.
• Where. A namespace breakdown. A namespace with one OutOfSync Application is a different signal from a namespace with twenty; the breakdown guides triage.
• When. A time-series of drift count. A spike is a different signal from a slow climb; the time-series distinguishes them.
• How long. A drift-age histogram. A long tail of old drift is a signal that the team has accepted the divergence and is operating against the declared state, not from it.

The dashboard is a diagnostic tool; the alert is a paging tool. The two serve different audiences and have different latencies. A team that uses one tool for both ends up with either a dashboard that pages (and is ignored) or an alert that diagnoses (and never fires).

Production discipline

1. Separate detection from alerting. The controller’s metrics are the detection; the alert rules are the alerting. Tuning one does not tune the other.
2. Alert on sustained, attributable drift. A 15-minute window is the typical starting point. Below 15 minutes the drift is likely transient or already being remediated.
3. Embed the runbook in the alert. A PagerDuty incident with a link to the runbook and the diff is a page the on-call engineer can act on without opening a dashboard.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on monitoring and alerting cover the cluster-side metrics the GitOps controller relies on.
• Prometheus for Production Sysadmins - Parts on alert rules cover the rules that turn controller metrics into pages.
• Terraform for Production Sysadmins - Parts on state-file monitoring cover the analogous metrics in declarative infrastructure.

Quiz