The Synced and Healthy matrix — four combinations

The Synced and Healthy matrix has four cells. Every GitOps application lives in exactly one of them at any reconciliation tick, and the cell it lives in tells the operator where the problem is. The matrix forces the operator to ask two questions instead of one: do the manifests in Git match the manifests on the cluster, and is the workload the cluster is running actually functioning.

Most production incidents in GitOps-managed systems can be classified into exactly one cell. The cell tells the on-call engineer whether to look at Git, the cluster, or the workload first.

The matrix

quadrantChart
    title "Synced vs Healthy matrix"
    x-axis "OutOfSync --> Synced"
    y-axis "Degraded --> Healthy"
    quadrant-1 "Quiet"
    quadrant-2 "Workload broken, manifests fine"
    quadrant-3 "Manifests diverged, workload broken"
    quadrant-4 "Manifests diverged, workload fine"

Each quadrant has a name, an operational meaning, and a remediation path. The names below are operational shorthand; the controllers use only Synced, OutOfSync, Healthy, and Degraded.

• Synced + Healthy: Quiet. The manifests agree, the workload is functioning. Nothing to do. This is the steady state for ninety-nine percent of applications ninety-nine percent of the time.
• OutOfSync + Healthy: Manifests diverged, workload fine. Git and the cluster differ, but the live workload is still doing its job. The next sync (or a self-heal) will reconcile Git to the cluster. Investigate the divergence: was it a deliberate change that should be committed, an accident, or drift?
• Synced + Degraded: Workload broken, manifests fine. The cluster matches Git. The workload is failing. The GitOps controller is not the source of the problem. Look at the workload’s dependencies, its configuration, and its downstream calls.
• OutOfSync + Degraded: Manifests diverged, workload broken. Git and the cluster differ and the workload is failing. The two problems may share a root cause or they may be independent. Investigate both.

Why the matrix is the right model

A single dashboard indicator - “the app is green” or “the app is red” - collapses two independent signals into one. The collapse hides information. A controller that reports only “Healthy” or “Degraded” tells the operator nothing about whether Git and the cluster agree. A controller that reports only “Synced” or “OutOfSync” tells the operator nothing about whether the workload is functioning.

The matrix restores the separation. Argo CD’s CLI and UI show both. Flux’s Kustomization status shows both as separate conditions: Ready and the last-applied revision’s match against the source. An operator who reads only the aggregated status is reading one number where there are two.

stateDiagram-v2
    [*] --> SyncedHealthy
    SyncedHealthy --> OutOfSyncHealthy: "manifest drift"
    SyncedHealthy --> SyncedDegraded: "workload failure"
    SyncedHealthy --> OutOfSyncDegraded: "both"
    OutOfSyncHealthy --> SyncedHealthy: "sync applies"
    OutOfSyncHealthy --> OutOfSyncDegraded: "workload fails during drift"
    OutOfSyncHealthy --> OutOfSyncDegraded: "secondary failure"
    SyncedDegraded --> OutOfSyncDegraded: "manifest drift while failing"
    SyncedDegraded --> SyncedHealthy: "workload recovers"
    OutOfSyncDegraded --> SyncedHealthy: "sync applies and workload recovers"
    OutOfSyncDegraded --> OutOfSyncHealthy: "workload recovers while drifted"

The most dangerous cell

Synced + Degraded is the cell where incidents go to hide. The dashboard is partially green. The cluster matches Git. The workload is broken. The on-call engineer, looking at the dashboard, sees a green Sync indicator and concludes that Git is not the source of the problem - correctly. But the green indicator also reduces the urgency of the page. The incident lives in the workload layer, but the dashboard framing suggests the system is mostly fine.

The fix is not on the controller side. The controller is correctly reporting the manifest layer is fine. The fix is at the workload layer - probes, dependency checks, domain-specific health resources - and at the operator workflow: a page from a Synced + Degraded application should be treated as a workload-layer incident from the start, with the dashboard’s Sync indicator used only to confirm Git is not the suspect.

Reading the matrix from the operator CLI

The matrix is visible in argocd app get and in the Application CR. Each cell corresponds to a unique combination of status.sync.status and status.health.status.

argocd app get "$APP" --show-operation

The output shows Sync Status and Health Status on separate lines. The cell the application is in is the unique combination of those two values. There is no cell called “Unknown” or “Pending” - those are computed from intermediate values during the first reconciliation.

kubectl get application "$APP" -n argocd -o yaml

The same combination is visible in the CR’s status.sync.status and status.health.status fields. When an application has multiple resources, the aggregated Sync and Health come from the worst component with the same exception rule as before: Progressing resources do not drag down a Healthy aggregated Health.

Production discipline

The production framing of the matrix has three rules:

1. Read both axes on every page. A green Sync indicator does not mean a green system. A green Health indicator does not mean a green system. Both green is the only configuration that means “the system is doing what we expect”.
2. Triage from the cell. The cell tells the operator where to look first. Synced + Healthy means look at metrics and logs. Synced + Degraded means look at the workload. OutOfSync + Healthy means look at Git and the cluster. OutOfSync + Degraded means look at both.
3. Alert on cell, not on colour. An alerting rule that fires only on “Degraded” misses the Synced + Degraded cell, because the application is degraded but the dashboard framing suggests the system is mostly fine. Alert on the cell combination.

Cross-course references

• Linux for Production Sysadmins - Part XXII (ProcessSupervision) covers the gap between “the unit is loaded” and “the unit is active”, which is the same two-axis framing at the process level.
• Kubernetes for Production Sysadmins - Parts XV-XVII (Probes) cover readiness versus liveness, which is a different axis-pair but the same multi-axis-operational-readiness framing.
• Observability for Production Sysadmins - Part VI (SLOs) covers the distinction between “the system is up” and “the system is serving correctly”, which is the workload-layer analogue of Healthy.

Quiz