When Synced is not Healthy — the failure modes the platform does not catch

Synced but Degraded is the cell where the dashboard is half-green. The manifests agree with Git. The workload is failing. The GitOps controller is not the source of the problem, and GitOps cannot fix it from the controller side. The fix is at the workload layer, and the operator needs to know which layer to fix.

Six recurring categories of failure produce this cell. Each category has a different root cause, a different diagnostic pattern, and a different operator action. Knowing the category is the difference between a five-minute fix and a five-hour investigation.

The six failure modes

The six modes are not an arbitrary taxonomy; they are the categories that account for the majority of Synced-but-Degraded incidents in production GitOps systems. Each mode corresponds to a layer the controller does not inspect.

1. Image digest pinning. The manifest references an image by tag, the kubelet pulls the image at the tag, but the tag points at a different image than the manifest intended. The Deployment spec is correct; the running image is not. Manifest layer: green. Workload layer: red.

2. Readiness probe coverage. The readiness probe checks a path that does not exercise the dependency the workload needs. The pod is Ready; the workload cannot serve real traffic. Manifest layer: green. Workload layer: red.

3. Network reachability. A NetworkPolicy, a CNI issue, a routing change, or a firewall rule blocks the workload from reaching a dependency it needs. The pods are Ready; the workload cannot reach what it depends on. Manifest layer: green. Workload layer: red.

4. Dependency staleness. The workload depends on a database schema, an API contract, or a configuration value that has changed without the workload being updated. The pods are Ready; the workload’s dependencies are out of date. Manifest layer: green. Workload layer: red.

5. Configuration correctness. A ConfigMap or Secret is syntactically valid and matches Git, but the values it carries are wrong for the workload’s current code. The pods are Ready; the configuration is wrong. Manifest layer: green. Workload layer: red.

6. Capacity saturation. The workload is running and Ready but at the limit of what its resources can serve. CPU is saturated, the connection pool is exhausted, the queue is full. The pods are Ready; the system is at capacity. Manifest layer: green. Workload layer: red.

flowchart LR
    SD["Synced + Degraded"] --> M1["1. Image digest pinning"]
    SD --> M2["2. Readiness probe coverage"]
    SD --> M3["3. Network reachability"]
    SD --> M4["4. Dependency staleness"]
    SD --> M5["5. Configuration correctness"]
    SD --> M6["6. Capacity saturation"]
    M1 --> F1["Pin by digest"]
    M2 --> F2["Probe exercises real path"]
    M3 --> F3["NetworkPolicy review"]
    M4 --> F4["Schema/version coordination"]
    M5 --> F5["Config validation"]
    M6 --> F6["Resource limits + autoscaling"]

Diagnostic pattern for each mode

The diagnostic pattern is the same shape for all six modes: read what the manifest says, read what the cluster shows, find the gap. The gap is in a different layer for each mode.

For image digest pinning, the gap is between the manifest’s image reference and the running image. kubectl get deployment shows the manifest’s image. kubectl get pods -o json shows the running image. If they differ, the tag was mutable.

For readiness probe coverage, the gap is between what the probe exercises and what the workload serves. The probe is in the Deployment spec. The workload’s real traffic path is in the application code. If the probe checks /ping and the workload serves /checkout, the probe covers a path the workload does not use.

For network reachability, the gap is between the pod’s network namespace and the dependency’s network namespace. kubectl exec into the pod and curl the dependency. If the connection times out, the network path is broken. kubectl get networkpolicy shows the policies in effect.

For dependency staleness, the gap is between the workload’s expected schema and the dependency’s current schema. The workload’s logs show errors that mention a missing column, an unknown field, or an unexpected value. The dependency’s logs or API responses show the new schema.

For configuration correctness, the gap is between the ConfigMap’s values and the workload’s expected values. The workload’s logs show configuration errors - missing keys, malformed values, or values that pass parsing but fail semantic validation.

For capacity saturation, the gap is between the workload’s resource limits and the workload’s actual load. The metrics show CPU at 100%, memory growing, queue depth increasing, connection pool exhausted. The manifests are correct; the load exceeds what the manifests describe.

Operator action for each mode

Image digest pinning. Replace tag-based image references with digest-based references. image: myorg/api:v1.2.3 becomes image: myorg/api@sha256:abc123.... Digests are immutable; the kubelet cannot pull a different image than the manifest specifies. The operator action is in the manifests, not the cluster.

Readiness probe coverage. Rewrite the readiness probe to exercise the dependency path. /ping is a process-liveness check; /readyz should be a dependency-reachability check. The probe is in the manifest; the fix is in the manifest. The Deployment must roll out for the new probe to take effect.

Network reachability. Investigate the NetworkPolicy, the CNI, the routing, and the firewall. The fix may be in a NetworkPolicy manifest (GitOps-managed) or in the cluster infrastructure (not GitOps-managed). Coordinate with the team that owns the network layer.

Dependency staleness. Coordinate the schema or API change with the workload’s release. The fix is in the workload’s code, the dependency’s schema, or both. A coordinated release closes the gap.

Configuration correctness. Add validation to the workload’s startup sequence so a wrong ConfigMap value fails the pod’s readiness, not its traffic. The fix is in the workload’s code or the ConfigMap’s values. Validation at startup converts the silent failure into a visible failure.

Capacity saturation. Increase the workload’s resource requests and limits, add horizontal pod autoscaling, or shed load at the gateway. The fix is in the manifests (limits, HPA) or in the load-balancing layer (shedding).

argocd app get "$APP" --show-operation

The dashboard confirms the Synced + Degraded cell. The per-resource detail shows which resource is Degraded. The CLI is the starting point, not the answer.

kubectl get application "$APP" -n argocd -o yaml

The CR confirms the dashboard. The per-resource status fields show what the controller sees. The diagnostic pattern then takes the operator outside the cluster - to image hashes, probe definitions, network policies, dependency schemas, configuration values, and capacity metrics.

Production discipline

The production framing of Synced-but-Degraded has three rules:

1. The fix is at the workload layer, not the controller layer. Forcing a sync, disabling self-heal, or resetting the Application does not fix a Synced + Degraded incident. The controller is correct; the workload is not.
2. Diagnostic pattern is read-spec, read-cluster, find-gap. Apply it to each of the six modes. The gap is in a different layer for each mode, but the pattern is the same.
3. Validation is the prevention. Each mode has a prevention: image digest pinning, probe coverage, network policy review, dependency version coordination, configuration validation, capacity planning. Building the prevention into the manifests and the workload’s startup sequence converts silent failures into visible failures.

Cross-course references

• Kubernetes for Production Sysadmins - Parts XV-XVII (Probes) cover probe design, which is the prevention for mode 2.
• Container Security for Production Sysadmins - Parts XII-XIV (Supply Chain) cover image digest pinning, which is the prevention for mode 1.
• Observability for Production Sysadmins - Parts V-VII (Metrics, Logs, Traces) cover the diagnostic signals for modes 4, 5, and 6.
• Linux for Production Sysadmins - Part XXII (ProcessSupervision) covers capacity saturation at the process level, which is the prevention for mode 6.

Quiz