Bad manifests — the controller refuses to apply

The fourth failure mode is the one the rest of the discipline is built to prevent: a manifest reaches the controller that the Kubernetes API server refuses to accept. The controller will diff, refuse to apply, mark the application Degraded, and leave the cluster in its previous state. No data is lost, no resource is corrupted. The application is stuck in a state that says “I know what I want to do, the cluster will not let me”.

The symptom

The application status carries the failure:

argocd app get guestbook

A bad manifest produces OutOfSync (the controller sees the diff) and Degraded (the apply failed). The Conditions block carries a SyncError condition with a message that names the resource and the API server’s error response. The controller log has the detail:

kubectl -n argocd logs statefulset/argocd-application-controller

The entry names the resource kind, namespace, name, and the API server error:

error validating data: ValidationError(Deployment.spec.template.spec.containers[0]):
unknown field "imagePullPolicy"

or, for a missing CRD:

no matches for kind "VaultSecret"

flowchart LR
    A[Commit to Git] --> B[Controller diff]
    B --> C{Manifest valid?}
    C -- yes --> D[Apply]
    D --> E{Cluster accepts?}
    E -- yes --> F[Synced]
    E -- no --> G["Per-resource error in status"]
    C -- no --> H["SyncError: schema invalid"]
    G --> I[Degraded, cluster unchanged]
    H --> I

Why the cluster is in its previous state

This is the safety property of the GitOps apply path. The controller sends each resource to the API server, captures the response, and if any resource fails the sync is reported as failed and the application is marked Degraded. Resources that succeeded before the failure are left in place - this is the Kubernetes apply model, not a two-phase commit.

For Flux, the equivalent safety property is the Kustomization controller’s prune=false default and the Helm controller’s atomic install. A failed release is rolled back to the previous revision rather than left in a half-applied state.

The implication: a bad manifest leaves the cluster in the last good state. The remediation is to fix the manifest, not to roll back the cluster.

The remediation sequence

1. Read the error message. The controller log names the resource and the field. Do not guess.
2. Revert the offending commit. git revert <sha> produces a new commit that undoes the change.
3. Fix the manifest in a branch. A follow-up PR with the corrected syntax is the path forward.
4. Patch the CI gap. Add the validator that would have caught it: a missing kubeconform schema, a missing CRD, a missing policy rule.

git revert is preferred over git reset --hard because revert preserves the audit trail. The bad commit remains in history with a follow-up commit that undoes it. reset --hard rewrites history and breaks the chain of custody.

Common shapes

• Schema error. A field that does not exist on the resource type, often from a typo or a copy from a different API version.
• Missing CRD. A custom resource the cluster does not know about because its CRD has not been installed.
• Immutable field change. An attempt to change a field the API server marks immutable, typically spec.selector.
• Admission webhook rejection. A manifest that passes schema validation but is rejected by OPA Gatekeeper, Kyverno, or ValidatingAdmissionPolicy.

Production discipline

• Revert, do not reset. The audit trail depends on history being append-only. A bad commit is a commit to revert, not a commit to erase.
• Treat the bad commit as a CI bug. Every bad manifest that reaches the controller is a CI validation that did not run.
• Keep the cluster in its last good state. The GitOps apply path is fail-stop. The operator’s job is to read the error and fix the manifest.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover package validation in CI, with the same fail-stop discipline.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers playbook syntax validation in CI, with the same revert-not-reset rule.

Quiz