Deployment passes CI but fails health checks

Reported symptoms

●The deploy job in CI exited 0; `kubectl apply` (or `argocd app sync`) reported success
●The Argo CD `Application` shows `Sync Status: Synced` to the new revision but `Health Status: Progressing` (then `Degraded`)
●`kubectl get pods -n <ns> -l app.kubernetes.io/name=<name>` shows the new ReplicaSet with `READY 0/<n>` and `RESTARTS` climbing
●The previous ReplicaSet is still serving traffic; the new one never reports ready
●`kubectl describe pod <new-pod>` shows `Liveness probe failed: ...` and `Readiness probe failed: ...` with HTTP 404 or `connection refused`
●`kubectl logs <new-pod> --previous` shows the application started cleanly and is listening on the expected port — it is the probe that disagrees with the application
●Rolling the manifest back to the previous SHA returns the Application to `Healthy` within one reconcile interval
●No CI step ever ran `kubectl exec ... wget -O- http://127.0.0.1:<port>/<path>` or equivalent to verify the path the probe uses

Evidence

· `argocd app get <app>` shows `Sync Status: Synced`, `Health Status: Degraded`, with the last operation status `Running -> Failed` and reason `ProgressingTimeout`
· `kubectl get pods -n <ns>` shows the new pods in `CrashLoopBackOff` with `Back-off restarting failed container`
· `kubectl describe pod <new-pod>` Events include `Warning Unhealthy ... Readiness probe failed: HTTP probe failed with statuscode: 404`
· The deployed manifest sets `readinessProbe.httpGet.path: /healthz`, but `kubectl exec <new-pod> -- wget -qO- http://127.0.0.1:8080/healthz` returns `404 Not Found` while `/health` and `/livez` return 200
· The image was rebuilt at the same commit; the Dockerfile exposes port 8080 and the binary serves `/livez` and `/readyz` only — `/healthz` was the framework default the developer copy-pasted
· The previous ReplicaSet had `readinessProbe.httpGet.path: /livez` (the framework-correct path); the new manifest set `/healthz` and was merged without a deploy test
· The CI workflow's `deploy` job runs `kubectl apply --dry-run=server` and `kubeconform` but never curls the in-cluster endpoint after rollout
· Argo CD's `health.lua` for the resource reports `Ready: false` because the Deployment is `ProgressingDeadlineExceeded`

Diagnosis and resolutionclick to reveal

Root cause

CI verified that the manifest was well-formed and that the container started, but never verified that the in-cluster pod actually answered the readiness path the manifest declared. The `kubeconform` and `kubectl apply --dry-run=server` checks confirm only that the YAML is valid Kubernetes; they do not run the application. The developer changed the readiness path from `/livez` to `/healthz` (a path the new framework does not expose) and the merge was approved on the diff alone — the change was one line, the consequence was that no pod ever became ready, and Argo CD's `Progressing` timeout is what finally surfaced it. The structural failure is the absence of a post-deploy probe check: a CI/CD system that can declare "synced" without verifying "healthy" will, on every rollout, rely on the controller's degraded detection to find out whether the deploy worked.

Remediation

Roll back the manifest to the previous SHA immediately: `argocd app rollback <app>` or revert the readiness path in git, push, and let Argo CD re-sync. Then add a post-deploy probe check to CI that fails the deploy if the in-cluster pod does not answer the readiness path: `kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=<name> --timeout=120s` followed by `kubectl exec <pod> -- wget -qO- http://127.0.0.1:8080<reppath>` and a 200 check. Treat the readiness path as a contract between the manifest and the binary; the contract should be enforced by a test in the image build (the binary should refuse to start if its registered probe path returns non-2xx from the same server) and by the deploy job. Finally, raise Argo CD's `progressingDeadlineSeconds` is the wrong knob — instead, add an `argocd app wait <app> --health` step to the deploy job so that "synced" is not accepted as "deployed". `argocd app wait` returns non-zero if the Application is not `Healthy` within the timeout, and that exit code is the signal the pipeline needs to either promote, hold, or roll back.

Verification

The Application returns to `Health Status: Healthy` after the rollback, within one reconcile interval. A new deploy job that re-introduces the broken readiness path fails the pipeline at the `argocd app wait --health` step before any downstream promotion. `kubectl exec <pod> -- curl -fsS http://127.0.0.1:<port><probe-path>` returns 2xx for every probe path declared by every container shipped by the team. The readiness path in the manifest matches a path the binary actually serves, and that match is now enforced by a unit test in the image build, not by human review.

Prevention

Synced is not deployed. The deploy job must include `argocd app wait <app> --health --timeout 5m` (or the equivalent Flux `kustomize controller` health check), and that step must gate every downstream promotion. Make the readiness probe path a build-time contract: a start-up test in the binary that hits its own declared probe path and exits non-zero if the response is non-2xx, so a wrong path fails the image build, not the production rollout. Treat probe path changes as a two-line PR: change the path in the binary, change the path in the manifest, in the same commit. Add a `conftest` policy that rejects any Deployment whose `readinessProbe.httpGet.path` does not match a configured allowlist of paths the application framework registers, so a typo cannot merge even if a reviewer is in a hurry. The principle is that the readiness probe is a contract between the manifest and the binary, and contracts belong in policy, not in reviewer attention.

“Synced” is what Argo CD can prove about a manifest; “healthy” is what the application must prove about itself. The deploy is not done until both are true, and the pipeline must treat the second as a gate, not as an observation.