Secret exposed in GitOps repo (plaintext in overlay)

Reported symptoms

●A secret appears in plaintext in a Kubernetes manifest under `data` (with a base64-encoded value) or under `stringData` (plaintext) in a Production overlay
●GitHub's secret-scanning alert was raised after the commit landed, not before; push protection did not block the push
●The same secret value is present in the GitOps repo, in the cluster (`kubectl get secret -o jsonpath={.data}`), and in at least one developer's `~/.kube/config` history
●Argo CD's repo-server log shows a successful clone containing the manifest; the Application has Synced and is now serving from the cluster-side secret
●The secret-scanning provider (GitHub native, gitleaks, trufflehog) was not configured on the GitOps repo at all, or was configured with a custom rule allowlist that excluded the secret type
●The secret was reused across multiple overlays (dev, staging, production), so rotation touches every environment
●The base64 decoding of the value in `data` matches a real database password
●No pull request review caught the plaintext because the diff was reviewed at the manifest level, not at the secret-content level

Evidence

· `git -C gitops-repo log --all -p -- overlays/prod/<app>/secret.yaml | base64 -d` (where applicable) reveals a real password
· `grep -rn "kind: Secret" overlays/prod/` shows the plaintext field
· GitHub secret-scanning alerts list the secret with `state: open`, `resolution: revoked` not set, `push_protection_bypassed: true` (or `push_protection_bypassed: null` if the scanner never ran)
· `gh api repos/<org>/gitops/secret-scanning/alerts?state=open` includes the offending commit and the secret fingerprint
· `kubectl get secret <name> -n <ns> -o jsonpath={.data.password} | base64 -d` returns the same value as the in-repo secret
· The repo's `.github/secret_scanning.yml` (if present) does not list the secret type, or the org-level setting has push protection disabled
· `gh api repos/<org>/gitops/contents/.github/secret_scanning.yml` returns 404 or an empty rule set
· The commit's `git show <sha> --stat` lists `overlays/prod/<app>/secret.yaml` as the only file in a one-line change

Diagnosis and resolutionclick to reveal

Root cause

GitOps stores desired state in git. If a secret is committed in plaintext, every clone of the repo (Argo CD repo-server, developer laptops, forks, CI runners, mirrors) is a copy of the secret, and revocation is a fleet-wide exercise. The structural failure is twofold: first, the secret was committed in plaintext at all instead of being referenced from an external secret store; second, the secret-scanning push protection either was not configured for this repo or was misconfigured to allow the secret type through. The combination — no external secret system AND no push-time detection — means a developer can paste a real password into a manifest, push to `main`, and the only signal that arrives is a post-commit alert, by which time the secret is in git history, in every clone, and in the cluster. The remediation is not "scan harder"; it is "do not store secrets in git, period".

Remediation

Rotate the exposed secret first; assume it is compromised, because in a GitOps repo it is. Force-rotate the database password at the database, the API key at the provider, the cloud credential at the IAM system. Replace the in-repo secret with a reference to an external secret store: External Secrets Operator with a `SecretStore` pointing at AWS Secrets Manager, Vault, or GCP Secret Manager; Bitnami Sealed Secrets if the operator pattern is not available; Mozilla SOPS with age or KMS if the team is comfortable with key management. The manifest then contains a `Secret` whose `metadata.annotations` declare a `secrets.external-secrets.io/*` reference, and the controller materialises the secret in the cluster from the source of truth. Delete the plaintext manifest from git and purge it from history (`git-filter-repo --invert-paths --path overlays/prod/<app>/secret.yaml` followed by a force-push that is itself a break-glass operation, captured in the runbook). Add secret-scanning push protection: configure GitHub Advanced Security push protection AND a `gitleaks` or `trufflehog` pre-commit AND a CI step that runs `gitleaks detect --redact` on every PR; block the merge on any finding. Finally, audit every other overlay and every other Application for plaintext secrets — `trufflehog git file://. --branch main --only-verified` against a shallow clone.

Verification

The plaintext value no longer appears in `git log --all -p -- overlays/` for any branch. The external secret is materialised in the cluster: `kubectl get secret <name> -n <ns> -o jsonpath={.metadata.annotations.external-secrets\.io/source-uuid}` is set, and the `data.password` decodes to the rotated value, not to the previously-exposed value. Push protection is enabled: a test PR that adds a real secret pattern (using a known fingerprint from gitleaks test fixtures) is blocked at the pre-receive hook. `gh api repos/<org>/gitops/secret-scanning/alerts?state=open` shows the previously-raised alert as `state: resolved`. Every Application''s manifest tree contains only references, never plaintext values, verified by `git grep -E "(password|apiKey|token):\s+\S+" overlays/`.

Prevention

Secrets do not live in the GitOps repo. They live in a secret store, and the GitOps repo declares references. Enforce this structurally: an `AppProject`-level restriction on `kind: Secret` resources that the GitOps controller is allowed to manage, so a plaintext secret cannot be synced even if it is committed. A CI check that runs `gitleaks detect --redact` on every PR and fails on any finding — combined with a `conftest`/OPA policy that rejects any `Secret` whose `data` or `stringData` contains non-referenced values (only `metadata.annotations` declaring an external source are allowed). Enable GitHub Advanced Security push protection org-wide. For Application-level defence, External Secrets Operator is the default pattern: the manifest declares intent (`spec.data[0].secretKey`), the operator materialises it, and the secret is never in git. Drill the recovery: simulate a plaintext secret commit, run the rotation, the External Secrets wiring, the history purge, and verify the final state of the cluster. The principle is that a secret in git is a secret in every clone of git, and "every clone" is the threat model External Secrets Operator is designed to invalidate.

A secret in git is a secret in every clone of git — and the clones include every Argo CD controller, every CI runner, every developer laptop, and every fork. The structural fix is to move secrets out of git entirely: declare references, not values, and let a secret operator materialise the values at deploy time. Scan as a backstop, not as the primary defence.