Secret dependency failure — External Secrets Operator cannot reach Vault

The sixth failure mode closes the Part by exposing the boundary between what GitOps reconciles and what GitOps does not reconcile. A GitOps controller manages Kubernetes resources. The secrets those resources reference, when managed by the External Secrets Operator (ESO) against HashiCorp Vault, live outside the GitOps loop: in Vault’s storage, under Vault’s auth, governed by Vault’s policy. When the chain between Git and Vault breaks, the GitOps controller reports Synced and Healthy because the Kubernetes resource is in place - but the secret the resource depends on is missing or stale, and the application inside the Pod cannot start.

The symptom in ExternalSecret status

The diagnostic starts with the ESO resource:

kubectl get externalsecret -A

The output shows READY=False or READY=True with a status age older than the configured refresh interval. The canonical read:

kubectl describe externalsecret $NAME

The Status block carries a Conditions array with the failure reason. Common shapes:

• SecretSyncedError: no secret data found. The path in Vault does not exist or the policy does not grant read.
• SecretSyncedError: invalid credentials. Token or service account auth has failed.
• SecretSyncedError: dial tcp: i/o timeout. Network reachability has failed.

flowchart LR
    A[GitOps controller] --> B["ExternalSecret resource"]
    B --> C[ESO controller]
    C --> D{Vault reachable?}
    D -- yes --> E{Vault auth valid?}
    E -- yes --> F{Vault policy grants read?}
    F -- yes --> G[SecretSynced: True]
    F -- no --> H["SecretSyncedError: permission denied"]
    E -- no --> I["SecretSyncedError: invalid credentials"]
    D -- no --> J["SecretSyncedError: network"]

Distinguishing the three layers

The failure can live in three layers, each with a different remediation:

• Network. DNS, firewall, network policy. The ESO log shows a dial tcp or connection refused error. The diagnostic is kubectl exec into the ESO pod and nslookup vault.example.com.
• Vault auth. Token expired, Kubernetes auth role misconfigured, service account not bound. The diagnostic is vault token lookup or vault auth list.
• Vault policy. The token is valid but the policy does not grant read on the path. The Vault audit log shows a denied read.

The bootstrap-order problem

The hardest variant is the bootstrap order. ESO needs a Vault token to read secrets. The token is often delivered through a Kubernetes ServiceAccount JWT or through an init container that reads a bootstrap token from a different secret. When the cluster is fresh, the bootstrap secret does not yet exist, and the ExternalSecret cannot be reconciled. The result is a chicken-and-egg loop.

The discipline that resolves this:

• Bootstrap secret in Git, not in Vault. The initial token is delivered through a Kubernetes Secret in Git, applied by the GitOps controller. Once ESO is running, the bootstrap token is rotated for a long-lived auth method.
• Init container for Vault auth. Use a Vault Agent sidecar or Vault CSI provider that handles auth renewal.
• Refresh interval shorter than the token TTL. A 24-hour TTL with a 7-day refresh interval will expire between refreshes.

The alert set

The minimum alert set for ESO failures:

• ExternalSecret not ready. Alert when kube_custom_resource_status_condition{condition="Ready", kind="ExternalSecret"} == False for more than the refresh interval plus 30 seconds.
• Secret not refreshed. Alert when the status.refreshTime on an ExternalSecret is older than spec.refreshInterval.
• ESO controller not running. Alert when the external-secrets deployment has zero available replicas.

Production discipline

• Refresh interval is part of the manifest. A 24-hour refresh interval is a Secret that may be 24 hours stale.
• Test the bootstrap on every cluster. A new cluster’s first reconciliation is the highest-risk moment.
• Rotate Vault auth, not Vault tokens. The Kubernetes auth method renews without rotating; the token auth method requires rotation.

Cross-course references

• Linux for Production Sysadmins - Parts XII (RepoSecurity) and XXXIV (ConfigMgmt) cover the equivalent failure mode for apt/dnf repository credentials, with the same refresh-interval discipline.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover AWS Secrets Manager rotation for Terraform, with the same refresh-interval discipline.

Quiz