Source Hydrator and the mirror — Argo CD sources that are not the application repo

The Source Hydrator and the mirror pattern are the answers to a question that the two-repo pattern leaves open: where does the CI pipeline write the new digest? In a strict two-repo design, the CI pipeline writes into the environment repo by opening a pull request. That works. It is also slow, requires a bot identity in the environment repo, and gives the CI pipeline a write path to the cluster’s source of truth. The Source Hydrator and the mirror pattern are the alternative: the CI pipeline writes into a generated repo, and the controller reads from that.

What the Source Hydrator is

The Source Hydrator is an Argo CD pattern (introduced as a proposal and progressively integrated) that lets an Argo CD Application declare its source as a hydrator rather than a plain Git reference. The hydrator is a side process that runs in the controller: a CI pipeline publishes an artifact (a Helm chart, a Kustomize overlay, a plain manifest set) to a location the hydrator can read, and the hydrator produces a Git commit in a generated repo that references the artifact.

flowchart LR
    A["App repo: source code"] --> CI["CI pipeline"]
    CI --> R["OCI artifact registry"]
    CI --> H["Source Hydrator"]
    H --> G["Generated repo: pinned manifests"]
    G --> AC["Argo CD controller"]
    AC --> K["Kubernetes cluster"]

The controller’s source is the generated repo, not the application repo. The CI pipeline never writes to the controller’s source - it writes to the hydrator, and the hydrator writes to the generated repo. The trust boundary between CI and the cluster is the hydrator.

What the mirror pattern is

The mirror pattern is the simpler version of the same idea. The CI pipeline writes desired state into a mirror repo - a dedicated repo per environment that contains only the rendered manifests, with no application code. The controller reads from the mirror repo. The application repo is not touched by the controller at all.

argocd app set payment-api \
  --repo https://github.com/example/mirror-prod \
  --path payment-api \
  --revision-policy tracked

The mirror repo can be rendered by a CI pipeline that runs helm template or kustomize build and commits the output. The controller reads the rendered output, not the chart. The CI pipeline is the only writer to the mirror repo.

Why the boundary matters

The two-repo pattern established that the cluster trusts the environment repo, not the application repo. The Source Hydrator and the mirror pattern strengthen that trust boundary by removing the application repo’s manifests from the picture entirely. The CI pipeline writes the rendered manifest (or the hydrator writes it) into a generated repo; the controller reads from the generated repo; the application repo is read only by the CI pipeline.

flowchart TB
    subgraph Trust["Trust boundary"]
        CI["CI pipeline"] -->|writes| GR["Generated repo"]
        GR -->|read only by| AC["Argo CD"]
    end
    subgraph Out["Outside the boundary"]
        APP["Application repo"]
        RG["OCI registry"]
    end
    APP --> CI
    RG --> CI
    AC --> K["Cluster"]

The CI pipeline is now the only writer to the trust boundary. The CI pipeline’s compromise is the only attack that can change production state, and the CI pipeline is the only place to audit the production change.

Comparing the two

The Source Hydrator and the mirror pattern are different implementations of the same idea. The Source Hydrator is an Argo-CD-native feature: the controller runs the hydrator as a sidecar, and the generated repo lives in a known location the hydrator manages. The mirror pattern is implementation-agnostic: the CI pipeline writes the mirror repo (or pushes a commit via a bot), and the controller reads it as a plain Git source.

The choice depends on the controller and the team:

• Argo CD with Source Hydrator: the hydrator is a managed sidecar, the generated repo is internal, and the CI pipeline publishes to the hydrator’s API. The boundary is the hydrator.
• Argo CD with mirror pattern: the CI pipeline publishes a Helm chart or a kustomize build output to a mirror repo, and the controller reads the mirror repo as a plain source. The boundary is the mirror repo and the CI bot’s credentials.
• Flux with mirror pattern: Flux’s GitRepository and OCIRepository resources support both shapes. The mirror repo is read by a GitRepository; the OCI artifact is read by an OCIRepository.

Under the hood

The Source Hydrator proposal is interesting because it moves the GitOps trust boundary from the Git repository to the controller itself. In the classical two-repo pattern, the trust boundary is the Git host: the controller reads the environment repo, and the Git host enforces who can write to it. In the Source Hydrator pattern, the controller reads from a generated repo that the controller (or its sidecar) writes to. The trust boundary is the controller’s process boundary, not the Git host’s permission model.

This is a stronger boundary in one sense: the Git host cannot be phished, because only the controller can write the generated repo. It is a weaker boundary in another sense: the controller is now the trust anchor, and the controller’s compromise is a production compromise. The model is right for teams that already trust the controller (which is everyone running Argo CD in production) and not right for teams that have not yet built that trust.

Production discipline

The production rules for the hydrator and the mirror pattern:

1. The CI pipeline is the only writer to the controller’s source. The controller’s source (generated repo or mirror repo) accepts commits only from the CI bot’s credentials. The bot’s credentials are scoped to that repo and rotated on a separate schedule from the application repo’s credentials.
2. The generated repo is read-only by humans. No engineer edits the generated repo directly. A change that needs to happen in production is made in the application repo, goes through CI, and lands in the generated repo via the CI pipeline.
3. The CI pipeline’s run log is the production audit log. The retention and access controls on the CI pipeline’s run log must match the controls on the generated repo, because the audit answer “what changed in production?” comes from the CI run log.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXVI-01 (Application vs Environment repos) is the model the hydrator strengthens; Part LXXIV (GitOps Controllers) is the controller layer that hosts the hydrator sidecar.
• Kubernetes for Production Sysadmins - Part XXIV (Supply Chain) covers the artifact trust chain that the hydrator depends on.
• Terraform for Production Sysadmins - Part XXVIII (CI for Terraform) covers the analogous pattern of CI rendering Terraform plans and committing them to a separate repo.

Quiz