Desired state in Git — the manifest repository structure

The manifest repository is the source of truth for cluster configuration. This lesson walks the repository structure, the manifest format choices, the Git workflow, and the operational discipline.

The repository structure

flowchart LR
    A[manifests-repo] --> B["clusters/"]
    A --> C["apps/"]
    A --> D["infrastructure/"]
    B --> B1["prod/"]
    B --> B2["staging/"]
    B --> B3["dev/"]
    C --> C1["billing/"]
    C --> C2["checkout/"]
    C --> C3["catalog/"]
    D --> D1["cert-manager/"]
    D --> D2["ingress-nginx/"]
    D --> D3["argocd/"]

The typical structure:

• clusters/ — per-cluster configuration. Each cluster has its own directory with the apps that run on it, plus environment-specific values.
• apps/ — per-app base configuration. Each app has a base manifest and overlays for each environment.
• infrastructure/ — cluster add-ons (cert-manager, ingress-nginx, Argo CD itself, monitoring stack).

The structure separates concerns: clusters know what apps to run; apps know how to run; infrastructure provides the platform.

The manifest format

flowchart LR
    A[Manifest format] --> B[Raw YAML]
    A --> C[Kustomize]
    A --> D[Helm]
    B --> B1["Simple, no tooling"]
    C --> C1["Base + overlays, environment-specific"]
    D --> D1["Packaged apps, templated values"]

The format choices:

• Raw YAML. Simple; no tooling needed. Good for small manifests that don’t vary across environments.
• Kustomize. Built into kubectl; overlays for environment-specific configuration. Good for apps with environment-specific values (replicas, resources, image tags).
• Helm. Templated; values files for environment configuration. Good for packaged applications with complex configuration.

Most production manifest repositories use a combination: Kustomize for application manifests with environment overlays, Helm for third-party add-ons.

A Kustomize example

# apps/billing/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app: billing
---
# apps/billing/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod-app
resources:
  - ../../base
namePrefix: prod-
patches:
  - patch.yaml

The base has the common configuration; the overlay adds environment-specific patches (replicas, image tag, resource limits). The GitOps controller applies the overlay to the cluster.

The Git workflow

flowchart LR
    A[Branch] --> B[Commit]
    B --> C[Push]
    C --> D[Pull request]
    D --> E[Review]
    E --> F{Approved?}
    F -->|No| G[Fix]
    G --> B
    F -->|Yes| H[Merge]
    H --> I[GitOps controller syncs]
    I --> J[Cluster updated]

The workflow:

1. Branch. Create a branch for the change.
2. Commit. Make the change in YAML manifests.
3. Push. Push to the remote.
4. Pull request. Open a PR for review.
5. Review. Code review by at least one peer.
6. Approve or fix. If approved, merge; if not, fix and re-review.
7. Merge. Merge to main.
8. Sync. The GitOps controller detects the change and applies it.

The PR-based workflow ensures every change is reviewed before reaching the cluster.

The branch strategy

flowchart LR
    A[Branch strategy] --> B[Trunk-based]
    A --> C[GitFlow]
    A --> D[Environment branches]
    B --> B1["main only, short-lived feature branches"]
    C --> C1[main + develop + release branches]
    D --> D1["main + dev/staging/prod branches"]

The branch strategy choices:

• Trunk-based. One main branch; short-lived feature branches. The simplest; the most aligned with GitOps (every merge to main triggers a sync).
• GitFlow. Main + develop + release branches. More complex; suited for software with versioned releases.
• Environment branches. Main + per-environment branches (dev, staging, prod). Common in legacy GitOps setups but increasingly discouraged.

Most modern GitOps uses trunk-based: every change goes through PR review to main.

The operational failure modes

Manifest repositories fail for predictable reasons:

• Repository compromised. An attacker gains write access; can take down the cluster.
• Manifest without review. Direct pushes to main bypass the review process.
• Secrets in Git. Kubernetes Secrets in Git are a security risk; use sealed-secrets or Vault.
• Drift undetected. The controller is down or misconfigured; drift accumulates.
• Branch skew. Long-lived branches cause conflicts; the merge becomes complex.

Quiz

Production discipline

The manifest repository in production rests on five non-negotiable elements:

• Access control. Write access to main is restricted; PR review required for every merge.
• PR review rigour. Every PR is reviewed substantively, not rubber-stamped.
• Secrets are external. Use sealed-secrets, Vault, or External Secrets Operator.
• Trunk-based workflow. Short-lived branches; frequent merges to main.
• Backup and audit. The repository is backed up; audit log of every change.

The manifest repository is production infrastructure. Treat it as such: access control, review, backup, monitoring.