Sync waves and phasing — ordering, dependencies, and the wave annotation

A cluster with fifty Applications has fifty reconcile loops. Without ordering, those loops are independent: each one applies its own diffs without waiting for the others. Most of the time that is fine; sometimes it is not. A CRD Application that defines a custom resource and a workload Application that consumes that custom resource must apply the CRD first. Without ordering, the consumer fails; with ordering, the dependency is expressed and the controller respects it.

Why ordering matters

The kinds of dependencies between Applications:

• CRD before consumer. The CRD Application’s CustomResourceDefinition must exist in the cluster before the consumer Application’s custom resources are applied. Without the CRD, the consumer’s apply fails.
• Namespace before workload. A workload Application that targets a new namespace needs the namespace to exist first. Without the namespace, the apply fails.
• ConfigMap before consumer. A ConfigMap consumed by a Pod via envFrom must exist before the Pod is rolled.
• Database before service. A Deployment that needs a database connection must wait for the database to be ready before its first rollout; otherwise the readiness probe fails.

Each dependency has a direction. The challenge is expressing the direction in a way the controller can act on.

Sync waves in Argo CD

Argo CD’s wave mechanism is the argocd.argoproj.io/sync-wave annotation on the Application manifest. The wave number is an integer; lower waves apply first, higher waves apply later.

metadata:
  name: payment-crd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
metadata:
  name: payment-namespace
  annotations:
    argocd.argoproj.io/sync-wave: "1"
---
metadata:
  name: payment-api
  annotations:
    argocd.argoproj.io/sync-wave: "2"

The wave-0 Application (the CRD) applies first. The wave-1 Application (the namespace) applies next. The wave-2 Application (the workload) applies last.

gantt
    title Argo CD sync waves - CRD, namespace, workload
    dateFormat HH:mm
    axisFormat %H:%M
    section Wave 0
    CRD applied       :active, 00:00, 2m
    section Wave 1
    Namespace applied :active, 00:02, 2m
    section Wave 2
    Workload applied  :active, 00:04, 5m

The default wave number is zero. An Application without the annotation applies in wave 0. Production wave numbers are typically 0 through 5; deeper nesting is a sign that the dependency graph has become unmanageable.

Sync phases

Waves are subdivided by phase. A sync phase is a string that controls the order within a wave:

• PreSync. Runs before the main apply. Used for database migrations, backup hooks, or pre-flight checks.
• Sync. The main apply.
• PostSync. Runs after the main apply. Used for smoke tests, cache warmers, or notifications.

The wave determines when a hook runs in the overall order; the phase determines where it runs within the Application.

dependsOn in Flux

Flux’s mechanism is spec.dependsOn on a Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: payment-api
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: apps
  path: ./clusters/production/payment-api
  prune: true
  dependsOn:
    - name: payment-crd
    - name: payment-namespace

The payment-api Kustomization waits for payment-crd and payment-namespace to reconcile and report Ready=True before applying. The dependency is a name, not a wave number; Flux’s controller resolves the dependency graph before each reconcile.

The Flux CLI form:

flux create kustomization payment-api \
  --source=GitRepository/apps \
  --path=./clusters/production/payment-api \
  --depends-on=payment-crd \
  --depends-on=payment-namespace

--depends-on repeats for multiple dependencies. The order matters for prune ordering but not for apply ordering; the controller builds a DAG and applies the leaves last.

Designing waves so a failure stalls the smallest subset

The principle: the wave that fails should be the smallest possible subset of the fleet. Three rules:

1. Wave 0 is for invariants. Namespaces, CRDs, RBAC. The things every other wave needs.
2. Wave 1 is for shared services. Databases, observability, service meshes. The things multiple wave-2 Applications need.
3. Wave 2 and above are for individual workloads. Each workload’s failure stalls only its own Application and any downstream wave that explicitly depends on it.

A failure in a wave-2 workload stalls the workload, not the cluster. A failure in a wave-0 CRD stalls every consumer.

Phasing pitfalls

The mistakes teams make with waves:

• Too many waves. A team with ten waves has a DAG too complex to reason about. Production teams cap at five.
• Cross-wave dependencies that are not in the DAG. A wave-3 workload that depends on a wave-4 workload that depends on a wave-2 workload is a cycle; the controller detects it and refuses to apply.
• Implicit dependencies via shared resources. Two Applications that share a ConfigMap have a dependency the controller cannot see. The failure mode is the consumer’s ConfigMap is missing because the producer’s chart dropped it.

Production discipline

1. Waves are integer-ordered; lower applies first. A team that uses waves 0-3 with explicit gaps (0, 10, 20) can insert intermediate waves without renumbering.
2. Wave 0 is for invariants. CRDs, namespaces, RBAC, network policies.
3. Wave failures stall downstream waves. The smaller the failed wave’s footprint, the smaller the blast radius.
4. dependsOn is for cross-Kustomization ordering in Flux. The name is the dependency; the DAG is the order.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXVII-04 (Sync policies and windows) is the policy frame; Part LXXVI (GitOps Repo Architecture) is where the wave structure is encoded in the repo layout.
• Kubernetes for Production Sysadmins - Part XVI (Operators and Controllers) covers the reconcile loop that respects the wave order.

Quiz