Reconciliation intervals — how often, and the trade-offs

The reconciliation interval is the operational knob that determines how often the controller asks the cluster “are we converged?”. Tight intervals buy responsiveness; loose intervals buy headroom. The right value is a function of the workload, the cluster size, and the operational cost of a tick. Most teams pick a default and never revisit it; the production discipline is to pick deliberately.

flowchart LR
    A["Tight interval 30s"] --> L["Low drift window"]
    A --> P["High API-server load"]
    B["Loose interval 10m"] --> H["High drift window"]
    B --> R["Low API-server load"]
    L --> C["Hot production"]
    P --> C
    H --> D["Cold staging"]
    R --> D

The diagram frames the trade-off. Tight and loose are not absolute; they are points on a curve. The curve has two axes - responsiveness (drift window) and load (API-server pressure) - and the right operating point is per-workload.

What the interval actually controls

The interval is the upper bound on how long a commit takes to land and the lower bound on how often the controller asks the cluster for state. A 3-minute interval means:

• A commit can take up to 3 minutes to be observed by the controller and up to another 3 minutes to be applied (worst case is the commit lands just after a tick).
• The controller issues a read per tick - per application, per resource - on the configured cadence.

The interval is not a guarantee of convergence time. A tick can fail, retry, and back off, which extends the convergence time beyond the interval. The interval is the baseline convergence time under healthy conditions.

The cost of a single tick

Each tick produces API-server load. The dominant costs:

• A List per resource kind the application owns. Argo CD and Flux both issue a List on every owned kind per tick. An application with 10 Deployment kinds, 5 Service kinds, and a ConfigMap issues at least 3 List calls per tick.
• A Get per resource to fetch the live state. An application with 200 resources issues 200 Get calls per tick to compute the diff.
• A Git fetch on the source controller. Argo CD and Flux both fetch from Git on a separate cadence from the reconcile interval, but a webhook from Git can also trigger an immediate reconcile.

The total cost per tick scales with the number of resources the application owns, not with the size of the cluster. A cluster running 1000 applications at a 30-second interval produces a steady-state API-server load of 30000 Get calls per minute at the application level alone - which is enough to be a real operational concern for a small API server.

Setting the interval per workload

The interval is per-controller and per-resource. A few common patterns:

• 30 seconds for hot production. The latency budget for a hot-fix is measured in minutes; the API-server can usually absorb the cost because the application count is moderate.
• 3 minutes for default production. Most controllers ship with this as a default. Comfortable balance between latency and load.
• 10 minutes for cold staging. The latency budget for staging is loose; the API-server cost matters more.
• Per-resource override. A controller’s interval is the outer bound; individual resources can be tuned. A Certificate resource that expires in 90 days does not need a 30-second reconcile; a Deployment that fronts a payment system may need one.

flux reconcile kustomization "$KS_NAME"

This Flux CLI command forces an out-of-band reconcile without waiting for the next interval. It is the lever for “I just committed and I cannot wait three minutes”. Forcing a reconcile is an exception, not a habit; if it is happening often, the interval is wrong.

Trade-offs and failure modes

The interval has two failure modes, one at each end of the curve.

Too tight. The controller produces sustained API-server load that has to be sized for. Symptoms:

• API-server latency rises during reconcile ticks.
• The etcd backend shows steady read load from controller List calls.
• Other controllers’ queues back up because the API server is saturated.

Remediation: relax the interval, or split the application into smaller ones with their own intervals.

Too loose. The drift window is wide. Symptoms:

• A manual fix is reverted up to N minutes after the operator applies it.
• A misconfigured resource is live for up to N minutes before the loop corrects it.
• The audit trail (commit SHA to applied state) shows a long tail latency.

Remediation: tighten the interval, or add an event-driven trigger (webhook from Git) that fires an immediate reconcile on commit without waiting for the next interval.

Event-driven reconciliations

Most modern GitOps controllers support event-driven reconciliation as a complement to the polling interval. A webhook from the source - Git, OCI registry, Helm repository - fires an immediate reconcile on push, so the latency for a commit is the network round-trip plus the next tick (which is immediate, because the webhook triggered it).

Event-driven reconciliation changes the trade-off: the polling interval can be relaxed (saving API-server load) without sacrificing responsiveness, because the controller still reacts to events.

The trade-off moves from “interval versus load” to “webhook reliability versus load”. A controller that cannot reach its webhook endpoint will fall back to polling, so the polling interval must still be set to a reasonable value.

Production discipline

1. Pick the interval deliberately. The default is a starting point, not a destination. A hot production workload may need a tighter interval than the default; a cold staging workload may need a looser one.
2. Watch the API server. Reconciliation latency, etcd read latency, and API-server request rate are the three signals that the interval is too tight. A dashboard that plots all three, with controller identity as a facet, makes the diagnosis fast.
3. Use event-driven reconciliation where it is available. Webhook-driven reconciles reduce the cost of the polling interval. The interval can be relaxed to a safety-net value (say, 10 minutes) without losing responsiveness on the common case of “commit lands, controller reacts”.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on controller work-queue tuning cover the lower-level mechanics the GitOps controllers inherit.
• Terraform for Production Sysadmins - Parts on polling state cover the analogous trade-off in Terraform Cloud.
• Ansible for Production Sysadmins - Parts on AWX scheduling cover the configuration-management analogue.

Quiz