Hybrid architectures — when neither the pure push nor the pure pull model fits

Neither pure push nor pure pull is the right tool for every system. Production deployments often combine the two: a push step for bootstrapping, a pull loop for steady-state reconciliation, and a mixed pattern for systems that are not Kubernetes-shaped. This lesson is about how teams combine both models without losing the security properties of either.

The spectrum in one picture:

flowchart LR
    P["Pure push: CI applies"] --> H["Hybrid: push bootstraps, pull reconciles"]
    H --> Pu["Pure pull: controller reconciles"]

The hybrid pattern is what most production systems actually run. Naming the patterns is the first step; combining them well is the second.

Why bootstrapping forces a push step

A new cluster has nothing in it: no controller, no GitOps loop, no reconciliation cadence. The first time someone wants to deploy Argo CD or Flux to a fresh cluster, they have to push state into it - there is no other option. The push step is the bootstrap.

flowchart LR
    Boot["CI runner"] -->|"push"| Empty["Empty cluster"]
    Empty --> Argo["Argo CD controller installed"]
    Argo -->|"pull loop begins"| Repo["Git repository"]

Three patterns handle this:

1. CI installs Argo CD / Flux. A pipeline applies the controller’s manifests from outside. This is a push step, and it is unavoidable.
2. A managed control plane. EKS Auto Mode, GKE Autopilot, or a similar service installs the controller as part of cluster creation. The push step is owned by the cloud provider.
3. A bootstrap controller. A minimal controller installed by an out-of-band tool (Terraform, eksctl, gcloud) reads a Git repository and installs the real controller. The bootstrap controller is itself a push-deployed thing.

The push step is acceptable at bootstrap because the cluster is empty: there is nothing to compromise. The discipline is to make the push step a one-time operation that is replaced by the pull loop the moment the controller is running.

Multi-cluster fan-out

A single controller cannot reconcile every cluster in a fleet; each cluster runs its own controller (or sharded controllers for very large clusters). The fan-out - “tell every cluster what state to be in” - is usually push-model work:

flowchart LR
    Pipe["CI pipeline"] -->|"push install"| C1["Cluster 1 controller"]
    Pipe -->|"push install"| C2["Cluster 2 controller"]
    Pipe -->|"push install"| C3["Cluster 3 controller"]
    C1 -->|"pull"| Repo["Git repository"]
    C2 -->|"pull"| Repo
    C3 -->|"pull"| Repo

The pipeline pushes the controller into each cluster once. From that point on, each cluster pulls. The push is the install; the pull is the operation. The two coexist.

The operator pattern for non-Kubernetes systems

Not every system can host a controller. A legacy application server, a network appliance, a cloud-provider resource managed by an imperative CLI - none of these have a reconciliation loop of their own.

The operator pattern is the pull-model analogue for these systems: a controller that runs somewhere (often in Kubernetes, sometimes on a VM) reads Git, computes a diff, and applies state to the system via its imperative API. The system being managed does not know it is being managed by GitOps; the controller presents a pull-model abstraction over an imperative target.

terraform apply -auto-approve

The Terraform operator pattern is the canonical example: a controller reads Terraform configurations from Git, runs terraform plan and terraform apply on a schedule, and stores state in a backend the controller manages. The Terraform binary is the imperative API; the controller wraps it in a pull loop.

Mixed-tenant pipelines

Some teams run shared CI for many clusters, some of which are GitOps-managed and some of which are push-managed. A common pattern is:

flowchart LR
    Repo["Git repository"] --> CI["Shared CI"]
    CI -->|"push"| Legacy["Legacy clusters (push)"]
    CI -->|"gitops push"| Argo["GitOps controller install"]
    Argo -->|"pull"| Modern["Modern clusters (pull)"]

Legacy clusters stay on push because the migration cost to GitOps is not yet justified. Modern clusters run on pull. The shared CI pushes to the legacy clusters directly and installs GitOps controllers in the modern ones. The blast-radius discipline is applied cluster-by-cluster, not as a single organisational rule.

Keeping the security properties

Combining push and pull does not have to weaken either. Three disciplines:

1. The push step is minimal. The push installs the controller and nothing else. It writes the smallest possible set of manifests - the controller, its ServiceAccount, its Role, its RoleBinding. No application workloads go through the push step.
2. The push credentials are bootstrap credentials. They are scoped to “install the controller”, not to “do everything the controller will ever do”. They are rotated after the controller is up.
3. The controller takes over as quickly as possible. The longer a push credential is active, the wider its blast radius in time. A push step measured in minutes is a different blast radius than a push step measured in days.

Production discipline

1. Name the boundary. Every pipeline should declare which step is push and which step is pull, and which credentials each step uses. An undeclared boundary is a credential that outlives its purpose.
2. Push is for install; pull is for operate. Once a system has a controller, the controller takes over. The push credential should not be the operating credential.
3. Audit the transition. The moment a cluster moves from push-installed to pull-operated is the moment a credential rotates. That transition is an audit event.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on cluster bootstrapping cover the push-install step in detail.
• Terraform for Production Sysadmins - Parts on the operator pattern cover the pull analogue for infrastructure.
• Ansible for Production Sysadmins - Parts on AWX / Tower cover the pull analogue for configuration management.

Quiz