Deployment, ReplicaSet, and other workload controllers

The workload controllers (Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob) are the cluster’s mechanism for turning declarations into running Pods. Each runs in the kube-controller-manager; each follows the reconcile loop pattern. This lesson walks each controller and its production behaviour.

The Deployment controller

The Deployment controller’s role:

• Watch Deployment objects.
• Compute the desired ReplicaSets (current spec + history).
• Roll forward (new ReplicaSet with new spec) and roll back (revert to old ReplicaSet).
• Pause / resume rollouts.

flowchart LR
    D["Deployment spec"] --> DC["Deployment controller"]
    DC -->|reconcile| RS1["ReplicaSet v1 (old)"]
    DC -->|reconcile| RS2["ReplicaSet v2 (new)"]
    RS1 --> RP["ReplicaSet controller"]
    RS2 --> RP
    RP -->|reconcile| Pods["Pods"]

The Deployment controller does not manage Pods directly. It manages ReplicaSets.

$ kubectl rollout status deployment web -n prod

deployment "web" successfully rolled out

A Deployment rollout

sequenceDiagram
    autonumber
    participant U as User
    participant DC as Deployment controller
    participant RS1 as ReplicaSet v1
    participant RS2 as ReplicaSet v2
    participant API as API server
    U->>API: kubectl set image deployment/web container=new
    API-->>DC: Deployment updated
    DC->>DC: compute new desired state
    DC->>API: create ReplicaSet v2
    DC->>API: scale up RS2, scale down RS1 (rolling update)
    loop rolling update
        DC->>API: scale RS2 +1, RS1 -1
        API-->>RS2: new Pod
        API-->>RS1: old Pod deleted
    end
    DC-->>U: rollout complete

The rollout is a step-by-step reconciliation: the Deployment controller does not scale RS2 to 100% and RS1 to 0% in one step; it scales gradually.

The ReplicaSet controller

The ReplicaSet controller’s role:

• Watch ReplicaSet objects.
• List the Pods owned by the ReplicaSet.
• Reconcile: if spec.replicas differs from observed Pods, create or delete Pods.

ReplicaSet: replicas=5
  Actual Pods: 3
  Controller action: create 2 more Pods

ReplicaSet: replicas=5
  Actual Pods: 7
  Controller action: delete 2 Pods

The controller’s create and delete actions are recursive: it scales the ReplicaSet to match the spec.

The StatefulSet controller

The StatefulSet controller’s role is unique:

• Ordered management: Pods are scaled up/down in index order (pod-0, pod-1, …, pod-N).
• Stable identity: each Pod keeps its name on reschedule.
• Persistent storage: each Pod keeps its PVC.

sequenceDiagram
    autonumber
    participant SSC as StatefulSet controller
    participant API
    SSC->>API: scale from 1 to 3
    loop ordered
        SSC->>API: create Pod web-1 (waits for ready)
        API-->>SSC: ready
        SSC->>API: create Pod web-2 (waits for ready)
        API-->>SSC: ready
    end

Each Pod waits for the previous Pod to be ready before the next is created. The scale-down is reverse-ordered: web-2 is deleted first, then web-1, then web-0.

The DaemonSet controller

The DaemonSet controller’s role:

• Watch DaemonSet objects.
• Reconcile: ensure each matching node has one Pod.

DaemonSet: spec.selector=role=monitoring
  Nodes matching: 5
  Pods running: 4
  Controller action: create Pod on the missing node

DaemonSet Pods are normally not eligible for eviction during node drains; their lifecycle is tied to the node.

The Job controller

The Job controller’s role:

• Watch Job objects.
• Manage Pods to completion.
• Honour completions, parallelism, backoffLimit, activeDeadlineSeconds.

flowchart LR
    J[Job spec] -->|completions=5| JC[Job controller]
    JC -->|parallelism=2| P[Create 2 Pods]
    P -->|run| RES[1 Pod finishes]
    JC -->|create replacement| P2[Pod #4]

The Job controller:

• Creates up to parallelism Pods.
• When a Pod succeeds, counts toward completions.
• When a Pod fails, retries up to backoffLimit times.
• Honours activeDeadlineSeconds (Job must finish within the deadline).

The CronJob controller

The CronJob controller:

• Watch CronJob objects.
• On schedule, create a Job.
• Manage the history (successful, failed).

CronJob schedule: "0 * * * *"
  Every hour, the controller creates a Job
  The Job runs (1 Pod)
  After completion, the Job's history is recorded

CronJob uses the Job’s controller for the underlying Pods. The CronJob controller is a thin schedule layer.

The controller flags

Each controller has flags:

# Deployment
--concurrent-deployment-syncs=5
--deployment-controller-sync-period=10s

# ReplicaSet
--concurrent-replicaset-syncs=5

# Job
--concurrent-job-syncs=5
--job-controller-workers=10

The sync concurrency is the parallelism of reconcile loops. Higher values mean more concurrent reconciles; the cost is API server load and CPU.

Production tuning:

• Default is fine for small clusters.
• For large clusters, increase sync counts to keep up.
• Monitor controller-reconcile-duration p99 for slow controllers.

The garbage collector

The garbage collector (GC) removes dependent objects when their owner is deleted. The relationship:

flowchart LR
    O[Owner: Deployment] -->|owns| R[ReplicaSet]
    R -->|owns| P[Pod]
    G[Garbage collector] -->|on owner delete| R
    G -->|cascade| P

When a Deployment is deleted:

• The ReplicaSets owned by the Deployment are deleted (cascade).
• The Pods owned by each ReplicaSet are deleted (cascade).
• Background propagation deletes dependents in the background; foreground waits for them.

kubectl delete deployment web --cascade=foreground
# Wait for the ReplicaSets and Pods to be deleted before
# the Deployment is removed from etcd

The metrics

- controller_reconcile_total{controller="deployment"}
- controller_reconcile_total{controller="replicaset"}
- controller_reconcile_duration_seconds{controller="deployment"}
- workqueue_depth{controller="deployment"}
- workqueue_depth{controller="replicaset"}

A growing work queue depth for a specific controller indicates that controller is falling behind.

Quiz

Production discipline

• Watch the work-queue depth. A growing queue for Deployment controller is a slow rollout.
• Verify PDBs do not block rollouts. A PDB with minAvailable > replicas is a guaranteed rollout block.
• Use the controller-reconcile counters. They surface slow controllers.
• StatefulSet is not for stateless workloads. Using it for stateless workloads adds overhead and complexity.
• CronJob history can grow. Set successfulJobsHistoryLimit and failedJobsHistoryLimit to bound.

The workload controllers are the cluster’s automation. Operating them well is operating the cluster’s reconciliation.