Custom controllers — extending the cluster with code

Custom controllers extend the cluster with code that reacts to API state. They are the cluster’s mechanism for declarative state management beyond the built-in controllers. Every CRD-driven operator is a custom controller. This lesson walks the controller pattern, the kubebuilder / operator-sdk toolchain, and the discipline of running custom controllers in production.

The controller pattern

A controller’s components:

flowchart LR
    AS[API server] -->|watch CR| INF[Informer]
    INF -->|events| Q[Work queue]
    Q -->|dequeue| R[Reconcile loop]
    R -->|write CR| AS
    R -->|write secondary| K8s[Pods, Services, etc.]
    R -->|write external| EXT[External system]

The standard components:

• Informer / Reflector. Watches the API server for state changes; populates a cache.
• Work queue. Receives events from the informer; buffers them for processing.
• Reconcile. Runs the controller’s logic; reads state, computes diff, writes corrections.
• Indexer. In-memory cache of objects indexed by labels / annotations.

type Reconciler interface {
    Reconcile(ctx context.Context, req Request) (Result, error)
}

The Reconcile function is the heart of a controller. It is idempotent: the same input produces the same output.

The kubebuilder toolchain

kubebuilder is the canonical tool for scaffolding Kubernetes controllers:

# Initialise a project
kubebuilder init --domain mycompany.com --repo github.com/myorg/myoperator

# Create a CRD and controller
kubebuilder create api --group apps --version v1alpha1 --kind Application

# Build and test
make manifests
make install  # installs CRDs
make run      # runs the controller locally
make test

The toolchain generates:

• api/<version>/<kind>_types.go: the CRD types in Go.
• controllers/<kind>_controller.go: the Reconcile function.
• config/: Kubernetes manifests (CRD, RBAC, Deployment).
• Makefile with all the build / test / deploy targets.

The Reconcile function

A Reconcile’s shape:

func (r *ApplicationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the Application
    var app appsv1alpha1.Application
    if err := r.Get(ctx, req.NamespacedName, &app); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Reconcile based on spec
    desiredReplicas := app.Spec.Replicas
    if desiredReplicas == 0 {
        // 3. Delete managed Pods if requested
        return r.cleanup(ctx, &app)
    }

    // 4. Reconcile Pods (find or create as needed)
    return r.reconcilePods(ctx, &app)
}

The function reads the CR, decides what to do, writes back as needed, and returns a Result:

type Result struct {
    Requeue bool
    RequeueAfter time.Duration
}

A Result{Requeue: true} causes the controller to re-enqueue the request; a Result{RequeueAfter: 30*time.Second} schedules a future re-enqueue.

The ownership and cascading

A controller typically owns secondary resources, expressed via metav1.OwnerReference:

metadata:
  name: web
  ownerReferences:
  - apiVersion: apps/v1alpha1
    kind: Application
    name: my-app
    uid: ...
    controller: true
    blockOwnerDeletion: true

When the Application is deleted, garbage collection deletes the owned Pods (cascade) if the controller has not deleted them itself.

$ kubectl get application my-app -o yaml | head -20

...

The watch predicates

Controllers can filter watch events:

func (r *ApplicationReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.Application{}).
        WithEventFilter(predicate.GenerationChangedPredicate{}).
        Complete(r)
}

GenerationChangedPredicate triggers reconcile only when the spec’s generation changes (i.e., on actual changes, not status updates). This reduces reconcile calls substantially for CRDs that update status frequently.

The webhook-based validation

Custom resources can have admission webhooks:

func (r *Application) ValidateCreate() (admission.Warnings, error) {
    if r.Spec.Replicas < 0 {
        return nil, fmt.Errorf("replicas must be >= 0")
    }
    return nil, nil
}

Webhooks can be:

• Validating — reject invalid objects.
• Mutating — modify the object before persistence.
• Conversion — translate between versions (for multi-version CRDs).

The webhook server runs as a separate Deployment (potentially) or sidecar; it is registered as ValidatingWebhookConfiguration / MutatingWebhookConfiguration in the API server.

The CRD versioning

Custom resources can have multiple versions:

versions:
- name: v1
  served: true
  storage: true
- name: v1beta1
  served: true
  storage: false

The cluster serves both; storage is the v1 version. The conversion webhook translates between versions for clients.

The production discipline

A custom controller in production must:

• Run as a Deployment with replicas ≥ 2. Leader election elects one active; the others standby.
• Use RBAC bounded to its CRDs. The controller’s ServiceAccount must have exactly the verbs it needs.
• Set resource requests and limits. A misbehaving controller that consumes unbounded memory is a cluster hazard.
• Implement graceful shutdown. On SIGTERM, finish in-flight reconciles before exit.
• Profile the Reconcile function. A 1-second reconcile over 1000 objects per event is a cluster-wide regression.

flowchart LR
    C[Controller] -->|writes| K8s[API]
    C -->|reads| K8s
    C -->|read metrics| P[Prometheus]
    C -->|logs| L[Logging]

The controller metrics

Custom controllers expose Prometheus metrics:

• controller_runtime_reconcile_total{controller="...", result="success|error|requeue"}
• controller_runtime_reconcile_duration_seconds{controller="..."}
• controller_runtime_work_queue_depth{controller="..."}
• controller_runtime_max_concurrent_reconciles{controller="..."}

A growing work queue depth signals a slow controller.

The testing patterns

kubebuilder scaffolds tests with envtest:

var _ = Describe("Application Controller", func() {
    It("should reconcile Application into Pods", func() {
        ctx := context.Background()
        app := &appsv1alpha1.Application{...}
        Expect(k8sClient.Create(ctx, app)).To(Succeed())

        controller := SetupController(ctx)
        _, err := controller.Reconcile(ctx, ctrl.Request{NamespacedName: ...})
        Expect(err).To(BeNil())

        var pods corev1.PodList
        Expect(k8sClient.List(ctx, &pods)).To(Succeed())
        Expect(pods.Items).To(HaveLen(1))
    })
})

envtest spins up a local API server + etcd for tests.

Quiz

Production discipline

• Run with 2+ replicas. Leader-elected; the standby takes over on failure.
• Tight RBAC for the controller’s ServiceAccount. Grant exactly the verbs needed.
• Resource requests and limits. A misbehaving controller with unbounded resources is a hazard.
• Implement Reconcile idempotently. The retry behaviour depends on it.
• Profile and instrument. Metrics + traces for every Reconcile call.

Custom controllers extend the cluster with declarative state management. Operating them well is operating the cluster’s automation extension.