The Operator pattern — custom resources, controllers, and the reconciliation loop

The Operator pattern is the de facto standard for production stateful workloads in Kubernetes. This lesson walks the pattern, the levels, and the production discipline for selecting and deploying Operators.

What an Operator is

An Operator is a Kubernetes controller that watches a Custom Resource (CR) and reconciles the actual state with the desired state encoded in the CR.

flowchart LR
    A[Custom Resource: PostgreSQL] --> B[API server]
    B --> C[Operator: reconciliation loop]
    C --> D{State matches?}
    D -->|yes| E[No action]
    D -->|no| F[Apply changes]
    F --> G[StatefulSet, Services, Secrets, ...]
    G --> B

The components:

• Custom Resource Definition (CRD): the schema for the application (e.g., PostgreSQL, Kafka, Redis).
• Custom Resource (CR): an instance of the CRD (e.g., a specific PostgreSQL cluster).
• Controller: a Pod (Deployment) that watches the CRs and reconciles the actual state.

The CRD

The CRD defines the schema:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: postgresqls.acid.zalan.do
spec:
  group: acid.zalan.do
  scope: Namespaced
  names:
    plural: postgresqls
    singular: postgresql
    kind: PostgreSQL
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              numberOfInstances:
                type: integer
              volume:
                type: object
                properties:
                  size:
                    type: string
              teamId:
                type: string

The user creates a CR:

apiVersion: acid.zalan.do/v1
kind: PostgreSQL
metadata:
  name: my-app-db
spec:
  numberOfInstances: 3
  volume:
    size: 100Gi
  teamId: my-team

The CR is a declarative description of the desired state.

The controller

The controller is a Pod that watches the CRs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-operator
  namespace: operators
spec:
  replicas: 1
  template:
    spec:
      serviceAccountName: postgres-operator
      containers:
      - name: operator
        image: registry.opensource.zalan.do/acid/postgres-operator:v1.10.0
        args:
        - --watch-namespace=*

The controller:

• Watches PostgreSQL CRs.
• For each CR, computes the desired state (StatefulSet, Services, Secrets, VolumeSnapshots).
• Compares with the actual state (queries the API server).
• Applies changes to converge.

The level concept

Operators vary in sophistication. The community uses a “level” concept:

• Level 1: Basic Operator. Manages a single CR; performs simple provisioning.
• Level 2: Reconciliation across multiple objects. Manages StatefulSets, Services, Secrets together.
• Level 3: Application knowledge. Performs backup, restore, scaling, upgrade with the application’s procedures.
• Level 4: Production-grade. Encodes best practices, has tests, is widely adopted.
• Level 5: Auto-pilot. Self-healing, self-tuning, predictive scaling.

Most production Operators are Level 3-4. Level 5 is rare and aspirational.

The standard Operators

The production-grade Operators (Zalando, Cloud Native PG, Strimzi, ECK) are widely adopted and tested.

The deployment

Operators are deployed as Deployments:

# Install the Operator
kubectl apply -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.22/releases/cnpg-1.22.1.yaml

# Verify the Operator is running
kubectl -n cnpg-system get pods

# Create a PostgreSQL CR
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: my-app-db
spec:
  instances: 3
  storage:
    size: 100Gi
  storageClass: db-ssd
  backup:
    barmanObjectStore:
      destinationPath: s3://my-bucket/backups
      s3Credentials:
        accessKeyId:
          name: barman-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: barman-creds
          key: SECRET_ACCESS_KEY
EOF

The Operator creates the StatefulSet, the Services, the Secrets, and configures backups.

The production pattern

sequenceDiagram
    participant U as User
    participant O as Operator
    participant K as API server
    participant S as StatefulSet
    participant DB as Database
    Note over U,DB: Provisioning
    U->>K: create PostgreSQL CR
    K->>O: CR detected
    O->>K: create StatefulSet, Services, Secrets
    K->>S: create StatefulSet
    S->>DB: start postgres-0, postgres-1, postgres-2
    O->>DB: configure replication
    Note over U,DB: Scaling
    U->>K: update CR: instances 3 -> 5
    K->>O: CR updated
    O->>K: scale StatefulSet to 5
    O->>DB: reconfigure replication
    Note over U,DB: Backup
    O->>DB: pg_start_backup
    O->>S: create VolumeSnapshot
    O->>DB: pg_stop_backup

Quiz

Production discipline

• Operators are the production standard for stateful workloads. Choose Level 3-4 with adoption.
• The Operator is part of the cluster bootstrap. Document it; maintain it; upgrade it.
• Test the Operator’s features. Backup, restore, scaling, upgrade. Each is a runbook entry.
• Validate the Operator’s RBAC. It needs broad permissions; review them.
• Monitor the Operator. The Operator is on the critical path for stateful workloads.