Production reference architecture — the canonical cluster

The production reference architecture is the canonical cluster. The diagnostic is the components, the relationships, and the systematic approach to the cluster. Every component in it earns its place by removing a failure mode, and the architecture names which one.

The reference architecture

The production reference architecture is the canonical cluster. The architecture is the canonical deployment of Kubernetes.

flowchart TB
    subgraph CP["Control Plane (3 nodes, multi-AZ)"]
        ETCD["etcd cluster (3 members)"]
        API1["kube-apiserver (3 replicas)"]
        API2["kube-controller-manager (2 replicas + leader election)"]
        API3["kube-scheduler (2 replicas + leader election)"]
        LB["Front-end load balancer (NGINX / cloud LB)"]
    end

    subgraph W["Worker Pools (multi-AZ)"]
        W1["Node pool: general (3 nodes)"]
        W2["Node pool: memory-optimized (3 nodes)"]
        W3["Node pool: compute-optimized (3 nodes)"]
    end

    subgraph N["Networking"]
        CNI["CNI (Cilium / Calico)"]
        DNS["CoreDNS (2 replicas)"]
        NLP["NetworkPolicy: default-deny + explicit allow"]
    end

    subgraph S["Storage"]
        CSI["CSI driver (e.g., EBS / Ceph)"]
        SC["StorageClass: fast-ssd, backup"]
    end

    subgraph IW["Ingress"]
        ING["Ingress / Gateway API"]
        TLS["cert-manager (TLS)"]
    end

    subgraph M["Observability"]
        PROM["Prometheus"]
        GRAF["Grafana"]
        LOG["Loki / Elasticsearch"]
        TRC["Tempo / Jaeger"]
    end

    subgraph B["Backup"]
        VEL["Velero (cluster backup)"]
        ETCD_SNAP["etcd snapshot (CronJob)"]
    end

    subgraph OPS["Ops"]
        ARG["ArgoCD (GitOps)"]
        POL["Polaris (anti-pattern)"]
    end

    subgraph WS["Workloads"]
        STAT["Stateless Deployment"]
        STFL["StatefulSet (database)"]
    end

    CLI --> LB
    LB --> API1
    API1 --> ETCD
    W1 --> STAT
    W2 --> STFL
    W3 --> STAT
    API1 --> W1
    API1 --> W2
    API1 --> W3
    W1 --> CNI
    W2 --> CNI
    W3 --> CNI
    W1 --> DNS
    W2 --> DNS
    W3 --> DNS
    ING --> STAT
    ING --> STFL
    STAT --> CSI
    STFL --> CSI
    PROM --> W1
    PROM --> W2
    PROM --> W3
    VEL --> ETCD
    VEL --> CSI
    ARG --> STAT
    ARG --> STFL

The reference architecture is the canonical cluster.

The control plane

The control plane is the cluster’s brain. The control plane is composed of:

• etcd cluster (3 members). The cluster’s state.
• kube-apiserver (3 replicas). The cluster’s HTTP API.
• kube-controller-manager (2 replicas). The cluster’s reconciliation.
• kube-scheduler (2 replicas). The cluster’s placement engine.
• Front-end load balancer. The cluster’s front door.

flowchart TB
    subgraph CP["Control Plane"]
        ETCD["etcd (3x)"]
        API["kube-apiserver (3x)"]
        CM["kube-controller-manager (2x)"]
        SCH["kube-scheduler (2x)"]
        LB["Load balancer"]
    end
    LB --> API
    API --> ETCD
    API --> CM
    API --> SCH

The control plane is the cluster’s brain.

The worker pools

The worker pools are the cluster’s compute. The worker pools are composed of:

• General node pool (3 nodes). The standard workloads.
• Memory-optimized node pool (3 nodes). The memory-intensive workloads (e.g., in-memory caches).
• Compute-optimized node pool (3 nodes). The compute-intensive workloads (e.g., batch processing).

flowchart TB
    subgraph W["Worker Pools"]
        W1["General (3 nodes)"]
        W2["Memory-optimized (3 nodes)"]
        W3["Compute-optimized (3 nodes)"]
    end
    W1 --> WORKLOAD_A["Stateless workloads"]
    W2 --> WORKLOAD_B["Memory-intensive workloads"]
    W3 --> WORKLOAD_C["Compute-intensive workloads"]

The worker pools are the cluster’s compute.

The networking

The networking is the cluster’s connectivity. The networking is composed of:

• CNI (Cilium / Calico). The cluster’s network.
• CoreDNS (2 replicas). The cluster’s DNS.
• NetworkPolicy: Default-deny + explicit allow.

flowchart TB
    subgraph N["Networking"]
        CNI["CNI (Cilium)"]
        DNS["CoreDNS (2x)"]
        NLP["NetworkPolicy"]
    end
    CNI --> DP["eBPF datapath"]
    DNS --> PODS["Pod DNS"]
    NLP --> DEFAULT["default-deny"]
    NLP --> EXPLICIT["explicit allow"]

The networking is the cluster’s connectivity.

The storage

The storage is the cluster’s data. The storage is composed of:

• CSI driver. The cluster’s storage runtime.
• StorageClass: fast-ssd, backup.

flowchart TB
    subgraph S["Storage"]
        CSI["CSI driver"]
        SC1["fast-ssd"]
        SC2["backup"]
    end
    CSI --> BACKEND["Storage backend"]
    SC1 --> CSI
    SC2 --> CSI

The storage is the cluster’s data.

The ingress

The ingress is the cluster’s external reachability. The ingress is composed of:

• Ingress / Gateway API. The cluster’s external routing.
• cert-manager (TLS). The cluster’s TLS.

flowchart TB
    subgraph IW["Ingress"]
        ING["Ingress / Gateway API"]
        TLS["cert-manager"]
    end
    ING --> TLS
    TLS --> CERT["Let's Encrypt"]

The ingress is the cluster’s external reachability.

The observability

The observability is the cluster’s insight. The observability is composed of:

• Prometheus. The metrics.
• Grafana. The dashboards.
• Loki / Elasticsearch. The logs.
• Tempo / Jaeger. The traces.

flowchart TB
    subgraph M["Observability"]
        PROM["Prometheus"]
        GRAF["Grafana"]
        LOG["Loki"]
        TRC["Tempo"]
    end
    PROM --> GRAF
    LOG --> GRAF
    TRC --> GRAF

The observability is the cluster’s insight.

The backup

The backup is the cluster’s safety net. The backup is composed of:

• Velero (cluster backup). The cluster’s restore.
• etcd snapshot (CronJob). The cluster’s state.

flowchart TB
    subgraph B["Backup"]
        VEL["Velero"]
        ETCD_SNAP["etcd snapshot"]
    end
    VEL --> S3["S3 / object storage"]
    ETCD_SNAP --> S3

The backup is the cluster’s safety net.

The ops

The ops is the cluster’s automation. The ops is composed of:

• ArgoCD (GitOps). The cluster’s deployment.
• Polaris (anti-pattern). The cluster’s anti-pattern detection.

flowchart TB
    subgraph OPS["Ops"]
        ARG["ArgoCD"]
        POL["Polaris"]
    end
    ARG --> WORKLOADS["Workloads"]
    POL --> WORKLOADS

The ops is the cluster’s automation.

The workloads

The workloads are the cluster’s purpose. The workloads are composed of:

• Stateless Deployment. The standard workloads.
• StatefulSet (database). The stateful workloads.

flowchart TB
    subgraph WS["Workloads"]
        STAT["Stateless Deployment"]
        STFL["StatefulSet"]
    end

The workloads are the cluster’s purpose.

Production discipline

The production reference architecture is the cluster’s hypothesis. The discipline is the same scale-free: every component is justified. The cluster’s discipline is the same for every workload.

• Identify the gaps. The gaps are the cluster’s missing components.
• Prioritise the fix. The high-impact gaps are the cluster’s most dangerous.

Quiz