Mission-critical Kubernetes capstone — the complete production estate

Capstone discipline. This capstone exercises every production competency taught in the course. It is not a quiz and not a walkthrough — it is the operational test of whether the student can take responsibility for a real Kubernetes estate. Expect to spend multiple sessions on it; budget 12 to 16 hours of focused lab time across the seven phases.

1. Reference architecture

The capstone estate is one HA Kubernetes cluster supporting a mixed workload portfolio: a stateless web tier, a stateful database tier, a batch processing tier, and an observability stack that itself becomes the cluster’s window into itself.

flowchart TB
    subgraph USR["Operators / Users"]
        OPS["kubectl / Helm / ArgoCD"]
        USERS["End users"]
    end

    subgraph EDGE["Edge / Ingress"]
        LB["HAProxy / cloud LB"]
        GW["Gateway API controller"]
        CM["cert-manager"]
    end

    subgraph CP["Control plane (3 nodes, multi-AZ)"]
        KAPI["kube-apiserver (3x, leader-elected)"]
        KSCM["kube-controller-manager (2x)"]
        KSCH["kube-scheduler (2x)"]
        ETCD["etcd (3x, stacked)"]
    end

    subgraph WP["Worker pools (3 nodes each, multi-AZ)"]
        WG["general pool (8C/32G)"]
        WS["system pool (4C/16G, taint dedicated)"]
        WM["memory pool (16C/128G)"]
        WGPU["gpu pool (GPU nodes, taint nvidia.com/gpu)"]
    end

    subgraph NET["Networking"]
        CNI["Cilium (eBPF)"]
        DNS["CoreDNS (2x) + NodeLocal DNSCache"]
        NP["NetworkPolicy: default-deny"]
        META["MetalLB / cloud LB"]
    end

    subgraph STR["Storage"]
        CSI["CSI driver"]
        SC["StorageClass: fast-ssd, slow-hdd, backup"]
    end

    subgraph SEC["Security"]
        PSS["Pod Security Standards: restricted"]
        RBAC["RBAC: ClusterRole, Role"]
        SA["ServiceAccounts (least privilege)"]
        ESO["External Secrets Operator"]
    end

    subgraph OBS["Observability"]
        PROM["Prometheus"]
        KSM["kube-state-metrics"]
        GRAF["Grafana"]
        LOK["Loki"]
        TMP["Tempo"]
        ALM["Alertmanager"]
    end

    subgraph BACK["Backup"]
        VEL["Velero"]
        ESNAP["etcd snapshot CronJob"]
    end

    subgraph WS["Workloads"]
        WEB["Stateless Deployment: web"]
        API["Stateless Deployment: api"]
        DB["StatefulSet: postgres"]
        CACHE["StatefulSet: redis"]
        BATCH["Job/CronJob: batch-processor"]
    end

    OPS --> LB
    USERS --> GW
    LB --> KAPI
    GW --> KAPI
    KAPI --> ETCD
    KAPI --> KSCM
    KAPI --> KSCH
    KAPI --> WG
    KAPI --> WS
    KAPI --> WM
    KAPI --> WGPU
    WG --> CNI
    WS --> CNI
    WM --> CNI
    WGPU --> CNI
    WG --> DNS
    WS --> DNS
    WM --> DNS
    WGPU --> DNS
    GW --> WEB
    GW --> API
    WEB --> SC
    API --> SC
    DB --> SC
    CACHE --> SC
    PROM --> KSM
    PROM --> WG
    PROM --> WS
    PROM --> WM
    PROM --> WGPU
    LOK --> WG
    LOK --> WS
    LOK --> WM
    LOK --> WGPU
    VEL --> ETCD
    VEL --> SC
    ESO --> DB
    ESO --> CACHE
    NP --> WG
    NP --> WS
    NP --> WM
    NP --> WGPU
    PSS --> WG
    PSS --> WS
    PSS --> WM
    PSS --> WGPU
    RBAC --> SA
    SA --> WEB
    SA --> API
    SA --> DB
    SA --> CACHE

The architecture above is the canonical estate. Every box is justified by a production requirement; every arrow is justified by a contract. Removing a box is allowed only with a written risk acceptance; adding a box without justification is a reviewable design defect.

2. HA control plane

The control plane is the cluster’s brain. Three nodes is the minimum for production HA; five is the budgeted ceiling for most regulated estates. Stacked etcd (etcd co-located with kube-apiserver on the same nodes) is the kubeadm default and is operationally simpler; external etcd (separate nodes for etcd) is the right choice when control-plane and etcd availability characteristics differ.

2.1 Topology

• Control plane nodes: cp-1, cp-2, cp-3. 4 cores, 16 GB RAM, 100 GB SSD. Spread across three failure domains (zones / racks / hosts).
• Load balancer: HAProxy or cloud LB fronting kube-apiserver on port 6443. Health check on /healthz. Sticky-by-default is wrong; kube-apiserver is stateless.
• etcd: stacked, one member per control plane node. WAL on a separate disk (--wal-dir). Data directory on a separate disk (--data-dir). Disk: NVMe SSD, 500+ IOPS per write, fsync < 10 ms.
• kube-controller-manager / kube-scheduler: two replicas each, one leader-elected. Co-located with kube-apiserver.

2.2 Configuration

# kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.34.x
controlPlaneEndpoint: "lb.acme.internal:6443"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/16"
etcd:
  local:
    extraArgs:
      quota-backend-bytes: "8589934592"
      auto-compaction-mode: "periodic"
      auto-compaction-retention: "8h"
---
localAPIEndpoint:
  advertiseAddress: 10.0.0.10
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock

2.3 Validation

# 1. All three control-plane nodes Ready and control-plane
kubectl get nodes -o wide
# NAME    STATUS   ROLES           AGE   VERSION
# cp-1    Ready    control-plane   1h    v1.34.x
# cp-2    Ready    control-plane   1h    v1.34.x
# cp-3    Ready    control-plane   1h    v1.34.x

# 2. etcd member list shows three healthy members
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  member list

# 3. API server reachable through the load balancer
kubectl get --raw=""
# ok

# 4. Leader-elected controllers report a single leader
kubectl get leases -n kube-system | grep kube-controller-manager

3. Worker pools

Multiple worker pools are the production answer to the ‘all workloads on one node” anti-pattern. Each pool is a tainted, labelled group of nodes sized for a specific class of workload.

3.1 Pool definitions

3.2 Pool creation

# Cluster API (CAPI) or kubeadm + manual labelling works.
# For the capstone, label and taint manually:

for node in sys-1 sys-2 sys-3; do
  kubectl label nodes $node workload=system
  kubectl taint nodes $node dedicated=system:NoSchedule
done

for node in mem-1 mem-2 mem-3; do
  kubectl label nodes $node workload=memory
done

for node in gpu-1 gpu-2; do
  kubectl label nodes $node workload=gpu nvidia.com/gpu.product=NVIDIA-H100
  kubectl taint nodes $node nvidia.com/gpu=present:NoSchedule
done

for node in gen-1 gen-2 gen-3 gen-4 gen-5 gen-6; do
  kubectl label nodes $node workload=general
done

3.3 Workload placement

# Deployment with toleration + nodeAffinity for the system pool
spec:
  template:
    spec:
      tolerations:
        - key: dedicated
          operator: Equal
          value: system
          effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: workload
                    operator: In
                    values: [system]
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: coredns

4. Production networking

4.1 CNI choice

Cilium (eBPF) is the production choice for new clusters: no iptables overhead, native NetworkPolicy enforcement, Hubble for observability, BGP support for bare-metal LoadBalancer services, and the ability to replace kube-proxy. Calico is the fallback if eBPF is constrained. Flannel is the development / edge choice — no NetworkPolicy, no L7 visibility.

4.2 CNI configuration

# Cilium via Helm (1.16.x for k8s 1.34)
helm install cilium cilium/cilium --version 1.16.x \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set bpf.masquerade=true \
  --set ipam.mode=kubernetes \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.metrics.enabled={``}

4.3 Cluster IP ranges

4.4 NetworkPolicy default-deny

# default-deny for the cluster
metadata:
  name: default-deny-all
  namespace: prod
  labels:
    app.kubernetes.io/component: baseline
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Every namespace inherits a default-deny via this template; per-namespace allow rules are added explicitly. Namespaces without a NetworkPolicy have no policy and traffic flows freely — this is the production trap.

5. Cluster DNS

5.1 CoreDNS

# coredns ConfigMap (Corefile)
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        cache 30
        loop
        reload
        loadbalance
        prometheus :9153
        forward . /etc/resolv.conf
        max-concurrent 1000
    }

Two CoreDNS replicas (Deployment scaled to 2). ndots:5 in /etc/resolv.conf to avoid search-path fan-out. NodeLocal DNSCache deployed as a DaemonSet for per-node caching.

5.2 NodeLocal DNSCache

  name: node-local-dns
  namespace: kube-system
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
    spec:
      priorityClassName: system-node-critical
      hostNetwork: true
      dnsPolicy: Default
      containers:
        - name: node-cache
          image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.x
          args:
            - -localip
            - 169.254.20.10
            - -conf
            - /etc/Corefile
          ports:
            - containerPort: 53
              name: dns
              protocol: UDP
            - containerPort: 53
              name: dns-tcp
              protocol: TCP

5.3 Stub domains and upstream

Stub domains (for the corporate resolver) and upstream forwarders (for external DNS) belong in the Corefile. Avoid huge ndots:5 defaults — they cause every unqualified name to fan out to seven searches before the lookup succeeds. Application-level DNS tuning (dnsConfig.options) can lower this for hot workloads.

6. Persistent storage

6.1 CSI driver

Choose the CSI driver that matches the underlying platform: aws-ebs-csi-driver, gcp-pd-csi-driver, azure-disk-csi-driver, csi-nfs for NFS, ceph-csi for Ceph, local-path for single-node. The driver is a Deployment (controller plugin) and a DaemonSet (node plugin).

6.2 StorageClasses

  name: fast-ssd
  annotations:
    storageclass.kubernetes.io/is-default-class: `}true"
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iopsPerGB: '1000"
  fsType: ext4
  encrypted: 'true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
---
  name: slow-hdd
  type: st1
  fsType: ext4

WaitForFirstConsumer delays PV creation until the first Pod that needs the volume is scheduled — required for zone-aware scheduling.

6.3 VolumeSnapshotClass

  name: csi-snap-class
driver: ebs.csi.aws.com
deletionPolicy: Delete
  tagSpecification_1: 'snapshot=true"

7. Workloads

7.1 Stateless Deployment (web)

  name: web
  namespace: prod
  labels:
    app: web
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: web
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: web
              topologyKey: kubernetes.io/hostname
      containers:
        - name: web
          image: registry.acme.internal/web:7.3.1@sha256:abcdef...
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 512Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: [ALL]
          startupProbe:
            httpGet:
              path: /healthz/startup
              port: http
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: http
            failureThreshold: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: http
            failureThreshold: 3
            periodSeconds: 30
          volumeMounts:
            - name: cache
              mountPath: /var/cache/web
            - name: config
              mountPath: /etc/web
              readOnly: true
      volumes:
        - name: cache
          emptyDir:
            sizeLimit: 1Gi
        - name: config
          configMap:
            name: web-config
            defaultMode: 0444

7.2 StatefulSet (postgres)

  name: postgres
  namespace: data
  replicas: 3
  serviceName: postgres
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      serviceAccountName: postgres
      securityContext:
        fsGroup: 999
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
      containers:
        - name: postgres
          image: postgres:16.x
          ports:
            - name: postgres
              containerPort: 5432
          resources:
            requests:
              cpu: 1000m
              memory: 4Gi
            limits:
              cpu: 4000m
              memory: 8Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            initialDelaySeconds: 10
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: config
              mountPath: /etc/postgresql
              readOnly: true
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

8. Configuration and Secrets

8.1 ConfigMap

  name: web-config
  namespace: prod
  labels:
    app: web
  web.yaml: |
    server:
      listen: ':8080"
      read_timeout: 5s
      write_timeout: 10s
    cache:
      backend: redis
      ttl: 300s
    metrics:
      enabled: true
      path: /metrics

For data that must not be hot-reloaded mid-rollout, set immutable: true on the ConfigMap — kubelet then refuses to update the volume mount in place, forcing a Pod restart.

8.2 Secret (raw, RBAC-scoped)

  name: db-credentials
  namespace: data
type: Opaque
stringData:
  username: postgres
  password: change-me-via-external-secrets
  # base64-encoded value if stringData is not used

Production deployments wire Secrets via External Secrets Operator (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) rather than committing them to Git. Encryption at rest in etcd is enabled at the API server (--encryption-provider-config).

8.3 External Secrets

  name: db-credentials
  namespace: data
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/data/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: prod/data/postgres
        property: password

9. Resource requests, limits, and QoS

9.1 Requests vs limits

• Requests are what the scheduler uses to place the Pod. Sum of requests ≤ node allocatable. Without requests, the scheduler cannot reason about capacity.
• Limits are what the kernel enforces (cgroups). CPU limit throttles; memory limit OOMKills.
• Always set requests; set limits deliberately for noisy-neighbour protection. For databases, requests == limits to land in the Guaranteed QoS class.

9.2 QoS classes

9.3 LimitRange and ResourceQuota

# LimitRange: defaults for unnamed Pods in the namespace
  name: prod-defaults
  namespace: prod
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: 2000m
        memory: 4Gi
---
# ResourceQuota: hard ceiling on the namespace
  name: prod-quota
  namespace: prod
  hard:
    requests.cpu: '32"
    requests.memory: 64Gi
    limits.cpu: "64"
    limits.memory: 128Gi
    pods: '200"
    persistentvolumeclaims: '50"

10. Probes

Three probes, three semantics. Mixing them is the canonical anti-pattern.

10.1 Startup probe

Use for slow-starting containers (JVM warmup, schema migration on first boot). Once the startup probe passes, liveness takes over.

startupProbe:
  httpGet:
    path: /healthz/startup
    port: http
  failureThreshold: 30
  periodSeconds: 5

10.2 Readiness probe

Use for application-level health: cache warmed, downstream dependency reachable, ready to serve traffic. Failed readiness removes the Pod from the EndpointSlice; traffic stops flowing without restart.

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  failureThreshold: 3
  periodSeconds: 5

10.3 Liveness probe

Use sparingly: only for deadlock detection. Liveness should not depend on downstream services (a transient DB outage should not restart the application). Liveness should not duplicate readiness (a slow request handler should not restart the Pod).

livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  failureThreshold: 3
  periodSeconds: 30

11. Affinity and topology spread

11.1 nodeSelector

nodeSelector:
  workload: memory

11.2 nodeAffinity

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: workload
              operator: In
              values: [memory]
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values: [zone-a]

11.3 podAntiAffinity

  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: web
        topologyKey: kubernetes.io/hostname

Forces one web Pod per node. Pair with topology spread for zone-level distribution.

11.4 Topology spread

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web

The zone constraint enforces even distribution across zones (HA); the hostname constraint softens single-node density.

12. PodDisruptionBudget

  name: web
  namespace: prod
  minAvailable: 4
  selector:
    matchLabels:
      app: web
---
  name: postgres
  namespace: data
  maxUnavailable: 1
  selector:
    matchLabels:
      app: postgres

minAvailable: 4 for a 6-replica web means at most 2 may be voluntarily disrupted at once. PDBs are advisory during voluntary disruption (drains, rollouts) and ignored during involuntary disruption (node failure).

13. Services, Ingress, and Gateway API

13.1 Service (ClusterIP)

  name: web
  namespace: prod
  type: ClusterIP
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: http

13.2 Ingress (legacy)

  name: web
  namespace: prod
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  ingressClassName: nginx
  tls:
    - hosts: [app.acme.com]
      secretName: app-acme-com-tls
  rules:
    - host: app.acme.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 80

13.3 Gateway API (production)

  name: cilium
  controllerName: io.cilium/gateway-controller
---
  name: prod-edge
  namespace: infra
  gatewayClassName: cilium
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: app-acme-com-tls
            kind: Secret
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-allowed: 'true"
---
  name: web
  namespace: prod
  labels:
    gateway-allowed: "true"
  parentRefs:
    - name: prod-edge
      namespace: infra
  hostnames: ["app.acme.com"]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: web
          port: 80

Gateway API gives explicit role separation: platform team owns the Gateway and GatewayClass, application teams own HTTPRoute in their namespaces. Cross-namespace routing, header rewrites, traffic splitting, and shared listeners are first-class.

14. TLS termination

TLS termination belongs at the Gateway / Ingress, not at the Pod. cert-manager automates the certificate lifecycle: issuance via ACME (Let’s Encrypt, internal CA), renewal before expiry, rotation on demand. Internal CA (Vault PKI, cfssl, step-ca) is the right choice for services that must not traverse public CAs.

  name: letsencrypt-prod
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@acme.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

Production expiry monitoring: alert at 30 days, 14 days, 7 days, 1 day. The kubelet, API server, and etcd certificates expire silently otherwise.

15. NetworkPolicies

NetworkPolicy is namespace-scoped L3/L4 policy. Default behaviour with no NetworkPolicy in a namespace is allow all. The smallest default-deny is one empty podSelector policy. NetworkPolicy depends on the CNI enforcing it (Cilium, Calico, Weave do; Flannel does not).

# Default deny for the prod namespace
  name: default-deny-all
  namespace: prod
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# Allow ingress from the gateway namespace only
  name: allow-from-gateway
  namespace: prod
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              gateway: cilium
      ports:
        - protocol: TCP
          port: 8080
---
# Allow egress to CoreDNS in kube-system only
  name: allow-dns-egress
  namespace: prod
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 5432  # database traffic

16. RBAC

Production RBAC is Role + RoleBinding for namespace scopes and ClusterRole + ClusterRoleBinding for cluster scopes. Default behaviour when no policy matches: deny. Always pair with kubectl auth can-i before granting, and audit ClusterRoleBindings quarterly.

# A read-only Role for the on-call SRE
  name: sre-readonly
  namespace: prod
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps, endpoints]
    verbs: [get, list, watch]
  - apiGroups: [apps]
    resources: [deployments, statefulsets, replicasets]
    verbs: [get, list, watch]
---
  name: sre-readonly
  namespace: prod
subjects:
  - kind: Group
    name: sre
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: sre-readonly
  apiGroup: rbac.authorization.k8s.io
---
# A narrowly-scoped ServiceAccount for the web Deployment
  name: web
  namespace: prod
automountServiceAccountToken: false

Production also wires OIDC for human users (Dex, Keycloak, Okta) so that kubectl carries real user identity into the audit log.

17. Workload security

17.1 Pod Security Standards

# Enforce `restricted` on the prod namespace
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

restricted forbids privileged containers, host namespaces, hostPath, hostPorts, most capabilities, root escalation, unsafe proc mounts, AppArmor / seccomp deviations. It is the production goal.

17.2 SecurityContext

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  runAsGroup: 10001
  fsGroup: 10001
  fsGroupChangePolicy: OnRootMismatch
  seccompProfile:
    type: RuntimeDefault
containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: [ALL]

seccompProfile: RuntimeDefault is the modern default; the container runs under the runtime’s default seccomp profile. Custom profiles are an option for tighter confinement.

17.3 Image supply chain

• Pin by digest: image: web@sha256:abc..., never latest.
• Scan every image (Trivy, Grype) in CI; fail the build on Critical CVEs.
• Sign every image (Cosign with keyless OIDC, or Notary v2).
• Admit only signed images via Kyverno / Connaisseur / sigstore-policy-controller.
• Generate SBOM (Syft, SPDX, CycloneDX) per build; archive in the registry.

18. Monitoring (Prometheus + Grafana)

The observability stack itself is a Kubernetes application; deploy it the same way as any other.

18.1 Prometheus

kube-prometheus-stack (Helm chart) ships Prometheus, Alertmanager, kube-state-metrics, node-exporter, and a starter Grafana. Tune for the cluster’s cardinality: 30,000 Pod targets at 15s scrape is the upper bound for a single Prometheus instance. Thanos or Cortex for multi-cluster or long-term storage.

# Prometheus scrape config excerpt
scrape_configs:
  - job_name: kubernetes-apiservers
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

18.2 kube-state-metrics

kube-state-metrics (KSM) exposes object-state metrics: Pod phase, Deployment status, Node Ready, PVC phase, Job status. KSM complements cAdvisor (host-level) and the kubelet (node-level).

18.3 Alerting

Production-grade alerts (representative, not exhaustive):

19. Logging (Loki)

Loki indexes log streams by labels, not by content. Ship from node-level (Promtail / Grafana Agent / Fluent Bit) or sidecar; aggregate in Loki; query in Grafana.

19.1 Loki deployment

# Loki single-binary / simple scalable deployment
  name: loki
  namespace: monitoring
  ports:
    - port: 3100
  selector:
    app: loki

19.2 Label discipline

High-cardinality labels destroy Loki. The label set is the contract; deviations are reviewable:

• Allowed: app, namespace, component, env, container, pod_template_hash, controller_revision_hash
• Forbidden: pod_ip, pod_name, request_id, trace_id, anything per-request

19.3 Retention

Loki’s retention is object-store-side (S3 lifecycle policy) plus a compactor-side retention window. A 30-day hot window and a 1-year cold window is the default production shape.

20. Tracing (Tempo)

Tempo ingests OpenTelemetry traces and stores them in object storage. Application instrumentation (OTel SDKs) emits spans; the OTel Collector ships to Tempo; Grafana correlates with metrics and logs via exemplars.

# OTel Collector config (excerpt)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]

21. Backup (etcd snapshots + Velero)

21.1 etcd snapshot

A CronJob that runs on a control-plane node and writes snapshots to off-cluster object storage:

  name: etcd-snapshot
  namespace: kube-system
  schedule: '0 */6 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector:
            node-role.kubernetes.io/control-plane: '"
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              effect: NoSchedule
          containers:
            - name: etcd-snapshot
              image: registry.k8s.io/etcd:3.5.x
              command:
                - /bin/sh
                - -c
                - |
                  set -euo pipefail
                  ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
                    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                    --cert=/etc/kubernetes/pki/etcd/peer.crt \
                    --key=/etc/kubernetes/pki/etcd/peer.key \
                    snapshot save /snapshot/etcd-$(date -u +%Y%m%d-%H%M%S).db
                  aws s3 cp /snapshot/etcd-$(date -u +%Y%m%d-%H%M%S).db \
                    s3://acme-k8s-backups/etcd/
              volumeMounts:
                - mountPath: /etc/kubernetes/pki/etcd
                  name: etcd-certs
                  readOnly: true
                - mountPath: /snapshot
                  name: snapshot
          volumes:
            - name: etcd-certs
              hostPath:
                path: /etc/kubernetes/pki/etcd
            - name: snapshot
              hostPath:
                path: /var/lib/etcd-snapshot
          restartPolicy: OnFailure

Validate every snapshot with etcdutl snapshot status. An untested snapshot is a wish.

21.2 Velero

Velero backs up Kubernetes resources and (with the CSI snapshotter or Restic / Kopia) persistent volume contents.

velero install \
  --provider aws \
  --bucket acme-k8s-backups \
  --prefix velero \
  --secret-file ./credentials-velero \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --use-restic

# Schedule daily namespace backups
velero schedule create daily-all \
  --schedule={``}
  --include-namespaces prod,data,infra \
  --ttl 720h

22. Restore procedures

22.1 etcd restore

# kubeadm runs the API server and etcd as static Pods, not as systemd
# services: there is no kube-apiserver.service and no etcd.service.
# The kubelet keeps one Pod running per manifest in
# /etc/kubernetes/manifests/, so a manifest moved out of that
# directory is a stopped component and moving it back is the restart.

# 1. On every control-plane node, park the API server and etcd manifests
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/kube-apiserver.yaml \
   /etc/kubernetes/manifests/etcd.yaml \
   /root/manifests-parked/
crictl ps | grep -E 'kube-apiserver|etcd'
# Expected: no rows, within one kubelet fileCheckFrequency (20s default)

# 2. On every control-plane node, restore the same snapshot, using that
#    node's own --name and --initial-advertise-peer-urls
etcdutl snapshot restore /backup/etcd-20260816-020000.db \
  --data-dir /var/lib/etcd-restore \
  --name cp-1 \
  --initial-cluster cp-1=https://10.0.0.10:2380,cp-2=https://10.0.0.11:2380,cp-3=https://10.0.0.12:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380

# 3. Swap the data directory rather than editing --data-dir in the
#    manifest: kubeadm mounts /var/lib/etcd from the host as a hostPath
#    volume, so the flag and the volume would both have to change.
mv /var/lib/etcd /var/lib/etcd.broken
mv /var/lib/etcd-restore /var/lib/etcd

# 4. Put etcd back first, let the members form a quorum, then the API server
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/
crictl ps | grep etcd
mv /root/manifests-parked/kube-apiserver.yaml /etc/kubernetes/manifests/

# 5. Validate
kubectl get nodes
kubectl get pods -A | grep -v Running

22.2 Velero restore

# Restore the prod namespace from yesterday`}s backup
velero restore create --from-backup daily-all-20260816-020000 \
  --include-namespaces prod \
  --wait

# Validate
kubectl get all -n prod
kubectl get pvc -n prod
kubectl wait --for=condition=Ready pod -l app=web -n prod --timeout=300s

22.3 Game days

Schedule monthly restore drills: pick a random backup, restore it onto a scratch cluster, validate the workload. An untested backup is a wish; a monthly drill is a recovery.

23. Rolling deployment strategy

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 2
    maxUnavailable: 0

maxUnavailable: 0 + maxSurge: 2 is the production default: capacity is preserved during the rollout (the PDB forbids it otherwise), two extra Pods are tolerated for the surge, traffic drains only on Ready Pods.

# Deploy a new revision
kubectl set image deployment/web web=registry.acme.internal/web:7.3.2@sha256:def...

# Watch the rollout
kubectl rollout status deployment/web --timeout=600s

# Inspect the revision history
kubectl rollout history deployment/web

# Pause and inspect (canary)
kubectl rollout pause deployment/web
kubectl get pods -l app=web -o wide

# Resume or rollback
kubectl rollout resume deployment/web
# OR
kubectl rollout undo deployment/web

24. Worker maintenance procedure

# 1. Confirm the maintenance window and CAB approval
# 2. Verify PDB does not block the drain
kubectl get pdb -A

# 3. Cordon the node
kubectl cordon worker-3

# 4. Drain the node (ignore DaemonSets, respect PDB)
kubectl drain worker-3 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=600s

# 5. Perform the maintenance (kernel upgrade, hardware swap)

# 6. Bring the node back into the cluster
systemctl restart kubelet
kubectl uncordon worker-3

# 7. Validate
kubectl get pods -o wide | grep worker-3
kubectl describe node worker-3

For zero-impact worker rollouts, surge-replace:

1. Add a new worker to the cluster.
2. Wait for it to be Ready.
3. Drain the old worker.
4. Once empty, decommission the old worker.

This keeps the cluster at full capacity throughout the rollout.

25. Kubernetes upgrade procedure

# Phase 1: plan
# Read release notes for kubeadm, kubelet, kubectl, etcd, and CNI
# Check API deprecations against manifests with pluto + kubent
pluto detect-files --target . --output wide
kubent --target .

# Phase 2: backup etcd on every control-plane node
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-pre-upgrade-$(date -u +%Y%m%d).db

# Phase 3: upgrade control plane (one node at a time)
# On cp-1:
apt-mark unhold kubeadm && apt-get update && apt-get install -y kubeadm=1.34.x-*
kubeadm upgrade plan
kubeadm upgrade apply v1.34.x
apt-mark unhold kubelet kubectl && apt-get install -y kubelet=1.34.x-* kubectl=1.34.x-*
systemctl daemon-reload && systemctl restart kubelet

# Repeat on cp-2 and cp-3

# Phase 4: upgrade CNI
helm upgrade cilium cilium/cilium --version 1.16.y --reuse-values

# Phase 5: upgrade workers (drain, upgrade, uncordon; or surge-replace)

# Phase 6: validate
kubectl get nodes -o wide
kubectl get pods -A | grep -v Running
kubectl get cs

26. Failure recovery procedures

26.1 Single control-plane node down

• The cluster remains operational (quorum = 2 of 3).
• Replace the failed node (or revive via kubelet restart).
• Validate the new node joins etcd as a learner, then promotes to a voting member.

26.2 etcd quorum loss

• Cluster is read-only or unwritable.
• Recovery: restore from the most recent etcd snapshot onto a fresh etcd cluster (3 new members), restart kube-apiserver, validate end-to-end.
• Prevention: 5-member etcd cluster, faster snapshot cadence, monthly restore drills.

26.3 Worker failure

• Pods on the failed worker are rescheduled by their controllers (Deployment, StatefulSet, DaemonSet).
• Reclaim the node; replace hardware if needed.
• Validate the new node joins and passes the kubelet registration checks.

26.4 Storage backend loss

• Pods with mounted PVCs from the lost storage backend go I/O-error.
• The CSI driver fails to re-attach; Pods enter ContainerStatusWaiting with reason RunContainerError or similar.
• Recovery: restore the storage backend (or replace nodes), force-delete the stuck Pods, validate PVCs re-attach.

26.5 cert-manager failure

• Ingress / Gateway certificates stop issuing or renewing.
• Existing certificates continue to serve until expiry.
• Recovery: restore cert-manager, validate ClusterIssuer, manually trigger renewals for certificates expiring within 7 days.

26.6 API server 503 / hang

• Check the load balancer health check.
• Check kubelet status on control-plane nodes.
• Capture kube-apiserver logs (journalctl -u kube-apiserver).
• Validate etcd is reachable.
• Restart kube-apiserver if needed (leader election handles the rest).

27. Failure injection scenarios

For each scenario, capture before / after evidence. The operational test is the discipline: define the symptom, identify the boundary, capture the evidence, form a hypothesis, test it, restore, validate, prevent.

27.1 Control-plane node failure

Setup: Stop kube-apiserver on cp-3. Cluster remains operational on the remaining two. Symptom: one fewer API endpoint, etcd member reports unreachable.

Detect: kubectl get nodes -o wide shows cp-3 as NotReady. Prometheus alert: KubeNodeNotReady.

Recover: Recreate the API server static Pod on cp-3 by moving /etc/kubernetes/manifests/kube-apiserver.yaml out of the directory and back — there is no kube-apiserver.service to restart. Validate the node returns to Ready within 5 minutes.

Prevent: Health-check the API server from the load balancer; alert on kube_apiserver_health_check_status{healthy="false"} absence.

27.2 etcd member loss (one of three)

Setup: stop etcd on cp-3 by moving /etc/kubernetes/manifests/etcd.yaml to /root/ — a kubeadm node has no etcd.service. Symptom: one etcd member unreachable, cluster still serves reads and writes (quorum = 2).

Detect: etcdctl endpoint status --cluster shows one member as unhealthy. etcdctl endpoint health reports 2 of 3.

Recover: move etcd.yaml back into /etc/kubernetes/manifests/; the kubelet recreates the static Pod within one fileCheckFrequency interval. Validate member joins and leader election converges.

Prevent: Alert on etcd_cluster_members{type=}available”} < 3`.

27.3 etcd quorum loss (two of three)

Setup: Stop etcd on cp-2 AND cp-3. Cluster is read-only.

Detect: kubectl get pods hangs. API server logs report etcd etcdserver: request timed out.

Recover: Restore etcd from the most recent snapshot onto a fresh etcd cluster; restart kube-apiserver on all control plane nodes; validate the cluster end-to-end.

Prevent: Schedule 5-member etcd, or use a regional failure-domain-aware 3-member cluster, and monthly restore drills.

27.4 API server certificate expiry

Setup: Move the system clock forward 400 days on cp-1. Symptom: kubectl get nodes returns x509: certificate has expired or is not yet valid.

Detect: Certificate expiry alert.

Recover: kubeadm certs renew on the affected node (where the kubelet cert is also affected); restart the relevant components.

Prevent: Alert at 30, 14, 7, and 1 day for every cluster certificate; monitor via kubeadm certs check-expiration.

27.5 CNI failure

Setup: kubectl delete pod -n kube-system -l k8s-app=cilium. function.

Detect: Pods stuck in ContainerCreating with event FailedCreatePodSandbox.

Recover: Cilium pods reschedule; new Pods get IPs.

Prevent: Run Cilium with replicas: 2 and priorityClassName: system-node-critical; alert on Cilium pod restarts.

27.6 CoreDNS outage

Setup: kubectl scale deployment/coredns -n kube-system --replicas=0. Symptom: name resolution fails cluster-wide. Existing connections keep working; new connections time out.

Detect: Alert CoreDNSDown.

Recover: kubectl scale deployment/coredns -n kube-system --replicas=2; validate resolution with kubectl exec + nslookup.

Prevent: Run NodeLocal DNSCache; alert on coredns_dns_requests_total flattening.

27.7 PVC pending (storage backend unavailable)

Setup: Stop the CSI controller plugin (or unplug a zone). Symptom: new PVCs sit in Pending.

Detect: Alert KubePVCPending.

Recover: Restore the CSI controller / plug the zone. Validate PVCs bind.

Prevent: Run the CSI controller with replicas; deploy the node plugin as a DaemonSet; alert on PVC Pending > 10 minutes.

27.8 Worker disk pressure

Setup: Fill the root disk of worker-3 to 95%.

Detect: kubectl describe node worker-3 shows DiskPressure=True. Prometheus alert: KubeNodeDiskPressure.

Recover: Drain the node, free disk space (or expand the volume), uncordon.

Prevent: Alert at 80% used; reserve 20% for kubelet eviction headroom; use imagefs and nodefs thresholds deliberately.

27.9 Worker memory pressure

Setup: Run a workload that allocates more memory than requested. Symptom: kubelet evicts Pods (Burstable, then Guaranteed last; never BestEffort unless it is the only thing).

Detect: kubectl describe node worker-3 shows MemoryPressure=True. Prometheus alert: KubeNodeMemoryPressure.

Recover: Drain the node, fix the workload (set requests, fix the leak), uncordon.

Prevent: Set requests == limits for critical workloads; alert on memory utilisation > 85% for 15 minutes.

27.10 Image pull failure (registry unreachable)

Setup: Blackhole DNS for the registry. Symptom: Pods in ImagePullBackOff.

Detect: kubectl describe pod shows Failed to pull image: dial tcp: lookup registry.acme.internal: no such host.

Recover: Restore DNS; kubelet retries the pull.

Prevent: Use image digest pinning; cache images on the node; pre-pull via a DaemonSet; multi-region registries.

27.11 ConfigMap mount failure (configmap deleted)

Setup: kubectl delete configmap web-config -n prod. startup) or read empty config (if hot-reloaded).

Detect: kubectl describe pod shows Error: configmap 'web-config" not found.

Recover: Restore the ConfigMap from Git or Velero.

Prevent: Treat ConfigMaps as versioned artefacts in Git; gate deletes on review; use immutable: true for stable config.

27.12 NetworkPolicy over-restriction

Setup: Apply a NetworkPolicy that drops egress to kube-system. Symptom: Pods in prod cannot reach CoreDNS; new connections fail name resolution.

Detect: kubectl exec ... nslookup kubernetes.default times out. Cilium Hubble shows dropped flows.

Recover: Correct the NetworkPolicy; validate egress with kubectl exec ... curl / nslookup.

Prevent: Test NetworkPolicies in a staging namespace first; alert on cilium_drop_count_total{reason="policy"} spikes.

27.13 RBAC over-restriction (deployer cannot apply)

Setup: Remove the RoleBinding for the deployer ServiceAccount. Symptom: kubectl apply fails with forbidden.

Detect: kubectl auth can-i create deployments --as= system:serviceaccount:prod:deployer returns no.

Recover: Restore the RoleBinding.

Prevent: Audit RoleBindings quarterly; test RBAC changes in staging; alert on kube_authorization_failures_total spikes.

27.14 Rolling rollout stuck

Setup: Deploy a broken image (web:7.3.3-bad). Symptom: rollout stops at 50%; new ReplicaSet has 0 Ready.

Detect: kubectl rollout status deployment/web reports deployment }web” successfully rolled out*but* the new Pods are inCrashLoopBackOff`.

Recover: kubectl rollout undo deployment/web.

Prevent: Canary via ArgoCD Rollouts; gate rollouts on SLO-based success; CI prevents bad images from being tagged.

27.15 etcd disk pressure

Setup: Fill the etcd data directory. Symptom: etcd logs etcdserver: backend quota exceeded; cluster stops accepting writes.

Detect: etcdctl endpoint status shows dbSize > quota-backend-bytes. Alert: etcd_backend_quota_exceeded.

Recover: etcdctl compact and etcdctl defrag. Increase quota. Move etcd to a larger disk.

Prevent: Alert at 70% of quota; run compaction and defragmentation on a schedule; size the disk to 2x the expected working set.

28. Deliverables

The capstone is complete when the student has:

• A documented reference architecture (the diagram in Section 1, with every box and arrow justified)
• A built and validated cluster (Sections 2-3)
• A working network, DNS, storage, and observability stack (Sections 4-6, 18-20)
• Stateless and stateful workloads with all production hygiene (Sections 7-12)
• Production-grade security posture (Sections 14-17)
• Backup, restore, and game-day exercises executed (Sections 21-22, 28)
• A tested rolling deployment (Section 23)
• A tested worker maintenance procedure (Section 24)
• A tested kubeadm upgrade procedure (Section 25)
• Evidence of every failure injection and recovery (Section 27, 15 scenarios)
• A post-capstone review identifying what was learned and what would harden the estate further
• A list of improvements that would harden the estate further (e.g., service mesh, admission policy library, image signing, OIDC for humans)