Defense in depth — layered controls for the cluster

Defense in depth is the discipline of assuming that every single control will fail, and stacking independent controls so a single failure does not yield compromise. In a Kubernetes cluster, the layers are well known: the network perimeter, the API server, the kubelet, the container runtime, the kernel, and the workload itself. This lesson walks each layer, the controls that belong there, and the failure modes that make a single layer insufficient.

The principle

If you have only one control, an attacker who bypasses it owns the cluster. If you have two independent controls, the attacker must bypass both. If you have six, the attacker must bypass all six. Each layer assumes the layer above it will fail, so the layers must be independent (a control that depends on another control gives no additional protection). Independence is the metric.

flowchart TB
    L1[Layer 1: Network perimeter] --> L2[Layer 2: API server]
    L2 --> L3[Layer 3: Kubelet]
    L3 --> L4[Layer 4: Container runtime]
    L4 --> L5[Layer 5: Linux kernel]
    L5 --> L6[Layer 6: Workload]
    L6 -.compromise.-> L7[Breach]

An attacker must traverse every layer. A control failure at layer 3 still leaves layers 4, 5, and 6 standing. A control failure at layer 6 (say, a CVE in the application) still leaves the runtime, the kernel, and the kubelet to limit the blast radius.

Layer 1: the network perimeter

The cluster sits inside a network. The perimeter is the firewall, the VPC, the on-premises VLAN, and the load balancer in front of the API server.

Layer 2: the API server

Every request enters through the API server. The controls here are the most mature and the most audited.

• TLS 1.2+ with a CA bundle the operator controls; --tls-min-version=VersionTLS12.
• Authentication — X.509, OIDC, projected SA tokens. No anonymous requests. The --anonymous-auth=false flag is the default in 1.34 and must stay that way.
• Authorization — RBAC, with system:anonymous and system:unauthenticated groups bound to nothing.
• Admission — MutatingWebhookConfiguration, ValidatingWebhookConfiguration, and ValidatingAdmissionPolicy for declarative policy.
• Audit log — every request, every verb, every resource, every userinfo. The audit log is the layer 2 “intrusion detection” — if the controls above fail, the audit log records it.
• Rate limiting — --max-requests-inflight=800 and --max-mutating-requests-inflight=400. Stops a runaway client from starving the control plane.

Layer 3: the kubelet

The kubelet is the API server’s window onto each node. It is also a frequently under-hardened endpoint.

• --anonymous-auth=false (default in 1.34)
• --authorization-mode=Webhook (RBAC decisions come from the API server, no node-local bypass)
• --read-only-port=0 (the unauthenticated read port is closed)
• --protect-kernel-defaults=true (kernel parameters that the kubelet needs are pinned)
• --rotate-server-certificates=true (no long-lived client certs)

# Verify kubelet hardening on a node
ssh node-01 'cat /var/lib/kubelet/config.yaml | grep -E "anonymous-auth|authorization-mode|read-only-port|protect-kernel"'

Layer 4: the container runtime

The runtime (containerd, CRI-O) is the boundary between container images and running processes.

• Seccomp — every container gets a seccomp profile (RuntimeDefault at minimum).
• AppArmor / SELinux — every container gets a profile.
• No --privileged — privileged containers are forbidden by PSS restricted.
• No host namespace sharing — hostNetwork, hostPID, hostIPC are forbidden by PSS restricted.
• No hostPath — hostPath volumes are forbidden by PSS restricted.
• Read-only root filesystem — containers that need to write use emptyDir or CSI volumes, not the root fs.
• Image pull policy — pull by digest, not tag.

Layer 5: the Linux kernel

The kernel is the layer that no Kubernetes component controls directly; it is configured by the node image and the runtime.

• Linux capabilities — drop everything by default; add only what the workload needs.
• Cgroups — cpu, memory, pids enforced, no unbounded access.
• Namespaces — pid, mount, network, ipc, uts, user all isolated per container.
• Kernel parameters — vm.overcommit_memory=1, kernel.panic=10, net.core.somaxconn=65535, etc. — set by the kubelet and the node image.
• AppArmor / SELinux — kernel-level enforcement of the runtime profile.

The kernel is the last line. If every layer above is breached, the kernel cgroups, capabilities, and namespaces still apply. A container that escapes its mount namespace is still subject to the kernel’s capability set.

Layer 6: the workload

The workload itself — the application code.

• Pod Security Standards — namespace labels enforce restricted (or baseline if the workload is incompatible with restricted).
• ServiceAccount scoping — every workload has the minimum RBAC it needs. No cluster-admin.
• No privileged escalation — allowPrivilegeEscalation: false.
• Run as non-root — runAsNonRoot: true, runAsUser: 1000 (or the appropriate UID).
• Read-only root filesystem — readOnlyRootFilesystem: true.
• Resource limits — CPU and memory requests/limits set so the workload cannot starve neighbours.
• Image provenance — signed images, pinned by digest, from a registry that enforces admission.

Layering in practice

A single compromise rarely means a single failure. The 2021 Codecov supply chain attack breached layer 6 (the workload); it would have been stopped by layer 2 (audit log detecting the unusual outbound destination), layer 4 (network policy blocking egress to unexpected IPs), and layer 6 (signed images from a trusted registry with admission enforcement). The cluster with only one of those controls fell; clusters with all three did not.

Cross-course references

• The Linux course covers the kernel primitives — cgroups, namespaces, capabilities, seccomp, AppArmor, SELinux — that layers 4 and 5 depend on.
• The OPNsense course covers the perimeter at layer 1.
• The Observability course covers the audit log that detects failures at every other layer.

Quiz

Production discipline

A defensible cluster ships controls at all six layers and tests each one in isolation. The audit log is the source of truth: every layer’s failure is recorded there, and the SIEM rules are tuned to alert on the indicators of a control bypass. The threat model (Part LVI lesson 1) drives which controls to ship at which layer; the defense in depth principle ensures no single control is the only one standing between the attacker and the cluster. A cluster that hardens only the API server is not defended; it is delay-prone.