Kubernetes · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 786
- Parts covered
- 131
- Of all lessons
- 100%
Part I
Container and Distributed Systems Foundations
6 checks
- Why distributed systems need an orchestratorThe failure modes of running containers by hand on a fleet of Linux hosts: scheduling, healing, scaling, configuration drift, secret sprawl. Why container orchestrators exist, what they actually solve, and what they move into a new failure domain.→
- Linux namespaces and cgroups — the kernel primitives Pods inheritHow Linux namespaces (mount, PID, network, IPC, UTS, user, cgroup) and cgroups v2 (cpu, memory, pids, io) underpin Pod isolation and resource control, what Kubernetes does on top of them, and where the abstractions leak.→
- Containers, images and OCI — what the kubelet actually pullsOCI image format, image layers, manifests, digests, registries, and the runtime contract between kubelet and containerd. What survives across pulls, what does not, and how Kubernetes ties image identity to Pod identity.→
- Container runtimes — runc, containerd, CRI and the kubelet boundaryThe layering between kubelet, CRI, containerd, runc, and the kernel. What each layer does, what it does not do, and how CRI versions, shims, and snapshotter choices affect production behaviour.→
- Distributed systems fundamentals every Kubernetes operator must internaliseThe CAP theorem, partial failure, consensus, eventual consistency, leases, and clock drift. What every distributed-systems concept means in the specific context of a Kubernetes cluster, and why each Kubernetes design choice is downstream of one of them.→
- Desired state and reconciliation — the model that runs the clusterWhy Kubernetes is fundamentally a desired-state system: the operator declares what should be true, controllers observe what is true, and reconcile to close the gap. The control loop as the unit of organisation, and why every Kubernetes primitive is shaped by this model.→
Part II
Kubernetes Architecture
6 checks
- Kubernetes control plane and worker architecture at a glanceThe components of a Kubernetes cluster: control-plane (API server, etcd, scheduler, controller manager, cloud controller manager) and worker (kubelet, kube-proxy, container runtime, CNI, CSI). What each component does, how they communicate, and where the failure domains are.→
- The API server — the front door of the clusterkube-apiserver internals: request pipeline (authn, authz, admission, validation), watch semantics, storage backends, the cache, aggregation layer, and how the API server scales. Production failure modes that originate at the API server.→
- etcd — the cluster's source of truthetcd architecture, Raft consensus, quorum, persistence, snapshot/restore, performance characteristics, and the operational discipline of running etcd for Kubernetes. What every Kubernetes operator must understand about the database that backs the cluster.→
- The scheduler — assigning Pods to nodesHow kube-scheduler decides where a Pod runs: the filter-score-bind pipeline, scheduling plugins, predicates and priorities, scheduling framework, and the operational patterns that arise from scheduling behaviour (Pending Pods, preemption, topology spread).→
- The controller manager and built-in controllerskube-controller-manager: the binary that runs dozens of controllers (Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, Service, Endpoint, Node, Namespace, ServiceAccount, PersistentVolume, ResourceQuota, ...). What each controller does, how leader election works, and operational patterns that arise from controller behaviour.→
- The worker node — kubelet, runtime, CNI, CSI, and the service dataplaneWhat runs on a Kubernetes worker node: kubelet as the agent, the container runtime via CRI, the CNI plugin for networking, the CSI driver for storage, and kube-proxy (or its replacement) for the Service dataplane. How these components coordinate to run a Pod.→
Part III
Kubernetes API
6 checks
- kubectl as a Kubernetes REST clientkubectl under the hood: how it discovers the API server, authenticates, requests the API, formats output, and supports verbs. The mapping from kubectl to the REST API, kubeconfig, context, and how production debugging goes through kubectl and curl interchangeably.→
- The API server pipeline — authentication, authorisation, admission, validationHow the API server processes every request: TLS termination, authentication (X.509, bearer token, OIDC, ServiceAccount), authorisation (Node, RBAC, Webhook), mutating admission, validating admission, schema validation, persistence. What each stage rejects, what each stage modifies, and how to debug request rejections.→
- Authentication — who is making this request?How the API server authenticates requests: X.509 client certificates, bearer tokens (OIDC, ServiceAccount projected tokens), bootstrap tokens, and webhook integration. The trade-offs between long-lived credentials and short-lived projected tokens in production.→
- Authorisation — RBAC, the Node authoriser, and webhook delegationHow the API server authorises requests: Role, ClusterRole, RoleBinding, ClusterRoleBinding, the Node authoriser, and webhook delegation. Why RBAC alone is not enough, how the authoriser chain works, and the operational patterns that arise from RBAC mistakes.→
- Admission control — mutating, validating, and policy enforcementHow admission control enforces cluster policy: built-in admission controllers (PodSecurity, LimitRanger, ResourceQuota, ImagePolicyWebhook, EventRateLimit), MutatingWebhookConfiguration, ValidatingWebhookConfiguration, and ValidatingAdmissionPolicy. How to design admission for production safety and observability.→
- etcd persistence — encoding, watch, and the cluster's write pathHow the API server persists objects in etcd: encoding (protobuf vs JSON), key layout, watch semantics, resourceVersion, performance characteristics, and the operational patterns that arise from etcd being the cluster's bottleneck.→
Part IV
Desired State and Reconciliation
6 checks
- The control loop — observe, diff, actThe Kubernetes control loop as a universal pattern: every controller, every reconciler, every operator follows observe-diff-act. Why level-triggered reconciliation is the design intent, how the loop runs against the API server, and what failure modes arise from broken loops.→
- Built-in controllers — what each one doesThe controllers built into kube-controller-manager: Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, Service, EndpointSlice, Node, Namespace, ServiceAccount, PersistentVolume, ResourceQuota, garbage collector. What each reconciles, its key status fields, and the failure modes that arise from each.→
- Convergence — when does the system stabilise?Convergence in Kubernetes: what it means, what blocks it, and how to recognise partial convergence. Steady state vs transient state, the role of grace periods, and the operational patterns that arise from non-convergent systems.→
- Reconciliation pitfalls — what breaks the loop in productionProduction reconciliation pitfalls: cascading failures, partial state, drift between spec and actual, the controller-equality problem, lost updates, and the operational patterns that distinguish a designed-for-resilience cluster from a fragile one.→
- Watch, informers, and events — how controllers observe the clusterThe watch protocol, the informer cache, the workqueue, and Kubernetes Events. How controllers maintain a local view of cluster state, how events stream from the API server to clients, and the operational patterns that arise from broken watches.→
- Level-triggered vs edge-triggered reconciliationWhy Kubernetes uses level-triggered reconciliation: the design rationale, what would break with edge-triggered, and the rare cases where edge-triggered is correct. How controllers stay correct under event ordering, dropped events, and restarts.→
Part V
Kubernetes Objects and Metadata
6 checks
- Kubernetes objects — apiVersion, kind, metadataThe anatomy of a Kubernetes object: apiVersion, kind, metadata (name, namespace, uid, resourceVersion, labels, annotations), spec, status. How objects are addressed, identified, and discovered. The base structure every Kubernetes manifest shares.→
- spec and status — the discipline of declared intent vs observed realityThe spec-vs-status contract: what it means to write spec, what it means to read status, who owns each, and the operational discipline that arises from the contract. Managed fields, server-side apply, and how the API server prevents lost writes.→
- Labels and annotations — identifying and describing objectsLabels vs annotations: what each is for, how labels drive selection, how annotations hold non-identifying metadata, and the production patterns that make label/annotation hygiene a real discipline.→
- Selectors — matching objects by labelsLabel selectors in depth: equality-based, set-based, the role of selectors in services, deployments, network policies, jobs, and HPA. The immutability of Deployment selectors and the production patterns that arise from selector discipline.→
- Owner references and garbage collectionHow ownerReferences drive cascading delete. The garbage collector walks the owner graph; deleting a parent deletes the children. Production patterns for owner refs, including adoption, orphaning, and the risks of multiple owners.→
- Finalizers — blocking deletion until cleanup completesHow finalizers prevent premature deletion of objects that controllers need to clean up. The mechanics of the deletionTimestamp + finalizers dance, common finalizer patterns (kubernetes.io/pv-protection, namespace finalizers, custom Operator finalizers), and how to debug stuck deletions.→
Part VI
kubectl for Administrators
6 checks
- kubectl contexts, kubeconfig, and multi-cluster administrationThe kubeconfig file anatomy: clusters, users, contexts, current-context. How kubectl selects which API server to talk to, how to merge multiple kubeconfigs, and the production discipline around explicit context selection.→
- kubectl get, describe, and explain — read-only triageThe kubectl read commands every operator uses constantly: get (list or single object), describe (object + events), explain (schema). Output formats, label selectors, field selectors, and how to use these commands to triage a cluster without writing anything.→
- kubectl output formatting — jsonpath, custom-columns, go-templateskubectl output options for production triage: jsonpath for extracting fields, custom-columns for tabular views, go-templates for complex transformations, and the kubectl-skill of piping through jq. When each format is the right choice.→
- kubectl logs, exec, cp, and debug — runtime inspectionRuntime kubectl commands: logs (container logs, multi-container, --previous), exec (interactive shell in a container), cp (file transfer to and from a container), and kubectl debug (ephemeral debug container). The runtime-inspection toolkit for production.→
- kubectl edit, label, annotate, set — in-place modificationIn-place modification of live objects: kubectl edit (open the object in $EDITOR), kubectl label/annotate (manage metadata), kubectl set (image, resources, env). When in-place editing is appropriate and when it should be replaced by manifest-driven apply.→
- kubectl plugins (krew) and shell completionExtending kubectl with plugins installed via krew (the plugin manager): kubectx, kubens, kubectl-tree, kubectl-neat, kubectl-debug, and others. Shell completion for bash, zsh, and fish. Plugin security and the discipline around installing third-party kubectl plugins.→
Part VII
Declarative Resource Management
6 checks
- kubectl apply — last-applied-configuration and three-way mergeHow kubectl apply works under the hood: the last-applied-configuration annotation, the three-way merge between manifest, last-applied, and live state, field ownership, and how apply differs from create and replace. The basis of GitOps-driven cluster management.→
- kubectl diff and server-side diff — preview before applyHow kubectl diff shows the difference between manifest and live state. The dry-run modes (client vs server), the value of diff in code review and CI pipelines, and how diff interacts with the three-way merge.→
- kubectl patch — strategic merge, JSON patch, JSON merge patchThe three patch formats kubectl supports: strategic merge patch (default, field-aware), JSON patch (RFC 6902), and JSON merge patch (RFC 7396). When each is the right choice, how to combine patch with apply, and the production discipline around surgical changes.→
- kubectl delete — propagation, foreground, background, orphansHow kubectl delete cascades through ownership: foreground propagation (waits for dependents), background propagation (default, immediate), orphan propagation (preserves dependents). Garbage collection, owner references, finalizers, and the production discipline around deletion.→
- Server-side apply — field ownership and conflict resolutionServer-side apply (SSA): the API server tracks field ownership via managedFields, conflicts are detected explicitly, multiple actors can apply without overwriting each other. The new model of declarative management introduced in Kubernetes 1.22.→
- Drift detection and remediation — keeping live and manifest in syncDrift is the divergence between the manifest in Git (or another source of truth) and the live state in the cluster. Detection via kubectl diff and controller-side tools; remediation through reconcile loops, controlled reverts, and alerts on sustained divergence.→
Part VIII
Pods
6 checks
- Pod anatomy — apiVersion, kind, spec, the atomic unit of schedulingThe Pod object: apiVersion, kind, metadata, spec (containers, initContainers, restartPolicy, dnsPolicy, serviceAccountName, nodeSelector, affinity, tolerations, schedulerName, priorityClassName). Why a Pod is the atomic unit of scheduling and how containers share namespaces.→
- Containers, images, ports, and environment variablesInside a Pod's spec.containers: image and imagePullPolicy, ports and containerPort, env and envFrom (ConfigMap/Secret), resource requests and limits, securityContext, and the discipline around pinning image tags and managing configuration.→
- Pod IP and the shared network namespaceHow a Pod gets one IP address, how that IP is implemented (the sandbox container holds the network namespace), how containers in a Pod share the namespace, and how Pod-to-Pod and Pod-to-Service networking works.→
- Shared PID, IPC, and volumes across Pod containersBeyond the network namespace, containers in a Pod can share the PID namespace (shareProcessNamespace), the IPC namespace, and volumes (emptyDir). Use cases for sharing each, and how the kernel primitives map onto Pod fields.→
- Volumes in Pods — emptyDir, hostPath, projected, and CSIPod-level volumes: how volumes declared in spec.volumes are mounted into containers, the volume types that matter in production (emptyDir, hostPath, projected, CSI/PVC), and how the kubelet ties volume lifecycle to Pod lifecycle.→
- Pod lifecycle, restart policy, and terminationThe full Pod lifecycle: Pending, Running, Succeeded, Failed, Unknown. How restartPolicy affects restart behavior (Always, OnFailure, Never), how containers signal completion vs failure, and how the Pod's termination sequence tears down containers gracefully.→
Part IX
Pod Lifecycle
6 checks
- Pod phases — Pending, Running, Succeeded, Failed, UnknownThe five high-level Pod phases and what each means. When a Pod is Pending (waiting for scheduling or image pull), Running (bound and at least one container running), Succeeded (all containers exited 0), Failed (at least one container exited non-zero), and Unknown (kubelet cannot report state).→
- Container states — Waiting, Running, TerminatedInside each container, the kubelet tracks one of three states: Waiting (not yet started, e.g., ImagePullBackOff, CrashLoopBackOff), Running (process active), Terminated (process exited, with exit code and reason). How to read containerStatuses[*].state for triage.→
- Pod conditions — PodScheduled, Initialized, ContainersReady, ReadyThe Pod condition types (PodScheduled, PodScheduled, Initialized, ContainersReady, Ready, DisruptionTarget) and what each says about the Pod's state. How conditions differ from phase and from container states, and the diagnostic patterns that use them.→
- Startup probes — slow-starting containers and the InitialDelay trapStartup probes: how they differ from readiness and liveness probes, when they are needed (slow-starting containers, JVM warmup, migration scripts), and how they interact with the readiness/liveness probe lifecycle.→
- Readiness probes — controlling Service traffic and rolling updatesReadiness probes: how they control whether a Pod receives Service traffic, the relationship with Endpoints, and how they interact with rolling updates (the new Pod must become ready before the old is removed).→
- Liveness probes and CrashLoopBackOff — when to restart a containerLiveness probes: how they trigger container restart, the difference from readiness probes (liveness restarts, readiness removes from traffic), the failure modes (liveness loops, false positives), and the diagnostic patterns for CrashLoopBackOff.→
Part X
Pod Termination and Signals
6 checks
- Graceful termination — SIGTERM and the grace periodHow the kubelet terminates a Pod gracefully: the order of operations (deletion timestamp, preStop hook, SIGTERM, grace period countdown, SIGKILL), why applications must handle SIGTERM, and how to size terminationGracePeriodSeconds.→
- preStop hooks — what to do before SIGTERM arrivespreStop hooks: how they run before SIGTERM, the three hook types (exec, httpGet, tcpSocket), when to use them (Service deregistration, state flush, drain delay), and the production patterns that work and the ones that backfire.→
- terminationGracePeriodSeconds and SIGKILL — when the kubelet gives upWhen SIGTERM is not enough: how the kubelet decides to send SIGKILL, how to size terminationGracePeriodSeconds to avoid forced kills, and the consequences of forced termination (lost work, partial state, exit code 137).→
- Force deletion, stuck Pods, and the PodDisruptionBudget connectionWhen Pods refuse to delete: finalizers blocking, kubelet unreachable, application hung. How to diagnose and force-delete stuck Pods safely, and how PodDisruptionBudgets (PDBs) protect applications during voluntary disruptions.→
- Node shutdown and Pod termination — graceful vs forcefulHow kubelet handles system shutdown: graceful node shutdown (systemd inhibitor, gracefulNodeShutdown), Pod termination ordering on the node, the difference between graceful shutdown and force shutdown, and how to configure kubelet for graceful node shutdown.→
- Troubleshooting termination — diagnosing stuck and slow shutdownsThe systematic approach to Pod termination problems: identify whether the issue is preStop, application SIGTERM handling, grace period sizing, finalizer blocking, or kubelet unreachable. Use logs, events, containerStatuses, and the kubelet's view to diagnose.→
Part XI
Init Containers and Sidecars
6 checks
- Init containers — sequential setup before the main containerInit containers run sequentially before the main containers in a Pod. Each must succeed before the next starts. Use cases: waiting for dependencies, schema migrations, configuration generation, certificate fetching. Production patterns and anti-patterns.→
- Init container ordering and readiness gatesInit containers run sequentially; each must succeed before the next starts. How the kubelet manages init container ordering, what happens on init failure, and how readiness gates can keep a Pod from being Ready until external conditions are met.→
- Native sidecar containers (KEP-753, 1.28+)Native sidecar containers: init containers with `restartPolicy: Always` that run alongside the main container, ordered shutdown, and the migration from annotation-based sidecars. The new model introduced in Kubernetes 1.28.→
- Sidecar lifecycle, restart, and resource semanticsNative sidecar lifecycle: how the kubelet starts, monitors, and stops sidecars; how restartPolicy: Always works for sidecars; resource requests and limits for sidecars; and the interaction with main containers.→
- Migrating from annotation-based sidecars to native sidecarsHow to migrate existing workloads from annotation-based sidecars (Istio, Linkerd, OpenServiceMesh) to native sidecars. The migration steps: enable native sidecar injection, restart workloads, verify ordering, and rollback if needed.→
- Production sidecar patterns — logging, mesh, init migrationsReal-world sidecar patterns: log shipping (Fluent Bit, Vector), service mesh (Istio, Linkerd), init migrations, secret bootstrapping. The architecture decisions, sizing, and pitfalls of each pattern.→
Part XII
Resource Requests and Limits
6 checks
- CPU and memory requests — scheduling and the resource modelHow Kubernetes resource requests work: CPU (compressible, throttled under contention) and memory (incompressible, OOMKilled). How the scheduler uses requests, how the kubelet enforces limits via cgroups, and the production discipline around sizing.→
- cgroups v2 and Linux CFS quotas — the kernel primitivesHow the kubelet enforces resource limits via cgroups v2: cpu.max for CPU throttling, memory.max for memory limits, the difference between cgroups v1 and v2, and what this means for production resource management.→
- Throttling, OOMKill, and the resource pressure lifecycleThe resource pressure lifecycle: how CPU throttling and memory pressure lead to degraded performance, eventually causing OOMKill or restart loops. Detection, diagnosis, and the production discipline around preventing resource exhaustion.→
- LimitRange defaults and constraints — namespace-level resource policiesLimitRange: how to set default requests and limits for containers in a namespace, per-container min/max constraints, and how LimitRange interacts with Pod specs that declare resources explicitly.→
- Node allocatable, kubelet reservations, and capacity planningHow node allocatable is calculated: capacity minus kubelet reservations minus eviction thresholds. How to size reservations, monitor allocatable, and plan cluster capacity for production workloads.→
- Troubleshooting resource pressure — a production triage frameworkA systematic approach to resource pressure incidents: identify whether the issue is CPU throttling, memory OOMKill, or node-level eviction; gather evidence at the right layer; apply the fix without making it worse.→
Part XIII
Kubernetes QoS Classes
6 checks
- QoS classes overview — Guaranteed, Burstable, BestEffortThe three Kubernetes QoS classes (Guaranteed, Burstable, BestEffort), how they are determined from resource requests and limits, and why QoS affects eviction order, scheduling, and resource accounting.→
- Guaranteed class — matching requests and limits for predictabilityThe Guaranteed QoS class: how to design a Pod with requests == limits for every container, the trade-offs (no bursting, predictable eviction), and when Guaranteed is the right choice.→
- Burstable class — requests with bursting headroomThe Burstable QoS class: requests below limits to allow CPU and memory bursting. The most common production QoS class, the trade-offs (eviction risk vs bursting), and sizing patterns.→
- BestEffort class — no reservations, evicted firstThe BestEffort QoS class: no requests or limits. The lowest priority for eviction; suitable only for batch jobs and tolerating interruption. The risk of running BestEffort in production.→
- Eviction order and QoS — node pressure and pod survivalHow the kubelet evicts Pods under node pressure: the algorithm, the role of QoS, the eviction signals (memory.available, nodefs.available, nodefs.inodesFree, imagefs.available, pid.available), and how to tune eviction thresholds.→
- QoS-based production patterns — designing for the right classProduction patterns for QoS: matching the QoS class to the workload (databases Guaranteed, HTTP Burstable, batch BestEffort), the audit process, and how to enforce QoS policy across a cluster.→
Part XIV
Namespace Architecture
6 checks
- Namespaces — logical isolation is not security isolationKubernetes namespaces as the standard tenancy boundary: what they isolate (names, quotas, RBAC, NetworkPolicy selectors, DNS suffix), what they do not (network without NetworkPolicy, node failure, resource pressure), and how to design namespace-based multi-tenancy safely.→
- RBAC and Roles per namespace — least privilege as tenancy boundaryHow RBAC binds to namespaces: Roles, RoleBindings, ClusterRoles, ClusterRoleBindings. The pattern of least-privilege per namespace, the production discipline around granting access, and the relationship between RBAC and multi-tenancy.→
- ResourceQuota per namespace — capping total resource consumptionResourceQuota as the namespace-level cap on total CPU, memory, storage, object counts. How quotas enforce fair sharing across teams, how they interact with LimitRange defaults, and the production patterns around quota sizing.→
- Pod Security Standards per namespace — restricted, baseline, privilegedPod Security Standards (PSS) as namespace-level admission control: the three levels (privileged, baseline, restricted), how to enforce them with namespace labels, and migrating from PodSecurityPolicy.→
- NetworkPolicy per namespace — microsegmentation at the cluster levelNetworkPolicy as the in-cluster network firewall: how to enforce isolation between namespaces, the default-allow vs default-deny models, and the production patterns around policy authoring.→
- Multi-tenancy patterns — namespaces as logical clustersMulti-tenancy patterns in Kubernetes: namespace-as-tenant, namespace-as-environment, the cluster-per-tenant vs shared-cluster trade-offs, and the production patterns for tenant isolation.→
Part XV
Deployments
6 checks
- Deployment anatomy and ReplicaSet — the controller chainThe Deployment object: how it manages ReplicaSets, how ReplicaSets manage Pods, the controller chain (Deployment -> ReplicaSet -> Pod), and how the API surface differs from a bare ReplicaSet.→
- Rollout and revision history — change tracking and rollbackHow Deployments track revision history via annotations, how kubectl rollout pause and resume work, and how kubectl rollout history and undo enable change management for production.→
- Rolling updates — maxSurge, maxUnavailable, and rollout phasesHow rolling updates work in Deployments: the rollout phases (scaling up new ReplicaSet, scaling down old), the meaning of maxSurge and maxUnavailable, and how to tune them for the workload.→
- Rollout status, pause, and resume — orchestrating long changeskubectl rollout status to monitor a rollout in progress, kubectl rollout pause/resume to stage changes, and the production discipline around coordinating rollouts across multiple resources.→
- Rollback — kubectl rollout undo and revision-aware recoveryHow to roll back a Deployment to a previous revision: kubectl rollout undo, --to-revision, the recovery procedure from a failed rollout, and the production discipline around rollback readiness.→
- Deployment strategies — Recreate, Rolling, Blue-Green, CanaryThe deployment strategies in Kubernetes: Recreate (downtime), RollingUpdate (zero-downtime), and the patterns beyond Deployments (blue-green, canary, A/B testing) using Services and labels.→
Part XVI
Deployment Strategies
6 checks
- Rolling Update — maxSurge, maxUnavailable, and the math of a safe rolloutThe default Deployment strategy in Kubernetes: how RollingUpdate replaces Pods in batches, the role of maxSurge and maxUnavailable, why defaults are tuned for safety, and how readiness gates coordinate the rollout with the application.→
- Recreate — destructive but simple when downtime is acceptableThe Recreate Deployment strategy: how it kills every old Pod before creating new ones, why it is incompatible with production availability, and the limited set of workloads where Recreate is the right answer (state migrations, schema-breaking upgrades, single-replica cron-style jobs).→
- Blue/Green — atomic Service swap between two complete environmentsThe blue/green release pattern in Kubernetes: two Deployments running different versions, one Service selector switched atomically, with rollback by re-pointing the selector. Where Kubernetes natively supports it, where it does not, and how to handle stateful migrations safely.→
- Canary — small fraction first, metric-driven promotionThe canary release pattern in Kubernetes: how a separate Deployment receives a small share of traffic, how promotion is gated on observed error rates and latency, and the difference between pure Deployments-based canaries and traffic-splitting controllers.→
- Readiness, preStop, and Service traffic — gating rollout safetyHow the Deployment controller uses readiness signals to drain traffic from old Pods and admit it to new ones; how preStop hooks and terminationGracePeriodSeconds coordinate with upstream load balancers; and why a missing readiness probe turns a rolling update into an outage.→
- Progressive delivery — controllers that automate canary and blue/greenHow Argo Rollouts, Flagger, and similar controllers extend Kubernetes Deployments with metric-driven promotion, traffic splitting, and automatic rollback. What they add, what they require, and what they cannot do.→
Part XVII
StatefulSets
6 checks
- StatefulSets — when Deployments are not the right controllerWhat StatefulSets provide over Deployments: stable network identity, ordered deployment and scaling, persistent per-Pod storage, and a deletion order. The workloads they are designed for and the workloads they are not.→
- Stable identity — ordinals, headless Services, and predictable DNSHow StatefulSet Pods get stable ordinal indexes, predictable hostnames, and a headless Service that exposes them as individual DNS A records. The DNS contract for clustered software, and the failure modes when DNS or the headless Service is misconfigured.→
- Persistent storage — volumeClaimTemplates, per-Pod PVCs, and reclaimHow volumeClaimTemplates create one PVC per StatefulSet Pod, how PVCs are bound to Pods by name (not selector), what reclaimPolicy does for the underlying PV, and how to handle StatefulSet deletion safely.→
- Ordered deployment and the partition field — controlling rollout sequenceHow the StatefulSet controller creates and deletes Pods in ordinal order, how the partition field enables staged rollouts, and why ordered lifecycle is the source of most StatefulSet operational complexity.→
- StatefulSet operations — scaling, rolling out, and the deletion sequenceDay-2 operations on StatefulSets: scaling up and down safely, rolling out a new template with the partition field, what to expect when the StatefulSet is deleted, and the recovery steps for PVC and PV states.→
- StatefulSet anti-patterns — when not to reach for the database controllerThe workloads that are wrong for StatefulSets: stateless services, single-replica databases, and any controller that could be a Deployment. The hidden cost of StatefulSets in operator complexity, headless-Service risk, and the storage commitment.→
Part XVIII
DaemonSets
6 checks
- DaemonSets — one Pod per node for node-local agentsHow DaemonSets differ from Deployments and StatefulSets: a Pod runs on every node (matching the selector) and follows the node lifecycle. The node-local pattern: log collectors, CNI agents, monitoring agents, kube-proxy replacements.→
- DaemonSet update strategies — RollingUpdate vs OnDeleteHow DaemonSets update their Pods across a cluster: RollingUpdate with maxUnavailable for safe cluster-wide rollouts, and OnDelete for manually-controlled updates. The trade-offs and the cluster-wide failure modes.→
- DaemonSet scheduling — nodeSelector, taints, and tolerationsHow DaemonSet Pods are scheduled onto nodes: the controller bypasses the normal scheduler, but nodeSelector, taints, and tolerations still apply. The common configurations for control-plane vs worker placement, dedicated node pools, and exclusion of unhealthy nodes.→
- DaemonSet use cases — log collectors, CNI agents, monitoring, and moreThe canonical workloads for DaemonSets: log shippers, CNI node agents, monitoring exporters, service-mesh data planes, storage fabrics, and Kubernetes-native components. The node-local pattern and what each one requires.→
- Drain and cordon interplay — what happens to DaemonSet Pods during node maintenanceHow kubectl drain interacts with DaemonSets: by default DaemonSet Pods are ignored (--ignore-daemonsets), but PDBs and eviction rules still apply. The cluster-wide implications of node maintenance and how to handle DaemonSet disruption deliberately.→
- Host networking, hostPath, and mount propagation — node-level access patternsThe host integration features a DaemonSet uses: hostNetwork to share the node network namespace, hostPath to mount node directories, hostPID for process visibility, and mountPropagation to share mounts bidirectionally with the host.→
Part XIX
Jobs and CronJobs
6 checks
- Jobs — running Pods to completionThe Job controller in Kubernetes: how a Job creates one or more Pods that must terminate successfully, how completion and parallelism are configured, and how Jobs differ from Deployments and StatefulSets.→
- completionMode — Indexed and NonIndexed, work-queue vs partitioned batchesThe two completion modes for parallel Jobs: NonIndexed (work-queue, the original behaviour) and Indexed (each Pod gets a stable index 0..N-1 for partitioning work). When each is correct and how Indexed Jobs replaced custom work-queue patterns.→
- Restart policy, backoffLimit, and podFailurePolicy — Job resilienceHow Jobs handle Pod failures: the restart policy (OnFailure vs Never), the backoffLimit retry budget, exponential backoff between retries, and the podFailurePolicy for selective failure handling.→
- CronJobs — schedules, concurrency policy, and missed-run handlingThe CronJob controller: how it creates Jobs on a schedule, what to do when a Job is still running at the next schedule (concurrency policy), how missed runs are handled, and the failure modes of misconfigured schedules.→
- Job patterns — work queues, parallel shards, and indexed batchesThe canonical patterns for parallel Jobs: work-queue (NonIndexed, shared queue), partitioned shards (Indexed, deterministic partition), and fan-out (one Job per work item). When each is correct and how to combine them with CronJob schedules.→
- Job troubleshooting — failed runs, TTL cleanup, and debuggingHow to diagnose a Job that does not start, a Job that fails repeatedly, a Job that succeeds but produces wrong output, and how the TTL controller cleans up completed Jobs.→
Part XX
Configuration
6 checks
- ConfigMaps — key-value configuration decoupled from container imagesConfigMaps as the standard mechanism for non-sensitive configuration in Kubernetes: how to author them, how to consume them as env vars and files, and the immutability and update-behaviour patterns that make them work in production.→
- Environment variables — ConfigMap keys as Pod env varsHow ConfigMap keys are exposed as environment variables in Pod containers, the difference between envFrom and valueFrom, the runtime-update semantics (env vars do not update), and the production patterns.→
- Mounted volumes — ConfigMap as files inside the containerHow ConfigMaps are projected as files into a container filesystem via volumes, the difference from env vars in update semantics, the use of subPath for selective mounts, and the patterns for hot-reloading applications.→
- Update behaviour — env vars are frozen, files are eventually consistentThe fundamental asymmetry in ConfigMap consumption: env vars are read at container start and frozen, files are updated by the kubelet within syncPeriod. The patterns for triggering rollouts on ConfigMap change and the production failure modes.→
- Immutable ConfigMaps and Secrets — preventing runtime driftHow the immutable: true flag on ConfigMaps and Secrets prevents the API server from accepting changes, the kubelet performance benefit, the migration from mutable to immutable, and when immutability is the wrong choice.→
- Configuration anti-patterns — what not to put in a ConfigMapThe anti-patterns: sensitive data in ConfigMaps, embedding config in container images, configuration that should be in PVCs or external stores, and ConfigMaps as code-bundling. The right alternatives for each.→
Part XXI
Secrets
6 checks
- Secrets — base64 is not encryptionKubernetes Secrets: what they are, the Secret types, base64 encoding (which is not encryption), how Secrets are stored in etcd, and the RBAC and visibility patterns that determine who can read them.→
- Consuming Secrets — env vars, mounted files, image pull credentialsHow Secrets are consumed by Pods: as environment variables, as mounted files (with file mode 0400), and as imagePullSecrets for private registry authentication. The trade-offs in each pattern and the security implications.→
- etcd storage — where Secrets live and who can read themHow Kubernetes Secrets are stored in etcd: the storage key format, the access chain (API server, etcd, file system), and the visibility chain that determines who can read the Secret values.→
- Encryption at rest — EncryptionConfiguration, AES-GCM, and key rotationHow to configure encryption at rest for Kubernetes Secrets: the EncryptionConfiguration file, the supported providers (AES-CBC, AES-GCM, secretbox), the KMS-based envelope encryption, and key rotation.→
- RBAC for Secrets — least-privilege access to credentialsHow to apply RBAC to Secrets: who can read them, who can write them, the principle of least privilege, and how to audit and verify Secret access in production.→
- External secret managers — Vault, External Secrets Operator, and patternsWhy production clusters use external secret managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) and how Kubernetes tools (External Secrets Operator, CSI drivers, Vault Agent) integrate them with Secrets.→
Part XXII
Scheduling Fundamentals
6 checks
- The scheduler pipeline — observe, filter, score, bindThe Kubernetes scheduler architecture: the two-phase pipeline (filter then score), the kube-scheduler component, the designators for binding (NodeName, NodeSelector), and where custom schedulers fit.→
- Filter phase — predicates that eliminate impossible nodesThe scheduler's filter phase in detail: every filter that can eliminate a node from consideration, the reasons for each filter, and the failure modes that leave Pods Pending.→
- Score phase — how the scheduler ranks feasible nodesThe scheduler's score phase: how feasible nodes are ranked, the standard scoring plugins (LeastAllocated, BalancedResourceAllocation, NodeAffinity, TaintToleration, InterPodAffinity), and how custom scoring fits in.→
- Binding — how the scheduler reserves a node and commits the choiceThe scheduler's reserve and bind steps: how the chosen node is reserved so concurrent schedulers do not double-book, how the Pod is bound to the node via the API server, and what happens when the bind fails.→
- Pending Pods — diagnosing a Pod that will not scheduleThe systematic approach to a Pending Pod: read the events, identify the filter that eliminated all nodes, fix the constraint or the cluster, and verify the Pod schedules. The most common production failure modes.→
- Multiple schedulers and profiles — custom and workload-specific schedulingWhen and how to run multiple schedulers (custom scheduler binary, scheduler profiles), the trade-offs vs. configuring the default scheduler, and the most common patterns: GPU schedulers, batch schedulers, latency-sensitive schedulers.→
Part XXIII
nodeSelector and Node Affinity
6 checks
- nodeSelector — the simplest node-placement mechanismHow nodeSelector matches Pods to nodes via labels, the syntax (key:value), the limitations (equality only), and the migration to Node Affinity for more expressive rules.→
- requiredDuringSchedulingIgnoredDuringExecution — hard node affinityNode Affinity with requiredDuringSchedulingIgnoredDuringExecution: hard constraint that must be satisfied at scheduling time, ignored at execution time. The syntax (matchExpressions, nodeSelectorTerms), the composition rules, and the production failure modes.→
- preferredDuringSchedulingIgnoredDuringExecution — soft node affinityNode Affinity with preferredDuringSchedulingIgnoredDuringExecution: soft constraint that adds weight to the score; if no node matches, the Pod still schedules. The scoring behaviour, weight values, and when to use preferred vs required.→
- Well-known node labels — kubernetes.io and topology.kubernetes.ioThe standard node labels Kubernetes sets automatically (hostname, OS, arch), the topology.kubernetes.io labels (zone, region), the instance-type and lifecycle labels, and how to use them in nodeSelector and Node Affinity.→
- Taints and tolerations preview — the counterpart to Node AffinityHow taints and tolerations complement nodeSelector and Node Affinity: taints repel Pods that do not tolerate them, tolerations allow specific Pods through. The interaction with required/preferred affinity and the production patterns.→
- Node Affinity patterns — GPU, dedicated pools, latency zonesThe canonical Node Affinity patterns in production: GPU nodes, dedicated node pools per tenant, latency-sensitive zones, ARM workloads, and combining required with preferred.→
Part XXIV
Pod Affinity and Anti-Affinity
6 checks
- Pod affinity and anti-affinity — co-locate or separate PodsPod-level placement in Kubernetes: how Pod affinity co-locates Pods (e.g., web and cache on the same node) and Pod anti-affinity separates them (e.g., replicas across nodes). The topologyKey, labelSelector, and the required/preferred variants.→
- requiredDuringSchedulingRequiredDuringExecution — hard inter-pod affinityThe strong variant of Pod affinity: required at both scheduling and execution time. If a node's topology changes after scheduling (a label is removed, a Pod is deleted), the Pod is evicted. Use cases and risks.→
- requiredDuringSchedulingIgnoredDuringExecution — hard inter-pod anti-affinityThe standard hard variant of Pod anti-affinity: required at scheduling, ignored at execution. The most common pattern: spread Deployment replicas across nodes. The cost of strict separation and the failure modes.→
- preferredDuringSchedulingIgnoredDuringExecution — soft inter-pod affinityThe soft variant of Pod affinity and anti-affinity: adds weight to the score; if no node satisfies, the Pod still schedules. The scoring behaviour, weight values, and the production patterns.→
- HA patterns — spreading replicas for availabilityThe canonical high-availability patterns: 3 replicas across 3 zones, N replicas across nodes, and how to combine pod anti-affinity with topology spread. The trade-offs and the failure modes.→
- Performance cost — large topology domains and broad selectorsThe hidden cost of Pod affinity and anti-affinity: how large topology domains and broad selectors slow the scheduler, the worst-case scheduling latency, and the patterns that mitigate it.→
Part XXV
Topology Spread Constraints
6 checks
- Topology spread constraints — balanced distribution across domainsThe TopologySpreadConstraints field: how it distributes Pods evenly across topology domains (zones, nodes, racks), the role of maxSkew, and the difference from pod anti-affinity.→
- maxSkew — the imbalance budgetHow maxSkew bounds the difference between the most-loaded and least-loaded domain, the difference between skew 1 and skew 2 in practice, and how to choose the right value for a workload class.→
- Topology keys — kubernetes.io/hostname, topology.kubernetes.io/zone, and beyondThe topology keys in topologySpreadConstraints: hostname (per node), zone (per zone), region (per region), and custom keys. The cost and use cases for each, and how to verify the cluster has the right labels.→
- labelSelector and matchLabelKeys — dynamic groupingHow labelSelector identifies the Pods to count, and how matchLabelKeys enables dynamic grouping by label keys (e.g., per-version or per-tenant). The difference between static and dynamic constraints.→
- Topology spread vs anti-affinity — choosing the right toolWhen to use topology spread and when to use pod anti-affinity: the cost, the failure modes, the configuration trade-offs, and the patterns that combine both.→
- Topology spread HA patterns — balanced distribution in productionProduction HA patterns with topology spread: layered constraints (hostname + zone), per-version distribution, per-tenant isolation, and the trade-offs in distribution vs flexibility.→
Part XXVI
Taints and Tolerations
6 checks
- Taints and tolerations — the node-repulsion modelTaints and tolerations as Kubernetes' native mechanism for repelling Pods from nodes. How taints are set on nodes, how tolerations are declared on Pods, the three effect types, and the operational difference between "do not schedule here" and "leave if you are running".→
- Effect types — NoSchedule, PreferNoSchedule, NoExecuteThe three effect types in detail: how the scheduler and kubelet interpret each one, the difference between scheduling decisions (NoSchedule, PreferNoSchedule) and eviction (NoExecute), tolerationSeconds, and how to combine effects for staged maintenance.→
- Dedicated nodes — infra, GPU, and workload isolationThe standard pattern for reserving nodes for specific workloads using taints and tolerations: infrastructure nodes, GPU nodes, latency-sensitive nodes, and compliance-scoped nodes. The difference between adding a taint manually and managing it via a node label controller or bootstrap automation.→
- tolerationSeconds and graceful eviction windowstolerationSeconds in detail: how the value translates to eviction grace, the interaction between tolerationSeconds on the Pod and on the taint, the precedence rules, and the operational patterns (batch jobs, spot interruptions, node maintenance) that depend on the window.→
- Taints for node problems — NotReady, unreachable, pressureThe built-in taints the cluster applies when nodes fail: not-ready, unreachable, memory-pressure, disk-pressure, pid-pressure, network-unavailable. How the node controller and kubelet set these taints, the production implications, and how to tolereate them deliberately.→
- Taints and tolerations in production — anti-patterns and disciplineThe production anti-patterns of taints and tolerations: taints that name Pods instead of reasons, tolerations that swallow pressure, missing label-taint pairs, forgotten taints after node replacement, and the operational cost of leaking capacity.→
Part XXVII
Scheduling Failures
6 checks
- The Pending Pod — diagsosing scheduling failuresThe Pending Pod as the primary symptom of a scheduling failure. How to read the FailedScheduling event, the difference between filter rejection and scoring avoidance, the enum of common rejection reasons, and the diagnostic workflow that turns a Pending Pod into a fix.→
- Resource-driven failures — insufficient CPU, memory, and storageHow the scheduler rejects a Pod for resource reasons: NodeResourcesFit, the four scoring modes (LeastAllocated, MostAllocated, BalancedAllocation, VolumeBinding), the difference between requests and limits, and the operational patterns for diagnosing and fixing resource-driven scheduling failures.→
- Affinity, taint, and topology failuresScheduling failures driven by node affinity, pod affinity, taints, and topology: how each filter plugin rejects the cluster, the diagnostic patterns, and the operational fixes. The interaction between filters and the order in which they reject nodes.→
- PVC and storage-driven scheduling failuresHow the scheduler treats a Pod with a PVC: the VolumeBinding filter, the relationship between a PV, a PVC, and the Pod, the storage-class binding modes, the topology mismatch failure mode, and the diagnostic workflow for storage-driven Pending Pods.→
- Preemption and priority — when a Pod evicts anotherHow the scheduler preempts a running Pod to make room for a higher-priority one. The priority admission control, the preemption algorithm, the candidate selection, the graceful termination, and the operational patterns that prevent preemption from becoming a denial-of-service vector.→
- Scheduling gates, profiles, and the extended schedulerAdvanced scheduling features: PodSchedulingGates that hold Pods until released, scheduling profiles that configure the scheduler per workload, scheduler extenders that delegate decisions to an external service, and the operational patterns for using these deliberately.→
Part XXVIII
Node Architecture
6 checks
- The Node object — cluster view of a worker hostThe Node object: how the API represents the worker host, the fields that the cluster and operator rely on, addresses, capacity, allocatable, conditions, and the lifecycle of the Node object from registration to deletion.→
- The components on a node — kubelet, runtime, CNI, kube-proxyThe four long-running processes on a Kubernetes worker node: kubelet, the container runtime, the CNI plugin, and kube-proxy. What each one does, who starts it, how they interact, and the failure modes that arise from each.→
- Node filesystem layout — /var/lib/kubelet, /var/log, /var/lib/containerdThe node's filesystem layout: the directories the kubelet, container runtime, and CNI write to, the lifetime of the contents, the failure modes of disk pressure, and the operational patterns for sizing and monitoring the node's storage.→
- Node registration — how a node joins the clusterHow a node joins the cluster: the kubelet's bootstrap, the TLS credentials, the API server admission, the cloud provider integration, the node controller's validation, and the failure modes of a node that is registered but not Ready.→
- Node addresses and topology — labels the cluster readsThe labels and addresses the cluster uses to reason about node topology: hostname, instance-type, zone, region, OS, architecture. How the kubelet and cloud provider populate them, the affinities that depend on them, and the operational patterns for managing the label inventory.→
- Node observers and the node controller — who decides statusThe cluster-level controllers that observe the nodes: the node controller, the cloud node lifecycle controller, the kubelet, and the lease. How each one contributes to the node's status, the difference between observers and actors, and the operational patterns for diagnosing authority conflicts.→
Part XXIX
kubelet
6 checks
- The kubelet — gRPC client, Pod sync loop, status reporterThe kubelet at the architectural level: the sync loop, the gRPC clients (CRI, CNI, device plugin), the status update loop, the probe executors, and the static Pod mechanism. The kubelet is the only Kubernetes component that runs as a systemd service on the node.→
- Pod lifecycle from the kubelet's view — admit, sync, run, terminateThe kubelet's view of a Pod's lifecycle: how the kubelet picks up a new Pod, sets up the network and volumes, starts the containers, monitors them, and terminates them. The state machine the kubelet maintains for each Pod, and the failure modes at each step.→
- Probes — liveness, readiness, and startupHow the kubelet executes probes: HTTP, TCP, gRPC, and exec. The semantics of liveness, readiness, and startup probes, the configuration fields, the failure modes, and the production patterns for designing probes that catch real failures without false positives.→
- Static Pods and the mirror Pod — bootstrap without an API serverStatic Pods: how the kubelet reads manifests from a directory and runs them without the API server. The mirror Pod mechanism, the use cases (control plane bootstrap, critical add-ons), and the operational patterns for managing static Pods safely.→
- kubelet garbage collection — images, containers, volumesThe kubelet's garbage collection subsystems: image GC, container GC, and the volume cleanup. The thresholds, the metrics, the operational patterns, and the failure modes of a kubelet that is not garbage collecting.→
- kubelet credentials and rotation — TLS, tokens, and certificatesThe kubelet's credentials: the client certificate, the bootstrap token, the serving certificate, and the rotation mechanism. How the kubelet authenticates to the API server, how the API server authenticates the kubelet, and the operational patterns for keeping the credentials fresh.→
Part XXX
Container Runtime and CRI
6 checks
- The Container Runtime Interface — kubelet-to-runtime contractThe Container Runtime Interface (CRI): the gRPC API the kubelet calls to create and manage Pods. The proto definitions, the relationship between PodSandbox and Container, the runtime services, and the operational implications of the API design.→
- containerd — the production CRI implementationcontainerd as the cluster's default CRI implementation: the architecture, the namespaces, the snapshotter, the runtime modes (runc, kata), the operational patterns, and the failure modes that containerd's logs reveal.→
- Image pulls — registry, layers, and the kubelet's roleHow the kubelet pulls images via the runtime: the registry protocol, the layer model, the image-pull secrets, the ImagePullPolicy, the failure modes (ErrImagePull, ImagePullBackOff), and the operational patterns for managing the image cache.→
- RuntimeClass and runtime alternatives — runc, kata, gVisorRuntimeClass as the Kubernetes mechanism for selecting the container runtime: runc (default), kata-containers (hardware isolation), gVisor (user-space kernel). The operational trade-offs, the security posture, and the production patterns for choosing the runtime.→
- Runtime failure modes — ErrImagePull, CrashLoopBackOff, OOMKilledThe most common runtime failure modes: ErrImagePull, ImagePullBackOff, ContainerCreating, CrashLoopBackOff, OOMKilled, RunContainerError. The diagnostic pattern for each, the root causes, and the operational moves to fix the failures.→
- Runtime migration history — Docker, dockershim, and the path to containerdThe history of the container runtime in Kubernetes: Docker, dockershim, cri-dockerd, and the move to containerd. Why Docker was deprecated, what changed in 1.24, the operational implications, and the patterns for cluster migration.→
Part XXXI
Node Lifecycle
6 checks
- Node conditions — Ready, MemoryPressure, DiskPressure, PIDPressureThe Node object's Status.Conditions: the five canonical conditions, the conditions the kubelet sets, the conditions the node controller sets, the transitions, and the production patterns for alerting on the conditions.→
- Node heartbeats — the lease mechanism and the controller's grace periodThe node heartbeat mechanism: the Lease object in the kube-node-lease namespace, the kubelet's renewal, the node controller's grace period, and the failure modes that arise from a stale lease. The difference between a kubelet-reported heartbeat and a controller-graced heartbeat.→
- The node controller — cluster-level node lifecycle managementThe node controller in kube-controller-manager: how it monitors the cluster's nodes, the taints it applies, the Pod CIDR assignment, the eviction logic, and the operational discipline for tuning the controller's behaviour.→
- Pressure-driven conditions — memory, disk, PID when the kubelet detects themHow the kubelet detects node pressure: the memory, disk, and PID checks, the thresholds, the conditions used to signal the cluster, and the eviction loop that runs when pressure is detected. The difference between the kubelet's detection and the cluster's reaction.→
- Unknown and NotReady nodes — partial observabilityThe Unknown and NotReady states: how the node controller transitions nodes from Ready to NotReady and to Unknown, the difference between the two, the diagnostic patterns, and the operational discipline for recovering nodes from the bad states.→
- Adding and removing nodes — node lifecycle operationsThe operator actions for adding and removing nodes: kubeadm join, cloud provider node group scaling, node drain, node deletion, the cluster autoscaler, and the operational patterns for managing node lifecycle at scale.→
Part XXXII
Node Pressure and Eviction
6 checks
- Eviction fundamentals — soft, hard, and the kubelet's loopThe kubelet's eviction subsystem: the soft and hard thresholds, the eviction signals, the lifecycle of an eviction, the relationship between the eviction and the Pod's QoS class, and the operational patterns for tuning the kubelet's behaviour.→
- Memory pressure eviction — the kubelet's reclaim logicMemory pressure eviction in detail: the kernel's memory metrics, the kubelet's threshold checks, the Pod selection algorithm, the reclaim logic, and the operational patterns for managing memory pressure.→
- Disk pressure eviction — imagefs, logs, and the node filesystemDisk pressure eviction in detail: the kubelet's filesystem checks, the imagefs and nodefs separation, the threshold logic, the garbage collection interaction, and the operational patterns for managing disk pressure.→
- PID pressure eviction — when the kernel runs out of PIDsPID pressure eviction in detail: the kernel's PID limits, the kubelet's PID threshold, the Pod selection algorithm, the typical culprits (fork bombs, leaked processes), and the operational patterns for managing PID pressure.→
- Eviction monitoring — metrics, alerts, and the kubelet's exposesHow the cluster monitors eviction: the kubelet's metrics, the Prometheus scrape configuration, the alert rules, the dashboard patterns, and the operational discipline for tuning the eviction thresholds based on the metrics.→
- Eviction flow — from pressure to eviction to controller replacementThe full eviction flow: from the kubelet's pressure detection to the kubelet's eviction API call to the cluster's controller replacing the evicted Pod. The graceful termination, the PodDisruptionBudget interaction, and the operational patterns for minimizing the eviction impact.→
Part XXXIII
Cordon, Drain and Uncordon
6 checks
- Cordon and uncordon — the soft scheduling gatekubectl cordon and uncordon: how the unschedulable field gates new Pods, the difference between the field and a taint, the diagnostic patterns, and the operational patterns for using cordon and uncordon in node maintenance.→
- kubectl drain — the eviction-based maintenance toolkubectl drain: how the drain cordons the node and evicts the Pods, the eviction strategies, the flags (--ignore-daemonsets, --delete-emptydir-data, --force, --grace-period), and the operational patterns for using drain in production.→
- Drain with DaemonSets — what runs through the drainHow the drain handles DaemonSets: the rejection without --ignore-daemonsets, the special behaviour of DaemonSet Pods, the operational patterns for draining a node with DaemonSets, and the design patterns for DaemonSets that survive drain.→
- Drain with PodDisruptionBudgets — the interruption limitHow the drain interacts with PodDisruptionBudgets: the eviction API checks the PDB, the drain fails when the PDB rejects the eviction, the operational patterns for designing PDBs that allow the drain, and the failure modes that arise from misconfigured PDBs.→
- Drain with local data — emptyDir, hostPath, and the data-loss riskHow the drain handles Pods with local data: emptyDir volumes, hostPath volumes, and local persistent volumes. The data-loss risk, the operational patterns for protecting the data, and the design patterns for Pods that survive the drain.→
- Drain automation — Cordoning, draining, and replacing nodes at scaleDrain automation: the cluster-autoscaler, the node lifecycle controller, the cluster's maintenance automation, the operational patterns for managing drain at scale, and the failure modes that arise from misconfigured automation.→
Part XXXIV
PodDisruptionBudgets
6 checks
- PodDisruptionBudgets — the voluntary disruption limitPodDisruptionBudgets as the cluster's protection against voluntary disruption: the minAvailable and maxUnavailable fields, the PDB selector, the PDB controller, the failure modes that arise from misconfigured PDBs, and the operational patterns.→
- minAvailable — the floor on running PodsThe minAvailable PDB field in detail: the integer and percentage values, the relationship to the replica count, the eviction logic, the failure modes that arise from a too-restrictive minAvailable, and the design patterns.→
- maxUnavailable — the ceiling on unavailable PodsThe maxUnavailable PDB field in detail: the integer and percentage values, the relationship to the replica count, the eviction logic, the failure modes that arise from a too-restrictive maxUnavailable, and the design patterns.→
- PDB and eviction — what the eviction API actually checksThe PDB's interaction with the eviction API: what the eviction API checks, the difference between voluntary and involuntary eviction, the controller's logic, the failure modes that arise from misconfigurations, and the operational patterns for debugging.→
- PDB failure modes — when the protection backfiresThe PDB's failure modes: too-restrictive PDBs that block the drain, too-permissive PDBs that do not protect the workload, missing PDBs, wrong selectors, and the operational patterns for diagnosing each failure.→
- PDB best practices — production discipline for protected workloadsBest practices for PodDisruptionBudgets in production: the standard patterns for HTTP and stateful workloads, the interaction with rolling updates, the operational patterns for managing PDBs at scale, and the failure modes that arise from misconfigured PDBs.→
Part XXXV
Kubernetes Networking Fundamentals
6 checks
- The Kubernetes networking model — four rules and a contractThe Kubernetes networking model: the four rules every cluster must implement, the model that every CNI plugin must support, the relationship between the cluster's networking and the underlying network, and the operational patterns for designing cluster networks.→
- Pod-to-Pod across nodes — the cross-node routing problemPod-to-Pod communication across nodes: the routing problem, the overlay and BGP solutions, the MTU implications, the CNI plugin's responsibility, and the operational patterns for designing cluster networks.→
- Pod-to-Service — the Service abstraction and the kube-proxy dataplaneThe Service abstraction: the virtual IP, the kube-proxy's iptables/IPVS/eBPF implementations, the EndpointSlice, the DNS, and the operational patterns for designing Service-based networking.→
- External-to-Service — NodePort, LoadBalancer, and IngressExternal traffic to a Service: the NodePort, LoadBalancer, and Ingress patterns. The traffic flow, the failure modes, the operational patterns for designing external access, and the security considerations.→
- DNS and service discovery — CoreDNS, the resolv.conf, and the FQDNDNS in Kubernetes: CoreDNS as the cluster's DNS server, the resolv.conf on the Pod, the FQDN format, the search path, the nodelocal DNS cache, and the operational patterns for designing DNS in production.→
- Network troubleshooting — the diagnostic workflowNetwork troubleshooting in Kubernetes: the systematic approach from the application to the network, the diagnostic tools (kubectl exec, tcpdump, nslookup, cluster-shell), the failure modes of each layer, and the operational patterns for organising the troubleshooting.→
Part XXXVI
CNI
6 checks
- CNI specification — the contract between kubelet and the network pluginThe CNI specification: what the kubelet calls, what the plugin returns, the JSON configuration format, the version of the spec, and the operational discipline of treating the CNI as a binary contract enforced by the runtime.→
- CNI lifecycle — ADD, DEL, CHECK, and the kubelet as the runtimeThe CNI plugin lifecycle: how the kubelet invokes ADD when a Pod sandbox is created, DEL when it is removed, and CHECK when the container status is queried. The state on disk, the result caching, and the failure modes of the lifecycle.→
- IPAM — how the CNI assigns Pod IPs deterministicallyIP Address Management (IPAM) under the CNI: how the plugin allocates an IP to a Pod, the difference between host-local and centralised IPAM, the Calico and Cilium IPAM strategies, and the operational patterns for sizing and auditing the IPAM pool.→
- CNI chaining — delegating to multiple plugins in orderCNI plugin chaining: how the conflist delegates to multiple plugins, the order of ADD and DEL, the built-in plugins (portmap, bandwidth, firewall, sbr), and the operational patterns for designing a plugin chain.→
- Comparing CNI plugins — Calico, Cilium, Flannel, Weave, and the choice matrixThe major CNI plugins compared: Calico, Cilium, Flannel, Weave, and Multus. The dataplane (iptables, eBPF, VXLAN, IPIP), the policy story, the operational maturity, and the production decision matrix for choosing a CNI.→
- CNI installation and upgrade — the operational disciplineInstalling and upgrading a CNI: the kubeadm discovery path, the DaemonSet pattern, the conflist and binary layout, the upgrade cycle, and the operational discipline of treating the CNI as critical infrastructure.→
Part XXXVII
Pod Networking
6 checks
- The Pod network namespace and veth pair — what the kubelet actually createsHow the kubelet creates a Pod network namespace, attaches the veth pair, and connects the Pod to the host network. The Linux kernel primitives (network namespace, veth pair, bridge, route) that the CNI plugin uses to give each Pod its own IP.→
- Pod-to-Pod communication — same node, different nodeHow Pods on the same node and Pods on different nodes communicate. The bridge, the route, the overlay (or BGP), the cluster network, and the failure modes of Pod-to-Pod traffic at every layer.→
- Host network and Pod network — the trade-off and the consequencesThe host network mode (hostNetwork: true) in Kubernetes: what it changes, why it is used, the consequences for NetworkPolicy, the kubelet probe, and scheduling, and the operational discipline of limiting its use.→
- Pod CIDR allocation — how the cluster splits the IP space across nodesPod CIDR allocation: how the cluster splits the Pod CIDR into per-node CIDRs, the kubelet's --node-cidr-mask, the CNI plugin's IPAM, the failure modes of CIDR exhaustion, and the operational discipline of sizing the CIDR.→
- MTU and fragmentation — the silent failureMTU in Kubernetes networking: the default 1500-byte MTU, the overlay encapsulation overhead, the difference between the link MTU and the interface MTU, the consequences of MTU mismatch, and the operational discipline of avoiding fragmentation.→
- Dual-stack networking — IPv4 and IPv6 side by sideDual-stack networking in Kubernetes: how the cluster runs IPv4 and IPv6 simultaneously, the kubelet configuration, the CNI plugin support, the Service dual-stack, the failure modes, and the operational discipline of evolving from IPv4 to dual-stack.→
Part XXXVIII
Services
6 checks
- Services and ClusterIP — the stable virtual IP for a set of PodsKubernetes Services: the stable virtual IP that fronts a set of Pods, the ClusterIP allocation, the selector, the kube-proxy translation, and the operational discipline of designing Services for production.→
- Service types — choosing the right exposure for the trafficThe Kubernetes Service types: ClusterIP, NodePort, LoadBalancer, ExternalName, and Headless. The use cases, the trade-offs, the failure modes, and the operational discipline of choosing the right Service type for the traffic profile.→
- NodePort — exposing a Service on every nodeThe NodePort Service type: how it works, the port range, the traffic flow, the use cases, the failure modes (node failure, port conflict, no HA), and the operational discipline of using NodePort only for development.→
- LoadBalancer — the cloud-managed external endpointThe LoadBalancer Service type: how it allocates a cloud-provider load balancer, the traffic flow, the cost, the use cases, the failure modes, and the operational discipline of using LoadBalancer for non-HTTP traffic.→
- ExternalName — the CNAME alias to an external serviceThe ExternalName Service type: how it creates a CNAME alias to an external DNS name, the use cases, the failure modes, the difference from ClusterIP, and the operational discipline of using ExternalName for external service integration.→
- Headless services and Service topology — direct Pod addressingHeadless services (clusterIP: None) and Service topology: when direct Pod addressing is required, the DNS records for headless services, the use cases (StatefulSets, peer-to-peer), and the operational discipline of choosing headless properly.→
Part XXXIX
Service Discovery
6 checks
- Service discovery fundamentals — the four patterns and the trade-offsService discovery in Kubernetes: the four patterns (DNS, environment variables, labels, API), the trade-offs of each, the operational discipline of choosing the right pattern, and the failure modes of service discovery.→
- DNS for Services — the cluster DNS, the records, and the search pathHow DNS for Services works in Kubernetes: the DNS service (CoreDNS), the records for Services (A, AAAA, SRV), the Pod DNS policy, the search path, the nodelocal DNS cache, and the operational discipline of DNS-based discovery.→
- EndpointSlices — the scalable service-to-Pod mappingEndpointSlices in Kubernetes: the scalable replacement for Endpoints, the label-based slicing, the EndpointSlice controller, the kube-proxy integration, and the operational discipline of monitoring the EndpointSlice for empty backends.→
- Labels, selectors, and the publishNotReadyAddresses optionThe role of labels and selectors in service discovery: how the Service selector matches Pods, the publishNotReadyAddresses option, the topologyKeys, the failure modes of label mismatch, and the operational discipline of label-driven discovery.→
- SRV records and port discovery — finding the port without an A recordSRV records in Kubernetes DNS: how Services expose ports via SRV records, the use cases (Consul, Cassandra, gRPC), the format of the SRV record, the failure modes, and the operational discipline of using SRV-aware clients.→
- Service discovery anti-patterns — the most common mistakesService discovery anti-patterns in Kubernetes: hard-coded IPs, environment variables for new apps, missing port names, ignoring the nodelocal cache, and the operational discipline of avoiding them.→
Part XL
kube-proxy and Service Dataplane
6 checks
- kube-proxy fundamentals — the Service dataplanekube-proxy fundamentals: what kube-proxy does, how it watches the API for Services and EndpointSlices, the modes (iptables, IPVS, eBPF), the role of the kube-proxy in the cluster, and the operational discipline of treating the kube-proxy as critical infrastructure.→
- iptables mode — the legacy default and the rule explosionThe kube-proxy iptables mode: how it programs the iptables rules, the rule chain structure, the performance characteristics (O(n) per Service), the rule explosion problem, the failure modes, and the operational discipline of using iptables for small clusters.→
- IPVS mode — high-performance Service load balancingThe kube-proxy IPVS mode: how it programs the IP Virtual Server rules, the scheduling algorithms (round-robin, least connections, etc.), the performance characteristics, the failure modes, and the operational discipline of using IPVS for large clusters.→
- eBPF mode — Cilium kube-proxy replacement and the kernel-bypass dataplaneThe Cilium kube-proxy replacement: how eBPF dataplane replaces kube-proxy, the advantages (no iptables, no IPVS), the requirements (Cilium CNI, kernel version), the failure modes, and the operational discipline of eBPF in production.→
- kube-proxy mode comparison — the 1.34 default and the choice matrixComparing kube-proxy modes: iptables (default), IPVS, and eBPF. The performance characteristics, the operational trade-offs, the default in Kubernetes 1.34, the choice matrix, and the operational discipline of choosing the mode based on the cluster.→
- kube-proxy troubleshooting — iptables, IPVS, and eBPF diagnosticsTroubleshooting kube-proxy: the diagnostic flow for iptables (rule count, sync), IPVS (rules, scheduler), and eBPF (kernel, maps). The failure modes, the metrics, and the operational discipline of running kube-proxy diagnostics.→
Part XLI
CoreDNS
6 checks
- CoreDNS fundamentals — the cluster DNS architectureCoreDNS fundamentals: what CoreDNS is, the architecture, the deployment model, the plugins, the Corefile, the cluster DNS architecture, and the operational discipline of treating CoreDNS as critical infrastructure.→
- The Corefile — CoreDNS configuration and the plugin chainThe Corefile: the CoreDNS configuration format, the server blocks, the plugin chain, the plugin-specific configuration, the reload behaviour, and the operational discipline of treating the Corefile as critical configuration.→
- The Kubernetes plugin — how CoreDNS serves cluster recordsThe CoreDNS Kubernetes plugin: how it watches the API, the records served (A, AAAA, SRV, PTR), the format of the records, the configuration options, the failure modes, and the operational discipline of monitoring the plugin.→
- Stub domains and upstream resolvers — extending CoreDNS for custom domainsStub domains and upstream resolvers in CoreDNS: how to configure stub domains for custom DNS zones, how to forward queries to upstream resolvers, the use cases (split-horizon, external services), the failure modes, and the operational discipline.→
- CoreDNS autoscaling and tuning — performance for the clusterCoreDNS autoscaling and tuning: the autoscaler for the CoreDNS Deployment, the metrics that drive the autoscaling, the tuning of the cache, the upstream concurrency, the failure modes, and the operational discipline of running CoreDNS at scale.→
- CoreDNS troubleshooting — diagnosing the cluster DNSCoreDNS troubleshooting: the diagnostic flow for the CoreDNS Pods, the Corefile, the upstreams, the cache, the failure modes, and the operational discipline of running CoreDNS diagnostics.→
Part XLII
Ingress
6 checks
- Ingress fundamentals — the cluster HTTP gatewayIngress fundamentals: what Ingress is, the architecture, the Ingress controller, the IngressClass, the HTTP routing, the TLS termination, the production patterns, and the operational discipline of treating Ingress as critical infrastructure.→
- Ingress controllers — choosing the right one for the workloadIngress controllers compared: ingress-nginx, Traefik, HAProxy, Contour, and cloud-managed controllers. The trade-offs, the performance, the operational maturity, the use cases, and the operational discipline of choosing the right controller.→
- IngressClass and defaults — managing multiple controllersThe IngressClass and the cluster-wide default: how to manage multiple Ingress controllers, the default IngressClass, the cluster operator's decisions, the failure modes, and the operational discipline of using IngressClass properly.→
- Ingress TLS termination — the cluster HTTPS gatewayIngress TLS termination: how the Ingress controller terminates TLS, the TLS secret, the cert-manager integration, the TLS options, the failure modes, and the operational discipline of using TLS for production.→
- Ingress path and host routing — the cluster URL spaceIngress path and host routing: how the Ingress routes traffic based on host and path, the path types (Prefix, Exact, ImplementationSpecific), the host-based routing, the rewrite rules, the failure modes, and the operational discipline of routing in production.→
- Ingress troubleshooting — diagnosing the cluster HTTP gatewayIngress troubleshooting: the diagnostic flow for the Ingress controller, the Ingress resource, the backend Service, the TLS configuration, the failure modes, and the operational discipline of running Ingress diagnostics.→
Part XLIII
Gateway API
6 checks
- Gateway API introduction — the next-generation IngressGateway API introduction: what Gateway API is, how it compares to Ingress, the ownership separation (infrastructure provider, cluster operator, application developer), the role-based design, the standardisation, and the operational discipline of adopting Gateway API.→
- GatewayClass — the infrastructure provider resourceGatewayClass in the Gateway API: the role of the GatewayClass, the controller reference, the parameters reference, the conformance level, the failure modes, and the operational discipline of managing the GatewayClass.→
- Gateway — the cluster operator resourceGateway in the Gateway API: the role of the Gateway, the listeners, the TLS configuration, the allowedRoutes, the address assignment, the failure modes, and the operational discipline of managing the Gateway.→
- HTTPRoute — the application developer resourceHTTPRoute in the Gateway API: the role of the HTTPRoute, the parentRefs, the hostnames, the rules, the matches, the backendRefs, the filters, the failure modes, and the operational discipline of managing the HTTPRoute.→
- Other route types — TCPRoute, UDPRoute, TLSRoute, GRPCRouteOther Gateway API route types: TCPRoute, UDPRoute, TLSRoute, GRPCRoute. The role of each route type, the configuration, the use cases (TCP services, UDP services, TLS passthrough, gRPC), the failure modes, and the operational discipline of using the right route type.→
- Gateway API adoption and migration — from Ingress to Gateway APIGateway API adoption and migration: the migration path from Ingress to Gateway API, the coexistence patterns, the migration order, the failure modes, and the operational discipline of adopting Gateway API in production.→
Part XLIV
NetworkPolicy
6 checks
- NetworkPolicy fundamentals — the cluster firewall and the CNI dependencyNetworkPolicy fundamentals: what NetworkPolicy is, the scope of enforcement, the CNI dependency (Calico, Cilium, Weave enforce; Flannel does NOT), the default-deny pattern, the failure modes, and the operational discipline of treating NetworkPolicy as critical security configuration.→
- NetworkPolicy ingress rules — controlling inbound trafficNetworkPolicy ingress rules: how to allow or deny inbound traffic, the podSelector, the namespaceSelector, the ipBlock, the port rules, the failure modes, and the operational discipline of using ingress rules for production.→
- NetworkPolicy egress rules — controlling outbound trafficNetworkPolicy egress rules: how to allow or deny outbound traffic, the DNS egress rule, the kube-apiserver egress rule, the ipBlock, the failure modes, and the operational discipline of using egress rules for production.→
- NetworkPolicy selectors — pods, namespaces, and IP blocksNetworkPolicy selectors: the podSelector, namespaceSelector, and ipBlock, the combination rules (AND within an entry, OR across entries), the failure modes, and the operational discipline of using selectors for production.→
- Default-deny policies — the zero-trust patternDefault-deny policies in Kubernetes: the pattern of denying all traffic by default and allowing only the explicit allow rules, the implementation, the DNS and API allow rules, the failure modes, and the operational discipline of using default-deny in production.→
- CNI-specific NetworkPolicy extensions — Calico and CiliumCNI-specific NetworkPolicy extensions: Calico GlobalNetworkPolicy, Cilium CiliumNetworkPolicy, the use cases (cluster-wide policies, FQDN-based policies, L7 policies), the failure modes, and the operational discipline of using CNI-specific extensions.→
Part XLV
Kubernetes Networking Troubleshooting
6 checks
- Systematic network troubleshooting — the methodologySystematic network troubleshooting in Kubernetes: the methodology from the application to the network, the seven layers (application, Pod, CNI, Service, EndpointSlice, DNS, NetworkPolicy, kube-proxy), the diagnostic tools, the failure modes, and the operational discipline of running network troubleshooting.→
- Pod-to-Pod troubleshooting — diagnosing the Pod networkPod-to-Pod troubleshooting in Kubernetes: the diagnostic flow for Pod-to-Pod traffic, the CNI plugin verification, the veth pair inspection, the route verification, the packet capture, the failure modes, and the operational discipline.→
- Service troubleshooting — diagnosing the Service dataplaneService troubleshooting in Kubernetes: the diagnostic flow for the Service, the kube-proxy, the dataplane (iptables, IPVS, eBPF), the failure modes, and the operational discipline of running Service diagnostics.→
- DNS troubleshooting — diagnosing the cluster DNSDNS troubleshooting in Kubernetes: the diagnostic flow for the cluster DNS, the CoreDNS Pods, the upstream, the cache, the failure modes, and the operational discipline of running DNS diagnostics.→
- NetworkPolicy troubleshooting — diagnosing the cluster firewallNetworkPolicy troubleshooting in Kubernetes: the diagnostic flow for the NetworkPolicy, the CNI plugin verification, the policy rules, the failure modes, and the operational discipline of running NetworkPolicy diagnostics.→
- Packet capture and network performance — diagnosing the networkPacket capture and network performance in Kubernetes: how to capture packets from a Pod, how to use tcpdump, how to analyze the capture, the network performance metrics, the failure modes, and the operational discipline of running packet capture for production.→
Part XLVI
Packet Capture in Kubernetes
6 checks
- Packet capture in Kubernetes — what you actually need and whyWhy packet capture matters in a Kubernetes cluster, what is on the wire between Pod, node, and control plane, and the difference between host capture, node-side capture, sidecar capture, and CNI-aware capture.→
- tcpdump in Kubernetes — the flags that matterThe tcpdump flags that matter when capturing Kubernetes traffic: BPF expressions for Pod CIDRs, Service IPs, overlay decoding, snap length, ring buffers, file rotation, and reading pcap files without contaminating the host.→
- Capturing inside a container — Pod-level network namespacesHow to capture traffic inside a Kubernetes Pod using kubectl debug, ephemeral containers, ephemeral debug Pods, and netns sharing with the host. What is visible at each layer and what is not.→
- Service mesh capture — Istio, Linkerd, Cilium, and the mTLS rewritesHow service mesh sidecars (Istio, Linkerd, Cilium service mesh) rewrite traffic, what packet capture sees at each layer, why mTLS makes capture harder, and the production patterns for capturing mesh traffic.→
- CNI-aware capture — Calico, Cilium, and the tools that know the clusterCNI-aware capture tools that understand Kubernetes objects: Calico's calicoctl and Felix logs, Cilium's Hubble, Weave's scope, and the kubectl plugin ecosystem. How they differ from raw tcpdump and when to use each.→
- Capture performance, retention, and the cost of seeing every byteThe performance cost of packet capture on a busy node, the production patterns for bounding capture cost, the legal and operational requirements for retention, and the trade-offs between wire capture and flow logs at scale.→
Part XLVII
MTU Problems
6 checks
- MTU fundamentals — what MTU is, what it does, and why it breaks in KubernetesThe MTU concept from the wire: maximum transmission unit, fragmentation, path MTU discovery, the difference between L2 MTU and L3 MTU, and why overlay encapsulation makes the effective MTU smaller than the underlay.→
- Overlay MTU — VXLAN, IPIP, Geneve, and the encapsulation overheadHow the choice of overlay (VXLAN, IPIP, Geneve, none) affects the Pod MTU. The exact overhead of each overlay, the trade-offs between routed and overlay networks, and the production implications of each choice.→
- MTU and the CNI — how Calico, Cilium, Flannel set the Pod MTUHow each major CNI sets the Pod MTU: Calico Installation custom resource, Cilium ConfigMap, Flannel ConfigMap, Weave Pod arguments. The flags and the defaults, and how to detect and fix misconfigurations.→
- MTU troubleshooting — finding the bottleneck, validating the fixHow to troubleshoot Kubernetes MTU problems end-to-end: symptom signature, diagnostic ladder, the tracepath / ping -M do technique, the kernel-level validation, and the production pattern for declaring an MTU incident resolved.→
- Cloud-provider MTU — AWS, GCP, Azure, and the underlay constraintsHow each major cloud provider constrains the MTU on its underlay: AWS VPC, GCP VPC, Azure VNet, inter-region and inter-AZ traffic, and the implications for Kubernetes Pod MTU.→
- MTU operations — incident response, runbook, and validationThe operational discipline for MTU in Kubernetes: incident response, the runbook entry, cluster bootstrap validation, monitoring the MTU configuration, and the production pattern for declaring an MTU incident resolved.→
Part XLVIII
Storage Fundamentals
6 checks
- Storage fundamentals — what storage is in Kubernetes and what it is notWhat storage means in a Kubernetes cluster: the kubelet's volume subsystem, the difference between ephemeral and persistent storage, what Kubernetes abstracts and what it does not, and the operational model for storage in production.→
- Block vs file vs object storage — and what Kubernetes usesThe three categories of storage: block, file, and object. How Kubernetes uses each, the trade-offs, and the production patterns for choosing between them.→
- The kubelet volume manager — how volumes become Pod mountsThe kubelet volume manager: the reconciliation loop that watches for Pods and PVCs, calls the CSI node plugin, manages the volume lifecycle on the node, and reports status to the API server.→
- The I/O path — from application syscall to storage backendThe Kubernetes I/O path: how an application's write travels through the container filesystem, the kernel VFS, the block layer, the kubelet's bind-mount, the CSI driver, and the storage backend. The latency budget at each layer.→
- Storage concepts — access modes, capacity, and the API surfaceThe Kubernetes storage API surface: access modes (RWO, ROX, RWX, RWOP), capacity requests, volume modes (filesystem, block), storage classes, and the binding semantics between PV and PVC.→
- Storage architecture — designing storage for production clustersThe production storage architecture: per-workload StorageClasses, reclaim policies that match the backup strategy, multi-AZ awareness, snapshot integration, and the operational discipline that prevents data loss.→
Part XLIX
Volumes
6 checks
- Ephemeral vs persistent volumes — the fundamental distinctionThe difference between ephemeral and persistent volumes in Kubernetes: lifecycle, use cases, what survives Pod restart, what survives node failure, and the production pattern for choosing between them.→
- emptyDir — the ephemeral directory, its variants, and its trapsThe emptyDir volume type: how it works, the medium options (disk, memory), the sizeLimit, when to use it, and the production pitfalls including node pressure eviction.→
- hostPath — mounting the node filesystem into a PodThe hostPath volume type: how it mounts a node filesystem path into a Pod, the security implications, the legitimate use cases, and the production anti-patterns.→
- Persistent volumes introduction — PVs, PVCs, and the binding contractAn introduction to persistent volumes: what a PersistentVolume is, what a PersistentVolumeClaim is, how they bind, and the lifecycle from request to mount.→
- Volume mounts — how Pods consume volumes, subPaths, and projectionsHow Pods consume volumes: the volumeMount declaration, subPath for selective mounting, projected volumes (ConfigMap, Secret, downwardAPI, serviceAccountToken), and the production patterns.→
- Volume lifecycle — from PVC creation to PV deletion and reclamationThe complete volume lifecycle: PVC creation, binding, Pod mount, Pod unmount, PVC deletion, PV reclamation, and the reclaim policies that determine what happens next.→
Part L
PersistentVolumes and Claims
6 checks
- PV-PVC binding — how Kubernetes matches storage requests to volumesHow the binding controller matches PVCs to PVs: by capacity, access mode, StorageClass, selector, and volumeName. The matching algorithm and its edge cases.→
- PV lifecycle — the states and transitions of a PersistentVolumeThe PersistentVolume lifecycle in detail: the four states (Available, Bound, Released, Failed), the transitions between them, and what each state means operationally.→
- PVC lifecycle — the states and transitions of a PersistentVolumeClaimThe PersistentVolumeClaim lifecycle: the two states (Pending, Bound), the transitions between them, and what each state means for the workload and the operator.→
- Access modes in depth — RWO, ROX, RWX, RWOP, and what each enablesThe four PVC access modes (ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod) in depth: what each enables, which backends support them, and the production patterns.→
- Reclaim policies in depth — Retain, Delete, Recycle, and the backup strategyThe three reclaim policies (Retain, Delete, Recycle) in depth: what each does to the PV and the underlying storage, the production implications, and the alignment with the backup strategy.→
- PV-PVC anti-patterns — common storage mistakes and how to avoid themThe common PV-PVC anti-patterns in production: emptyDir for stateful data, hostPath for databases, Delete reclaim on critical data, mismatched access modes, and the missing backup strategy.→
Part LI
StorageClasses
6 checks
- StorageClass basics — provisioning templates, parameters, and binding modesThe StorageClass as a provisioning template: how it defines the provisioner, the parameters, the binding mode, the reclaim policy, and the volume expansion policy.→
- The default StorageClass — implicit bindings and the production riskThe default StorageClass: how PVCs without an explicit StorageClass bind to it, the production risks of relying on defaults, and the discipline for explicit StorageClass references.→
- Provisioners and CSI drivers — what calls the storage backendThe provisioner in the StorageClass: how it calls the storage backend, the difference between controller and node plugins, and how to verify a provisioner is healthy.→
- Volume binding modes — Immediate vs WaitForFirstConsumer in depthThe two volume binding modes in depth: Immediate (bind at submission) and WaitForFirstConsumer (bind at Pod schedule). When each is appropriate and how topology-aware provisioning works.→
- Volume expansion — growing PVCs online and the capacity disciplineVolume expansion in Kubernetes: how the PVC's capacity can be grown, the role of the StorageClass's allowVolumeExpansion, online vs offline expansion, and the production discipline for capacity planning.→
- StorageClass anti-patterns — misconfigurations that cause incidentsThe common StorageClass anti-patterns: relying on the default, wrong binding modes for multi-AZ, missing reclaim policy for stateful data, missing volume expansion, missing topology constraints.→
Part LII
CSI
6 checks
- CSI overview — the Container Storage Interface standard and its lifecycleThe CSI standard: the gRPC-based interface, the controller and node plugins, the lifecycle operations (Create, Delete, Attach, Mount, Format, Snapshot), and how CSI replaced the in-tree storage plugins.→
- The CSI controller plugin — Create, Delete, Attach, Snapshot, ExpandThe CSI controller plugin in depth: the gRPC operations it implements (CreateVolume, DeleteVolume, ControllerPublishVolume, CreateSnapshot, ControllerExpandVolume), the deployment topology, and the production patterns.→
- The CSI node plugin — Stage, Publish, Format, MountThe CSI node plugin in depth: the gRPC operations (NodeStageVolume, NodePublishVolume, NodeUnstageVolume, NodeUnpublishVolume, NodeExpandVolume), the DaemonSet deployment, the staging path, and the bind-mount into the Pod.→
- The CSI gRPC protocol — the wire format, errors, and idempotencyThe CSI gRPC protocol: the Identity, Controller, and Node services, the request/response messages, the gRPC status codes for errors, and the idempotency requirement that makes Kubernetes' reconciliation safe.→
- Attach vs mount vs format — the three CSI operations and their separationThe three CSI operations that take a volume from \"exists on the backend\" to \"filesystem mounted in a Pod\": attach (controller-level), mount (node-level), and format (node-level). Why they are separate and how the ordering matters.→
- Deploying and operating CSI drivers in productionDeploying CSI drivers in production: the controller and node plugin Deployments, the CSIDriver registration, RBAC, secrets, topology constraints, and the operational discipline for upgrades.→
Part LIII
Storage Failure Modes
6 checks
- PVC Pending — the diagnostic ladder for a stuck volume claimWhy a PVC remains Pending: the diagnostic ladder from events to provisioner logs to backend health. The four common causes (no matching StorageClass, provisioner errors, capacity limits, topology constraints) and the production fixes.→
- Attach failures — multipath, device limits, and the kernel-level diagnosticsVolume attach failures in Kubernetes: the controller-level operation (ControllerPublishVolume), the common causes (multipath, device limits, instance type, AZ mismatch), and the diagnostic patterns.→
- Mount failures — wrong fsType, missing secrets, and the kubelet-level diagnosticVolume mount failures in Kubernetes: the kubelet-level operation (NodePublishVolume), the common causes (wrong fsType, missing secret, format error, stale mount, permission), and the diagnostic patterns.→
- Topology affinity — the binding constraints between zones, regions, and nodesVolume topology in Kubernetes: zone affinity, region affinity, node-level topology, the allowedTopologies field in the StorageClass, and the production patterns for multi-AZ clusters.→
- Volume expansion failures — when growing a PVC does not workVolume expansion failures in Kubernetes: the StorageClass setting, the CSI driver support, online vs offline resize, and the diagnostic patterns for failed expansion.→
- Storage incident runbook — the complete diagnostic and recoveryA complete storage incident runbook: the diagnostic ladder from PVC events to backend metrics, the recovery procedures for each failure mode, the prevention checklist, and the production discipline.→
Part LIV
Stateful Workloads
6 checks
- Application-consistency vs crash-consistency — what backups actually meanThe difference between application-consistent and crash-consistent backups: what each guarantees, why it matters for databases, and the techniques for achieving each (quiesce, freeze, fsync).→
- StatefulSet alone is not enough — why stateful workloads need an OperatorWhy StatefulSet is necessary but not sufficient for production stateful workloads: what StatefulSet provides (stable identity, ordered deployment, per-Pod PVCs) and what it does not (backup, restore, scaling, upgrades, monitoring).→
- The Operator pattern — custom resources, controllers, and the reconciliation loopThe Operator pattern in depth: custom resources (CRDs) that model the application, controllers that reconcile the desired state with the actual state, and the standard production Operators for common databases.→
- Quiesce, freeze, and application hooks — the techniques for consistent backupsThe techniques for achieving application-consistent backups: quiesce (application-level pause), freeze (filesystem-level pause), fsync (flush dirty pages), and the application-specific hooks for PostgreSQL, MySQL, and others.→
- Backup and restore patterns for stateful workloadsThe backup and restore patterns for stateful Kubernetes workloads: snapshot-based, file-based, logical; restore procedures; backup retention; testing backups; and the production discipline.→
- Stateful workload anti-patterns — the storage mistakes that cause data lossThe common stateful workload anti-patterns in Kubernetes: emptyDir for databases, hostPath for state, missing backup strategy, scaling without operator, and the recovery failures they cause.→
Part LV
Storage Snapshots
6 checks
- Snapshots are not backups — what CSI snapshots actually give youThe fundamental distinction between a snapshot and a backup: what each is, what each protects against, why a CSI snapshot is not a backup, and the production discipline for combining them.→
- The VolumeSnapshot CRD — schema, lifecycle, and the restore procedureThe VolumeSnapshot CRD in depth: the schema (VolumeSnapshot, VolumeSnapshotContent, VolumeSnapshotClass), the lifecycle states, the restore procedure via dataSource, and the production patterns.→
- Snapshot lifecycle — creation, status, deletion, and the retention policyThe VolumeSnapshot lifecycle in depth: creation, status transitions, deletion policy, retention, and the operational discipline for managing snapshots at scale.→
- Restore from snapshot — the dataSource procedure and the validation stepsRestoring a PVC from a VolumeSnapshot: the dataSource procedure, the recovery Pod, the validation steps (read-only mount first, then read-write), and the production discipline for safe restore.→
- Application-consistent snapshots — quiesce, freeze, and the application-level hooksAchieving application-consistent CSI snapshots: the techniques (quiesce, freeze, fsync), the application-level hooks (pg_start_backup, FLUSH TABLES WITH READ LOCK), and the production pattern.→
- Snapshot operations — schedulers, Velero, and the production disciplineOperating CSI snapshots at scale: scheduling via Velero, k8up, or custom controllers; retention policies; cross-region replication; monitoring; and the production discipline for snapshot operations.→
Part LVI
Kubernetes Security Foundations
6 checks
- Threat modeling Kubernetes — who, what, where, whyThreat modeling for a Kubernetes cluster: identify trust boundaries (the kube-apiserver, the kubelet, etcd, the container runtime), enumerate actors (humans, workloads, operators, attackers), enumerate assets (Secrets, RBAC, etcd data, kubelet credentials), and produce a credible attack tree that drives the security roadmap.→
- Defense in depth — layered controls for the clusterDefense in depth for Kubernetes: layering controls at the network perimeter, the API server, the kubelet, the runtime, the kernel, and the workload. The principle that no single control is sufficient and that each layer assumes the layer above will fail.→
- Attack surface mapping — what is exposed to whomMapping the Kubernetes attack surface: every endpoint that accepts connections (API server, kubelet, etcd, runtimes, ingress controllers, webhooks), every credential that can be stolen (kubeconfigs, tokens, cloud creds), and every misconfiguration that expands the surface (anonymous auth, hostNetwork, privileged pods).→
- CVE landscape and known issues — staying currentThe Kubernetes CVE landscape: the cadence of releases, the severity tiers (Critical, High, Medium, Low), the historic CVEs that shaped the platform (CVE-2018-1002105, CVE-2019-11249, CVE-2020-8554, CVE-2022-3172, CVE-2023-3676, CVE-2024-3177), and the process for staying current without breaking the cluster.→
- Security posture assessment — measuring the clusterMeasuring the security posture of a Kubernetes cluster: the framework (CIS Benchmark, NSA hardening guide, distribution posture scores), the tooling (kube-bench, kube-hunter, kubeaudit, polaris, falco), the cadence (continuous, not annual), and how to turn the score into a roadmap.→
- Zero trust in Kubernetes — never trust, always verifyApplying zero-trust principles to Kubernetes: identity for every actor (human, workload, node), authentication for every request, authorisation for every verb, encryption for every channel, and continuous posture. The relationship between zero trust and defense in depth, and the failure modes of zero-trust theatre.→
Part LVII
Authentication
6 checks
- X.509 client certificates — the legacy defaultX.509 client certificates for Kubernetes authentication: how the TLS handshake authenticates the client, the CN/OU/O mapping to UserInfo, the long-lived credential problem, the rotation workflow with cert-manager, and why client certs are now reserved for bootstrapping and node identity, not human users.→
- Bearer tokens — long-lived, file-based, deprecatedBearer tokens for Kubernetes authentication: how the API server validates a token from `Authorization: Bearer ...`, the long-lived token file (`--token-auth-file`), the legacy ServiceAccount token Secret (deprecated in 1.24+, removed in 1.32+), and why both are superseded by projected tokens and OIDC.→
- Projected ServiceAccount tokens — TokenRequest and audienceProjected ServiceAccount tokens: how the kubelet issues short-lived tokens via the TokenRequest API, how the `audience` claim scopes them, how `expirationSeconds` controls the lifetime, and how the kubelet rotates the mounted token before it expires.→
- OIDC — corporate identity for human usersOIDC integration for Kubernetes: how the API server delegates human user authentication to a corporate IdP via `--oidc-issuer-url`, the `--oidc-client-id`, the `--oidc-username-claim` and `--oidc-groups-claim` mappings, and the kubeconfig flow for `kubectl` to obtain an ID token.→
- Anonymous access — the request that has no identityAnonymous requests in Kubernetes: how the API server maps an unauthenticated request to `system:anonymous` / `system:unauthenticated`, the production failure mode of leaving `--anonymous-auth=true` on the API server or kubelet, the `AlwaysAllow` authorization mode, and the audit log signals that anonymous access leaves behind.→
- Webhook authentication — TokenReview and external identityWebhook authentication for Kubernetes: how the API server delegates authentication to an external service via the TokenReview API, the use cases (custom IdP, Vault, SPIFFE/SPIRE, cloud workload identity), the implementation pattern (HTTPS service, TokenReview response), and the operational risks.→
Part LVIII
RBAC
6 checks
- Role and ClusterRole — the verbs the cluster allowsKubernetes RBAC Roles and ClusterRoles: the verbs (get, list, watch, create, update, patch, delete, deletecollection), the resources (pods, services, nodes, custom resources), the aggregation rules, and the production patterns for writing the smallest role that allows the work.→
- RoleBinding and ClusterRoleBinding — granting the roleKubernetes RoleBindings and ClusterRoleBindings: how a binding connects a subject (User, Group, ServiceAccount) to a role, the difference between namespace-scoped and cluster-scoped bindings, the production patterns for CI/CD, operators, and human users, and the failure modes of binding to the wrong subjects.→
- Verbs and resources — the RBAC decision matrixThe Kubernetes RBAC decision matrix: which verbs (`get`, `list`, `watch`, `create`, `update`, `patch`, `delete`, `deletecollection`) apply to which resources (Pods, Deployments, Services, Secrets, ConfigMaps, CRDs, RBAC objects), with concrete production examples for each combination.→
- kubectl auth can-i — testing RBAC decisionsUsing `kubectl auth can-i` to verify RBAC decisions: testing what a specific identity can do, the `--list` flag for the full set of permissions, the `--as` and `--as-group` flags for impersonation, the use cases (CI/CD, debugging, audits), and the failure modes (over-permissioned SAs, escaped impersonation).→
- Over-permissioned ServiceAccounts — the most common RBAC failureOver-permissioned ServiceAccounts in Kubernetes: the most common patterns (cluster-admin via Helm charts, wildcard ClusterRoles, default SA with broad bindings), how to audit them with `kubectl auth can-i`, the migration path to minimum-surface SAs, and the production failure modes.→
- RBAC anti-patterns — what to avoid and whyRBAC anti-patterns in production Kubernetes: wildcard resources, wildcard verbs, wildcards on system:masters, RBAC escalation primitives, over-broad binding subjects, the default SA with broad bindings, and the migration paths away from each.→
Part LIX
kubectl auth
6 checks
- kubectl auth can-i — verifying RBAC decisions in practicePractical use of `kubectl auth can-i` for verifying RBAC: the flags (--as, --as-group, --list, --subresource), the namespace scope, the integration with SubjectAccessReview, the CI/CD use cases, and the audit patterns for finding over-permissioned SAs.→
- kubectl auth reconcile — generating bindings from fileskubectl auth reconcile for RBAC management: applying RBAC manifests from version-controlled files, the difference between `kubectl apply` and `kubectl auth reconcile`, the use cases (GitOps RBAC, dry-run, drift detection), and the operational patterns.→
- kubectl auth impersonate — acting as another identitykubectl auth impersonate for RBAC debugging: how the API server validates impersonation, the flags (--as, --as-group, --as-uid), the audit log implications, the operational patterns (debugging, break-glass, testing), and the security risks of over-broad impersonation.→
- kubectl whoami — knowing what the cluster seeskubectl auth whoami (kubectl 1.30+) and related commands for identifying the current identity: how the API server reports the UserInfo, the flags for impersonation, the integration with OIDC and projected tokens, and the operational patterns for verifying kubeconfig.→
- kubectl create token — issuing and caching tokenskubectl create token for ServiceAccount token issuance: how the TokenRequest API is invoked, the flags (--audience, --duration, --bound-object), the caching on disk (~/.kube/cache), the use cases (CI/CD, debugging, third-party integrations), and the security risks.→
- Troubleshooting authentication — the diagnostic workflowTroubleshooting Kubernetes authentication: the systematic approach from `kubectl auth whoami` to kubeconfig inspection to API server logs, the common failure modes (expired token, wrong context, missing CA, anonymous access, RBAC denial), and the operational patterns for resolving each one.→
Part LX
ServiceAccounts
6 checks
- ServiceAccount anatomy — the workload identityKubernetes ServiceAccount anatomy: the workload identity object, the relationship to Secrets (legacy tokens), the auto-creation by the namespace controller, the fields (name, namespace, labels, annotations, automountServiceAccountToken), and the production patterns for minimum-surface SAs.→
- Token volume projection — the SA token in the PodToken volume projection in Kubernetes Pods: the projected token volume mounted by the kubelet, the path (`/var/run/secrets/kubernetes.io/serviceaccount/`), the contents (ca.crt, namespace, token), the expiry and rotation, and the operational patterns for the in-Pod credentials.→
- TokenRequest API — programmatic token issuanceThe Kubernetes TokenRequest API: programmatic issuance of projected tokens for ServiceAccounts, the request and response shape, the audience and duration controls, the bound-object form for stronger scoping, and the production use cases.→
- Default bindings and automountServiceAccountTokenThe default ServiceAccount bindings and automountServiceAccountToken: the `default` SA in every namespace, the auto-mounted token volume, when to set `automountServiceAccountToken: false`, the production patterns for minimum-surface defaults, and the failure modes.→
- SA tokens in pod spec — explicit projection and audiencesConfiguring SA tokens explicitly in the Pod spec: custom audience, custom expiry, custom path, multiple tokens in one Pod. The use cases for non-default audiences (Vault, AWS STS, custom APIs), and the failure modes.→
- External token issuers — IRSA, Workload Identity, SPIFFEExternal token issuance for Kubernetes workloads: AWS IRSA, GCP Workload Identity, Azure Workload Identity, and SPIFFE/SPIRE. How each one exchanges a projected SA token for cloud credentials, the configuration required, and the production patterns.→
Part LXI
Admission Control
6 checks
- Admission pipeline — the gatekeeper between authn and persistenceThe Kubernetes admission pipeline: how the API server runs MutatingAdmissionWebhook, ValidatingAdmissionWebhook, and built-in admission controllers between authentication and etcd persistence, the chain order, the failurePolicy, and the operational patterns.→
- ValidatingAdmissionPolicy — declarative policy in CELKubernetes ValidatingAdmissionPolicy: declarative policy expressed in CEL (Common Expression Language), the bindings, the failurePolicy, the common policies (no privileged, image registry whitelist, label requirements), and the migration from webhooks.→
- MutatingAdmissionWebhook — modifying objects at admissionKubernetes MutatingAdmissionWebhook: how the chain mutates objects before persistence, the common use cases (sidecar injection, default labels, image rewriting, secret transformation), the configuration, the failurePolicy, and the operational risks.→
- ValidatingAdmissionWebhook — external policy at admissionKubernetes ValidatingAdmissionWebhook: how external policy services reject or accept objects at admission, the configuration, the failurePolicy, the use cases (OPA, Kyverno, image signature), and the operational patterns for HA webhooks.→
- ImagePolicyWebhook — deprecated, the predecessor to CosignImagePolicyWebhook: the deprecated admission controller for image policy enforcement. How it works, why it was deprecated, the migration path to Cosign and Kyverno, and the lessons from its design.→
- Admission best practices — the operational disciplineAdmission control best practices: the layered approach (built-in controllers + declarative policies + webhooks for external lookups), the failurePolicy choice, the observability, the migration from webhooks to CEL, and the production failure modes.→
Part LXII
Pod Security Standards
6 checks
- Pod Security Standards — overview of the three profilesPod Security Standards overview: the three profiles (privileged, baseline, restricted), what each one allows and forbids, the relationship to PodSecurityPolicy (deprecated) and OPA/Kyverno, and the production pattern of enforcing `restricted` cluster-wide.→
- Privileged profile — when and how to allow full accessThe Pod Security Standards `privileged` profile: when full access is appropriate (system workloads, operators, debug), the namespace label configuration, the real manifest examples, and the operational discipline of containing privileged workloads.→
- Baseline profile — preventing known privilege escalationsThe Pod Security Standards `baseline` profile: the middle ground that prevents known privilege escalations while allowing common operational patterns. The fields it forbids, the fields it allows, the multi-tenant use case, and the production patterns.→
- Restricted profile — minimum-allow for productionThe Pod Security Standards `restricted` profile: the strictest profile, the production target. Every field it requires (runAsNonRoot, allowPrivilegeEscalation: false, capabilities drop ALL, seccompProfile RuntimeDefault, the allowed volume types), the real manifest examples, and the failure modes of over-restriction.→
- PSA labels and modes — enforcing PSS at admissionPod Security Admission (PSA) labels and modes: the namespace labels (enforce, audit, warn), the three modes, the version label for opt-out, the operational patterns, and the failure modes of over-restrictive enforcement.→
- PSS migration — moving workloads to restrictedMigrating workloads to Pod Security Standards `restricted`: the systematic approach (audit, fix, enforce), the tools (kube-linter, Kyverno, polaris), the per-workload patterns, the production failures, and the operational discipline.→
Part LXIII
Linux Security Controls in Kubernetes
6 checks
- Linux capabilities — the kernel privilege boundaryLinux capabilities in Kubernetes: how the kernel splits root privileges into discrete capabilities, how Pod Security Standards restrict capabilities, the right pattern (`drop ALL`, add only what is needed), and the production failure modes.→
- Seccomp RuntimeDefault — the runtime's safe baselineSeccomp RuntimeDefault profile in Kubernetes: how the container runtime's default seccomp profile constrains the syscall surface, the relationship to PSS `restricted`, the configuration, and the operational patterns.→
- Custom seccomp profiles — Localhost and workload-specific rulesCustom seccomp profiles in Kubernetes: the `Localhost` profile type, the JSON profile format, the workflow for authoring a profile (audit syscalls, generate the profile, test), the operational patterns, and the failure modes.→
- AppArmor profiles — kernel-enforced workload confinementAppArmor profiles in Kubernetes: how the kernel enforces workload confinement, the built-in profiles (runtime/default), the custom profiles via the AppArmor CRD, the configuration in the Pod spec, and the production patterns.→
- SELinux — kernel-enforced confinement on RHEL-based nodesSELinux in Kubernetes: how the kernel enforces confinement on RHEL-based distributions, the SELinux modes (enforcing, permissive, disabled), the `seLinuxOptions` in the Pod spec, the multi-category security (MCS) labels, and the production patterns.→
- Security Context — runAsUser, runAsNonRoot, readOnlyRootFilesystemThe Pod securityContext: the field that controls runAsUser, runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation, and fsGroup. The right configuration for production, the interaction with PSS profiles, and the failure modes.→
Part LXIV
Kubernetes Supply Chain Security
6 checks
- Image tags vs digests — pinning for supply chain integrityContainer image tags vs digests in Kubernetes: how tags are mutable and digests are immutable, the security implications of `latest`, the practice of pinning by digest, and the production patterns for image integrity.→
- SBOM — Software Bill of Materials for supply chainSoftware Bill of Materials (SBOM) for Kubernetes workloads: the standards (SPDX, CycloneDX), the formats (JSON, XML, protobuf), the tools (Syft, Trivy, Bomctl), the integration with admission (Kyverno, Cosign), and the production patterns.→
- Vulnerability scanning — detecting known CVEs in imagesVulnerability scanning for Kubernetes workloads: the tools (Trivy, Grype, Clair, Snyk), the integration with admission (Kyverno, ValidatingAdmissionPolicy), the failurePolicy for CVE thresholds, the workflow for triaging findings, and the production patterns.→
- Cosign — image signing and verificationCosign for Kubernetes supply chain security: signing images with a private/public key pair, attaching signatures to OCI artifacts, verifying at admission (Kyverno, ValidatingAdmissionPolicy), the key management (keyless via Fulcio), and the production patterns.→
- Notary v2 — supply chain attestations beyond signaturesNotary v2 (Notation) for Kubernetes supply chain: signing OCI artifacts with the Notary v2 specification, the relationship to Cosign, the use cases (SBOMs, vulnerability reports, SLSA provenance), the integration with admission, and the production patterns.→
- SLSA provenance — attesting the build processSLSA (Supply-chain Levels for Software Artifacts) for Kubernetes: the levels (1-4), the provenance attestation, the integration with build systems (Tekton, BuildKit, GitHub Actions), the verification with Notation and Cosign, and the production patterns.→
Part LXV
Secrets Security
6 checks
- Secret types — the Kubernetes Secret primitivesKubernetes Secret types: the built-in types (Opaque, kubernetes.io/service-account-token, kubernetes.io/dockercfg, kubernetes.io/dockerconfigjson, kubernetes.io/tls, basic-auth, ssh-auth), the use case for each, and the security implications.→
- Etcd encryption — Secrets at restEncrypting Kubernetes Secrets at rest in etcd: the EncryptionConfiguration API, the providers (identity, aescbc, kms, secretbox), the key rotation, the operational patterns, and the failure modes.→
- Secret mounting risks — how Secrets leakThe risks of mounting Kubernetes Secrets: env vars (visible to all processes in the container), volume mounts (file permissions), the kubelet audit, the risks of shell history and crash dumps, and the production patterns.→
- Secrets RBAC — controlling who can read SecretsSecrets RBAC: the `secrets` resource, the verbs (`get`, `list`, `watch`, `create`, `update`, `patch`, `delete`), the production patterns (per-Secret RBAC, ResourceNames, default-deny), the common failure modes (broad access, log exposure), and the audit workflow.→
- External Secrets Operator — Secrets outside KubernetesExternal Secrets Operator (ESO) for Kubernetes: fetching Secrets from Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault; the SecretStore and ExternalSecret CRDs; the rotation; the production patterns.→
- Secret best practices — the operational disciplineKubernetes Secret best practices: the layered controls (etcd encryption, RBAC, external stores, sealed-secrets, mounting), the rotation procedures, the audit workflow, and the production failure modes.→
Part LXVI
etcd
6 checks
- etcd as the Kubernetes database — what it stores, who writes, who readsetcd in the Kubernetes control plane: the API object graph, the WAL, the snapshot, the storage backend, the failure domain. The concepts every operator must hold in mind before the first incident.→
- Raft consensus — leader election, terms, log replicationRaft as etcd uses it: terms, elections, log replication, commit semantics, and what guarantees Raft gives a Kubernetes cluster under member loss and network partitions.→
- Write-ahead log, bbolt, and the on-disk formatetcd on disk: the write-ahead log (WAL), the snapshot store, the bbolt-backed key-value store, the directory layout on a member, what gets fsynced when, and how a restart reconstructs state.→
- Snapshots, compaction, defragmentation — 8 GB warning, 2 GB recommendedWhy etcd has an 8 GB warning and a 2 GB recommended DB size: snapshot creation, key compaction, bbolt fragmentation, defragmentation procedure, and the maintenance discipline of running etcd at production scale.→
- Performance limits — disk fsync, write rate, latency budgetsetcd performance limits that drive the cluster size, the request rate, and the latency budget: disk fsync, write rate, object count, key size, and the operational signals that predict an outage.→
- etcd flags, environment, and tuning the operator controlsThe etcd command-line flags, the kubeadm-managed static pod, environment variables, certificates, the Election/Heartbeat/Quota tunables, and the discipline of changing them safely.→
Part LXVII
etcd Quorum
6 checks
- Quorum math — floor(N/2)+1, odd members, fault toleranceetcd quorum: the majority rule, why odd numbers matter, fault tolerance by cluster size, the difference between "members down" and "quorum lost", and the calculations every etcd operator runs.→
- Three vs five members — design choices, write latency, fault toleranceTrade-offs between 3-member and 5-member etcd clusters: write latency, member-replacement cost, failure-domain coverage, and the operational considerations that drive the choice.→
- Split-brain prevention — how Raft refuses to fabricate leadershipRaft's safety property that prevents split-brain: a leader cannot commit unless it has quorum, a partitioned leader cannot commit unless partition heals, and how Kubernetes operators reason about leader authority under failure.→
- Member lifecycle — add, remove, replace, retireHow etcd members join, leave, are removed, and are replaced: the operational sequence for each, kubeadm-managed member changes vs manual etcdctl, and the production discipline of growing and shrinking an etcd cluster safely.→
- Failure domain placement — AZs, racks, hosts, and cost of spreadPlacing etcd members across failure domains: how AZ layout determines fault tolerance, why tie-breaking across an odd number of AZs is essential, and the production patterns for spread vs stacked.→
- Quorum loss recovery — when a cluster is stuck, snapshot restore is the pathWhat to do when etcd quorum is lost: how to recognise the state, the snapshot restore sequence on a 3-member cluster, the bootstrap with --initial-cluster, and the production discipline of rehearsing this recovery path.→
Part LXVIII
etcd Backup
6 checks
- Why back up etcd — the failure modes only snapshots coverThe cases where etcd is the only thing to recover from: cluster state corruption, accidental deletions, control-plane loss, rollback of API-level mistakes, and DR exercises.→
- etcdctl snapshot save — flags, options, and integrationThe `etcdctl snapshot save` command in depth: the TLS flags, the endpoint selection, the off-cluster destination, the runtime impact on a live member, and the kubeadm `kubeadm-init` style integration patterns.→
- Backup storage strategy — local, off-cluster, object storageWhere to put etcd snapshots so they survive the failures they need to survive: local fast volume, off-cluster copy, object storage with versioning and lifecycle, and the discipline of multi-region DR.→
- Encryption at rest for snapshots — what to encrypt and howEncrypting etcd snapshots: the secrets they contain by default, the API server encryption-at-rest configuration, server-side encryption in object storage, client-side encryption before upload, and the key management discipline.→
- Snapshot validation — verify before you trustVerifying etcd snapshots: `etcdutl snapshot status` checks, hash verification, restore-to-temp validation, schema sanity checks, and the alerting discipline around broken snapshots.→
- Backup cadence and retention — schedule, RPO, RTO, complianceDesigning a backup cadence for etcd: hourly and daily cadences, retention windows, alignment with RPO and RTO, compliance retention, and the trade-offs with cost and verification load.→
Part LXIX
etcd Restore
6 checks
- When to restore — the decision treeThe decision tree that determines whether to restore from a snapshot vs reset/restore/reconcile: identifying the failure mode, capturing the snapshot, mapping the recovery path, and validating readiness.→
- Pre-flight checks — what to verify before running the restoreThe pre-flight checks before any etcd restore: snapshot verification, certificate mapping, network reachability, peer URL consistency, and the host-level preparation that prevents restore-time surprises.→
- Stopping the control plane — API server, controller manager, schedulerThe sequence to stop the Kubernetes control plane before etcd restore: API server first (the only writer), then kube-controller-manager, kube-scheduler, kubelet considerations, and the production discipline of clean shutdowns.→
- etcdutl snapshot restore — the per-host commandsThe `etcdutl snapshot restore` command in depth: arguments for each member host, --initial-cluster, --initial-advertise-peer-urls, the data-dir placement, the timing expectations, and the per-host sequence.→
- Restarting the cluster — bringing etcd back up safelyBringing the restored etcd cluster up: per-host static pod start, election sequence, member discovery via --initial-cluster, validation of the new cluster state before restarting the API server.→
- Validating the restored cluster — production readiness checksVerification after etcd restore: API server responses, Namespace and object count match, RBAC binds, workload reconciliation, secrets accessibility, storage and networking validation, and the production readiness gate.→
Part LXX
API Server
6 checks
- API server role — the cluster gateway and only writer to etcdThe kube-apiserver's role in a Kubernetes cluster: the front door for every request, the only writer to etcd through the API contract, the source of the watch feed, and the reconciler for control-plane components.→
- Request lifecycle — authentication, authorisation, admissionA kube-apiserver request from arrival to response: TLS handshake, authentication, authorisation, mutating admission, schema validation, optimistic concurrency, validating admission, encryption, etcd write, audit, watch fan-out, response.→
- Watch semantics — list-watch, resourceVersion, informerHow the API server's watch feed works: list-watch, resourceVersion as a cursor, watch bookmarks, informer caches in client-go, and the production discipline of reliable watching.→
- Aggregated API servers — kube-aggregator, extension pointsKubernetes API extension: aggregated API servers, the kube-aggregator, the apiservice registration, the certification flow with the API server, and the production discipline of running extended API surfaces.→
- API server flags and configuration — secure-port, audit, encryptionkube-apiserver flags in production: secure-port, service-account signing keys, audit log policy, encryption at rest, profiling, the kubeadm-managed static pod, and the discipline of changing flags safely.→
- API server HA — multiple instances, load balancing, failoverHigh availability for the kube-apiserver: multiple instances, load balancing, failover behaviour, the load balancer configuration (HAProxy, keepalived, AWS NLB), session affinity considerations, and the production discipline of HA topographies.→
Part LXXI
Scheduler
6 checks
- Scheduler architecture — informer, queue, scheduling cycleThe kube-scheduler: the informer that watches unscheduled Pods, the internal queue, the scheduling cycle (filter then score then reserve then permit then bind), the cache of node state, and the production discipline of scheduler health.→
- Filtering — feasibility across nodesThe filter phase of scheduling: how the scheduler determines which nodes are feasible for a Pod, the role of NodeAffinity, NodeSelector, Taints, PodAffinity, resources, volumes, and the production patterns of filtering.→
- Scoring — least allocated, balanced, topology, customThe score phase of scheduling: how the scheduler ranks feasible nodes, the default score plugins (LeastAllocated, BalancedAllocation, NodeAffinity preferred, TaintToleration, TopologySpread), and how custom scores plug in.→
- Reserve, permit, bind — the cycle's latter phasesThe reserve, permit, and bind phases of the scheduling cycle: how the scheduler claims resources, awaits permit approval, and binds the Pod to the chosen node; failure modes and the production discipline of these phases.→
- Scheduling framework — plugins, profiles, extensibilityThe Kubernetes scheduling framework: the extension points (PreFilter, Filter, PostFilter, PreScore, Score, NormaliseScore, Reserve, Permit, PreBind, Bind, PostBind, Unreserve), custom plugin patterns, scheduling profiles, and the discipline of extending the scheduler.→
- Multiple schedulers and profiles — coexistence and opt-inKubernetes' model for multiple schedulers and profiles: leader election, two schedulers in one cluster, schedulerName-based opt-in, profile composition, and production patterns for mixed workloads.→
Part LXXII
Controller Manager
6 checks
- Controller manager overview — the cluster's automation enginekube-controller-manager: what it does, the controllers it runs, how they reconcile, leader election between instances, and the production discipline of running it.→
- Node controller — heartbeats, NotReady, evictionThe node controller: the leader-elected controller that watches kubelet heartbeats, marks NotReady after grace, applies the NoExecute taint, evicts Pods, and decides when nodes return to Ready.→
- Deployment, ReplicaSet, and other workload controllersThe Deployment controller and the ReplicaSet, StatefulSet, DaemonSet, Job, CronJob controllers: how each reconciles to desired state, the rollout dance, and the production discipline of workload controllers.→
- EndpointSlice controller — Service-to-Pod traffic surfaceThe EndpointSlice controller: how it discovers Pods matching a Service selector, builds EndpointSlices, propagates them, and the production patterns for managing many Endpoints.→
- ServiceAccount and Token controllers — identity for PodsThe ServiceAccount controller and the Token controller: how Kubernetes ensures every Pod has a token-bound identity, the legacy token semantics, the projected tokens, and the production discipline of identity in the cluster.→
- Custom controllers — extending the cluster with codeThe pattern of writing custom Kubernetes controllers: informer / work queue / reconcile, the kubebuilder / operator-sdk toolchain, CRD-backed custom resources, webhook-based validation, and the production discipline of running custom controllers.→
Part LXXIII
Control Plane High Availability
6 checks
- HA topology — stacked vs external etcd, design choicesControl-plane HA topology decisions: stacked etcd on the control-plane hosts vs external etcd on dedicated hosts, the trade-offs in failure isolation, operational cost, and scaling; how to choose for production.→
- Load balancer — HAProxy, keepalived, MetalLB, cloud LBLoad balancer for the Kubernetes API server: HAProxy with keepalived for on-prem, AWS NLB / GCP LB / Azure LB for cloud, MetalLB for service IP advertisement, and the production discipline of LB configuration.→
- DNS and kubeconfig — distributing the API server endpointHow the kubeconfig reaches the API server: DNS records for the API VIP, distributing kubeconfig to operators, kubelet kubeconfig on workers, credential rotation, and the production discipline of kubeconfig management.→
- Failure domain design — AZ placement and topologyDesigning the control-plane failure domains: AZ placement of etcd, control-plane hosts, and the LB; racks, power domains, network segments; the discipline of avoiding correlated failure.→
- Backup before change — the snapshot as a change gateThe discipline of taking an etcd snapshot before any control-plane change: cluster upgrades, kubeadm config changes, control-plane node operations, and the production gate that prevents irreversible mistakes.→
- HA validation — chaos testing, drills, observabilityValidating control-plane HA: chaos testing (kill a host, kill etcd, kill the LB), restore drills, observability for HA, drill cadence, and the production discipline of continuous validation.→
Part LXXIV
kubeadm
6 checks
- kubeadm init — phases, output, and post-init sanity`kubeadm init` in depth: the 12 phases (certs, control-plane, etcd, kubeconfig), the output (admin.conf, join command), the post-init sanity checks, and the production discipline of init.→
- kubeadm join — control-plane and worker node bootstrapping`kubeadm join` for adding nodes: the worker flow, the control-plane flow with --control-plane, the certificate distribution, and the production discipline of cluster growth.→
- Certificate management — kubeadm PKI, rotation, validationCertificate management in kubeadm-managed clusters: the PKI layout, the certs involved, kubeadm certs phases, rotation timing, validation, and the production discipline of certificate expiry prevention.→
- kubeadm-config ConfigMap — ClusterConfiguration, KubeletConfigurationThe kubeadm-config ConfigMap: what it contains (ClusterConfiguration, KubeletConfiguration, KubeProxyConfiguration), how it is read at runtime, how to edit it safely, and how upgrades preserve it.→
- Upgrade plan — prepare, control-plane, workers in orderPlanning a Kubernetes upgrade: the upgrade plan, version-slew policy, backup-before-upgrade, etcd backup + restore drill, control-plane upgrade first then workers, and the production discipline of staged upgrades.→
- kubeadm upgrade apply — phases, validation, post-upgrade checksThe `kubeadm upgrade apply` command in depth: the per-host phases, control-plane static pod upgrades, kubelet restart, validation at each step, and the production discipline of staged upgrades.→
Part LXXV
Building a Production Cluster
6 checks
- Capacity sizing — control plane and worker node sizingSizing the control plane and workers: hardware minimums (≥2 vCPU, ≥4GB for small clusters; ≥4 vCPU, ≥16GB for production), worker sizing by role, capacity headroom, and the production discipline of sizing.→
- OS and kernel tuning — sysctl, transparent huge pages, disk schedulersLinux tuning for Kubernetes production: kernel parameters (sysctl), transparent huge pages, disk schedulers, swap, kernel modules, file descriptors, and the OS-level preparation that makes a production cluster stable.→
- Container runtime — containerd, runc, and runtime choiceThe container runtime in a Kubernetes node: containerd as the de facto standard, runc as the low-level runtime, runtime alternatives, configuration, and the production discipline of runtime operations.→
- CNI selection — Cilium, Calico, Flannel, and choosingCNI plugin selection for production: Cilium (eBPF, leading), Calico (mature, flexible), Flannel (simple). Coverage, performance, NetworkPolicy, eBPF vs iptables, and the production discipline of CNI choice.→
- Load balancer for the cluster — keepalived, HAProxy, cloud LBThe cluster's load balancer: keepalived (VIP) and HAProxy (proxy) on-prem, cloud-managed LBs on cloud platforms, BGP-based advertisement with MetalLB for Service-type-LoadBalancer, and the production discipline of LB choice.→
- Time sync, DNS, certificates — foundations and observability stackOperational foundations for production: time sync (chrony / NTP) and what breaks without it, DNS for cluster services and ingress, certificates (cert-manager) and renewal, and the choice of observability stack.→
Part LXXVI
Cluster Certificates
6 checks
- Cluster certificates — the PKI that holds the cluster togetherThe kubeadm-managed cluster PKI in /etc/kubernetes/pki: every certificate, what it identifies, when it expires, and how the cluster breaks when any one of them fails.→
- Default 1-year expiry — the clock that is always runningThe 1-year default validity of kubeadm-managed certificates, why that default exists, when to shorten it, and how to keep the cluster alive across the boundary.→
- check-expiration — inventorying the cluster certsThe kubeadm certs check-expiration command: usage, output format, integration with monitoring, and how to use it as the source of truth for the cluster cert inventory.→
- Certificate renewal — kubeadm certs renew and the rotation sequenceRenewing kubeadm-managed certificates: the renew subcommand, the order of operations (followers first, leader last), the static pod restart, and the verification steps.→
- Cert rotation in /etc/kubernetes/pki — the file-by-file walkThe file-by-file walk of /etc/kubernetes/pki during cert rotation: which files are written, which are read, the relationships between CA, server, and client certs, and the static pod manifest that triggers the restart.→
- Cert expiry monitoring — alerting before the outageMonitoring kubeadm-managed certs in production: the Prometheus textfile exporter, the blackbox_exporter probe, alerting thresholds (30/15/7 days), and the runbook integration.→
Part LXXVII
Kubernetes Upgrades
6 checks
- Upgrade sequence — read release notes, backup, control plane, then workers in wavesThe full Kubernetes upgrade sequence: read release notes, backup etcd, upgrade control plane one node at a time, validate, then workers in waves. The discipline of staged upgrades.→
- kubeadm upgrade apply — the leader's upgradeThe kubeadm upgrade apply command: usage, phase-by-phase execution, the static pod manifest updates, and the verification steps. The leader's upgrade is the most disruptive part of the upgrade sequence.→
- Pre-upgrade backup — the snapshot that catches the rollbackThe pre-upgrade etcd snapshot: when to take it, how to verify it, where to store it, and how it integrates with the upgrade procedure. The snapshot is the safety net for any upgrade.→
- Worker drain and upgrade — the in-place upgrade patternThe pattern for upgrading a worker in-place: drain, kubeadm upgrade node, restart kubelet, uncordon. The PDB-driven wave strategy and the validation steps.→
- CNI and add-on upgrades — the post-control-plane migrationsThe post-control-plane upgrades: CNI, CoreDNS, kube-proxy, ingress controllers. Compatibility with the new Kubernetes version, the order of migrations, and the verification steps.→
- Post-upgrade validation — confirming the cluster is healthyThe validation steps after a Kubernetes upgrade: node health, pod health, API server health, etcd health, workload smoke tests, and the post-upgrade snapshot. The discipline of confirming the upgrade succeeded.→
Part LXXVIII
Version Skew
6 checks
- Version skew policy — the compatibility matrixThe Kubernetes version skew policy: what combinations of kube-apiserver, kube-controller-manager, kube-scheduler, cloud-controller-manager, kube-proxy, kubelet, kubectl, and etcd are supported. With 1.34 as the verified target.→
- kube-apiserver vs kubelet — the asymmetric skewThe asymmetric skew between kube-apiserver and kubelet: how a kubelet can be up to 3 minor versions behind, why the asymmetry exists, and how it enables rolling worker upgrades.→
- Controller-manager and scheduler — the symmetric skewThe symmetric skew between kube-apiserver and kube-controller-manager / kube-scheduler: both components can be up to 1 minor ahead or behind the apiserver. The rationale and the upgrade implications.→
- kubectl compatibility — the operator clientThe kubectl version compatibility policy: within 1 minor of the apiserver, the kubectl skew practice, and the operational discipline of keeping kubectl current.→
- etcd compatibility — the K8s minor and the etcd minorThe etcd version compatibility policy: how the Kubernetes minor version maps to the etcd version, the supported combinations, and the upgrade implications.→
- Skew validation tooling — verifying the cluster is in policyThe tooling for validating the Kubernetes version skew: kubectl version, kubectl get nodes, kubeadm upgrade plan, and the team-specific scripts that enforce the policy before any upgrade.→
Part LXXIX
API Deprecation
6 checks
- kubectl deprecations — inspecting manifest compatibilityThe kubectl deprecations command introduced in Kubernetes 1.30: how to use it to identify deprecated APIs in manifests, the format of the output, and the integration with the upgrade workflow.→
- pluto — finding deprecated APIs in Helm chartsThe pluto tool from Fairwinds: detecting deprecated Helm chart APIs, the helm-pluto plugin, and the integration with CI/CD pipelines for upgrade-safety checks.→
- kube-no-trouble — the cluster-wide deprecation auditThe kube-no-trouble (kubent) tool: detecting deprecated APIs in the live cluster, the use cases for upgrade validation, and the integration with the upgrade workflow.→
- --dry-run=server for manifest review — server-side validationUsing kubectl --dry-run=server to validate manifests against the live apiserver before applying. The interaction with deprecated APIs, schema validation, and CI/CD integration.→
- API removal planning — the 9-month calendarThe Kubernetes API removal policy: 9 months deprecation, 3 minor versions, the timeline for upgrading, and the planning involved in addressing deprecated APIs before they are removed.→
- CI/CD gating for deprecated APIs — the pipeline enforcementIntegrating the deprecation tools into CI/CD pipelines: pluto, kubent, and kubectl deprecations as pipeline gates. The PR-blocking rules and the audit trail.→
Part LXXX
Worker Node Upgrades
6 checks
- Worker upgrade patterns — in-place vs surgeThe two patterns for upgrading Kubernetes workers: in-place (kubeadm upgrade node on the host) and surge (replace the worker with a new VM at the new version). The tradeoffs and the use cases for each.→
- Drain before upgrade — the rule of patienceDraining a worker before an upgrade: the kubectl drain command, the flags, the PDB-bounded wave size, and the failure modes of a missed drain.→
- Surge and replace — the cloud-native worker upgradeThe surge-and-replace pattern for cloud workers: cordoning the old worker, launching a new worker at the new version, and terminating the old worker. The cluster autoscaler integration.→
- kubelet-only upgrade — the patch-level maintenanceThe kubelet-only upgrade pattern: upgrading the kubelet binary on a worker without changing the cluster version. Used for patch-level maintenance and security updates.→
- Workers in waves — the PDB-driven schedulingThe wave strategy for worker upgrades: calculating the wave size from the workload PDBs, sequencing the waves, and the validation steps. The discipline of staged upgrades.→
- Worker validation after upgrade — confirming the cluster is healthyThe validation steps after a worker upgrade: node version, pod health, workload smoke tests, and the cluster-wide health check. The discipline of confirming the upgrade succeeded.→
Part LXXXI
Cluster Autoscaling Concepts
6 checks
- Cluster Autoscaler — the cluster-level scalerThe Kubernetes Cluster Autoscaler: scaling the cluster's worker node count based on unschedulable pods, the cloud-provider integration, and the relationship with the Horizontal Pod Autoscaler.→
- Node group configuration — the scaling targetsConfiguring node groups for the Cluster Autoscaler: the AWS ASG, GCP MIG, Azure VMSS, the instance types, the min/max/desired capacity, and the auto-discovery tags.→
- Scale-up triggers — when the cluster growsThe Cluster Autoscaler scale-up triggers: unschedulable pods, the scan interval, the scale-up delay, the bin-packing, and the failure modes of a missed scale-up.→
- Scale-down triggers — when the cluster shrinksThe Cluster Autoscaler scale-down triggers: underutilized nodes, the scale-down delay, the unneeded time, the PDB-bound scale-down, and the failure modes of an aggressive scale-down.→
- Cooldown and balance — the multi-node-group strategyThe Cluster Autoscaler cooldown and balance strategies: scale-down delay after add, similar node group balancing, the expander priority, and the production tuning of the autoscaler.→
- Cluster Autoscaler integration with cloud providers — the contractThe Cluster Autoscaler integrations with AWS, GCP, Azure, and other cloud providers: the IAM permissions, the API contracts, the cloud-specific tags, and the failure modes of misconfigured integration.→
Part LXXXII
Horizontal Pod Autoscaler
6 checks
- HPA — Horizontal Pod Autoscaler conceptsThe Horizontal Pod Autoscaler: scaling the number of pod replicas based on metrics, the metrics.k8s.io API, the controller loop, and the production patterns.→
- metrics.k8s.io and the Metrics Server — the default metrics sourceThe metrics.k8s.io API and the Metrics Server: the deployment, the kubelet integration, the API surface, and the integration with HPA and kubectl top.→
- Custom metrics API — application-specific scalingThe Custom Metrics API and the External Metrics API: deploying a custom metrics adapter (Prometheus Adapter), the API surface, and the integration with HPA.→
- Behavior block — stabilization and scaling policiesThe HPA behavior block: stabilizationWindowSeconds, scaling policies, scaleUp and scaleDown configuration. The production tuning of the HPA for stable scaling.→
- Scaling policies — controlling the rate of changeThe HPA scaling policies: Percent and Pods, selectPolicy, the periodSeconds, and the production tuning of the rate of change. The asymmetric scaling and the production tests.→
- HPA anti-patterns — the pitfalls to avoidThe HPA anti-patterns: scaling on memory, custom metrics that are not pod-level, mixing HPA with VPA, missing resource requests, and the production discipline of HPA design.→
Part LXXXIII
Vertical Pod Autoscaling Concepts
6 checks
- VPA — Vertical Pod Autoscaling conceptsThe Vertical Pod Autoscaler (VPA): scaling the resource requests and limits of pods, the components (recommender, updater, admission controller), and the relationship with HPA.→
- VPA recommender — the recommendation engineThe VPA Recommender: how it computes the recommended requests, the percentile-based approach, the historical data collection, and the inspection of the recommendations.→
- VPA updater — evicting pods with new requestsThe VPA Updater: how it evicts pods with sub-optimal requests, the eviction strategy, the PDB-aware eviction, and the failure modes of an aggressive updater.→
- VPA admission controller — the mutating webhookThe VPA admission controller: how it sets the resource requests on new pods, the mutating webhook, the interaction with the Pod spec, and the failure modes of an unavailable admission controller.→
- VPA vs HPA — when to use whichThe VPA vs HPA decision: when to use VPA, when to use HPA, the workloads that benefit from each, and the production patterns that combine them.→
- VPA limitations — the boundaries of vertical scalingThe VPA limitations: the pod restart requirement, the interaction with HPA, the resource limits, the storage of usage history, and the production patterns that work around these limits.→
Part LXXXIV
Resource Capacity Planning
6 checks
- Capacity planning — the discipline of resource budgetingKubernetes capacity planning: the discipline of resource budgeting, the inputs (workload requirements, headroom, growth), the outputs (cluster sizing, cost projection), and the production patterns.→
- Resource requests and limits — the resource budgeting primitivesKubernetes resource requests and limits: the resource budgeting primitives, the scheduler integration, the QoS classes, and the production discipline of resource budgeting.→
- Cluster utilization analysis — the metrics of capacityAnalyzing cluster utilization for capacity planning: the Prometheus metrics, the cluster efficiency ratio, the per-workload usage, and the production patterns for capacity decisions.→
- Bin packing — fitting workloads on nodesKubernetes bin packing: fitting workloads on nodes, the scheduler algorithm, the resource fragmentation, the affinity and anti-affinity, and the production patterns for bin packing.→
- Right-sizing — the discipline of resource optimizationThe right-sizing discipline: the tools (VPA, kubectl top, Prometheus), the timing, the workflow, and the production patterns for keeping the workload's resources aligned with the actual usage.→
- Capacity forecasting — the planning horizonCapacity forecasting: the planning horizon, the inputs (workload growth, business projections), the outputs (capacity plan, cost projection), and the production patterns for capacity forecasting.→
Part LXXXV
Cluster Observability
6 checks
- Cluster observability — the four pillarsKubernetes cluster observability: the four pillars (metrics, logs, traces, events), the signals, the collection, the storage, and the production patterns for cluster observability.→
- Signals — metrics, logs, traces, events in depthThe four observability signals in depth: the metrics types (counter, gauge, histogram, summary), the log levels, the trace spans, and the event types. The production patterns for each signal.→
- Cluster-level vs workload-level observabilityThe two scopes of Kubernetes observability: cluster-level (the cluster's health, the API server, the kubelet) vs workload-level (the application's behavior). The signals, the tools, and the production patterns.→
- Observability stack components — the toolboxThe observability stack components: collectors, storage, query layer, dashboards, alerting. The relationship between the components, the deployment patterns, and the production architecture.→
- SLO/SLI/SLA — the observability-driven targetsSLOs, SLIs, and SLAs in cluster observability: the SLO defines the target, the SLI measures it, the SLA is the contract. The error budget, the burn rate, and the production patterns.→
- Observability maturity — the progression of the disciplineThe observability maturity model: from reactive monitoring to proactive observability, the levels, the signals, the alerts, and the production patterns for each level.→
Part LXXXVI
kube-state-metrics
6 checks
- kube-state-metrics — the cluster's object metricsThe kube-state-metrics (KSM) tool: generating metrics from the Kubernetes API objects, the deployment, the metrics taxonomy, and the integration with Prometheus.→
- KSM metrics taxonomy — the cluster's object stateA walk through the KSM metrics taxonomy: pod metrics, deployment metrics, node metrics, job metrics, service metrics, and the labels that accompany each metric.→
- KSM deployment — installing the metrics collectorThe kube-state-metrics deployment: the manifest, the Helm chart, the security model (RBAC), and the production patterns for high availability and scaling.→
- KSM vs cAdvisor — different scopes of metricskube-state-metrics vs cAdvisor: the difference between the object metrics and the resource metrics, the metrics sources, the metrics types, and the production patterns for combining them.→
- KSM Prometheus integration — the metrics flowThe kube-state-metrics integration with Prometheus: the ServiceMonitor, the scrape config, the recording rules, the alerts, and the production patterns.→
- KSM labels and annotations — the metadata for queriesThe kube-state-metrics labels and annotations: the metric labels for filtering, the annotations for enriching, the queries, and the production patterns.→
Part LXXXVII
Metrics Server
6 checks
- Metrics Server — the resource metrics APIThe Kubernetes Metrics Server: the resource metrics API, the kubelet integration, the API surface, and the integration with HPA and kubectl top.→
- Resource metrics API — the metrics.k8s.io surfaceThe metrics.k8s.io API surface: the nodes, the pods, the namespaces, the API versions, the resource usage, and the production patterns for the API.→
- Metrics Server deployment — installing the resource metricsThe Kubernetes Metrics Server deployment: the Helm chart, the kubelet TLS configuration, the RBAC, the API Service, and the production patterns for high availability.→
- kubectl top internals — the interactive metrics querykubectl top: the interactive metrics query, the internal flow, the columns, the sort options, and the production patterns for using kubectl top.→
- HPA integration with Metrics Server — the scaling loopThe HPA integration with the Metrics Server: the resource metrics API, the scaling loop, the tolerance, the recommendation, and the production patterns for the HPA.→
- Metrics Server scaling — large clusters and high availabilityScaling the Metrics Server for large clusters: the bottlenecks, the configuration, the HA, and the production patterns for high-throughput Metrics Server deployments.→
Part LXXXVIII
Prometheus Monitoring
6 checks
- Prometheus — the cluster metrics foundationPrometheus on Kubernetes: the deployment model, the Operator vs kube-prometheus-stack, the service discovery role, and the integration with Grafana.→
- Prometheus Operator vs kube-prometheus-stack — the deployment choicePrometheus Operator vs kube-prometheus-stack: the difference, the deployment scope, the customization, the upgrade path, and the production patterns for each.→
- Service discovery — the Kubernetes integrationPrometheus service discovery in Kubernetes: the kubernetes_sd_configs, the ServiceMonitor, the PodMonitor, the relabeling, and the production patterns for discovery.→
- Recording rules — pre-computing the metricsPrometheus recording rules on Kubernetes: the PrometheusRule CRD, the rule files, the recording pattern, the evaluation interval, and the production patterns for the recording rules.→
- Alerting rules — the Prometheus to Alertmanager flowPrometheus alerting rules on Kubernetes: the PrometheusRule CRD, the alert configuration, the Alertmanager integration, the routing, and the production patterns for the alerting rules.→
- Long-term storage — Thanos for PrometheusThanos for long-term Prometheus storage: the architecture, the sidecar, the store, the query, the bucket, and the production patterns for long-term storage.→
Part LXXXIX
Kubernetes Logging
6 checks
- Container stdout to logs — the logging pipelineKubernetes container logging: the stdout/stderr path, the kubelet log files, the structured logs, the JSON formatters, and the integration with the log collection.→
- Node log paths — the files on the hostKubernetes node log paths: /var/log/pods, /var/log/containers, the kubelet logs, the container runtime logs, and the production patterns for log rotation.→
- Loki and Promtail — the log collection and storageLoki and Promtail for cluster logging: the deployment, the Promtail configuration, the Loki storage, the LogQL queries, and the cross-reference to the Observability course.→
- Fluentd and Fluent Bit — the alternative log collectorFluentd and Fluent Bit as alternatives to Promtail: the architecture, the deployment, the configuration, the parsers, and the production patterns for log collection.→
- Log aggregation — the cluster-wide pipelineKubernetes log aggregation: the cluster-wide pipeline, the multi-tenant considerations, the log retention, the log-based metrics, and the production patterns for the cluster-wide logging.→
- Log-based alerting — the queries and the alertsLog-based alerting in Loki: the LogQL queries, the Loki ruler, the Alertmanager integration, the alert routing, and the production patterns for log-based alerts.→
Part XC
Distributed Tracing
6 checks
- Distributed tracing — the request journeyDistributed tracing in Kubernetes: the trace, the span, the context propagation, the OpenTelemetry SDK, the trace backend (Jaeger, Tempo), and the production patterns.→
- OpenTelemetry — the SDK and the CollectorOpenTelemetry in Kubernetes: the SDK, the Collector, the auto-instrumentation, the exporters, and the production patterns for the OpenTelemetry integration.→
- Trace context propagation — the W3C standardW3C Trace Context propagation in Kubernetes: the traceparent header, the tracestate header, the propagation across services, the sampling flags, and the production patterns.→
- Jaeger and Tempo — the trace backendsJaeger and Tempo as trace backends: the architecture, the deployment, the storage, the query, and the production patterns for the trace backends.→
- Trace instrumentation — manual and auto-instrumentationTrace instrumentation in Kubernetes: the manual instrumentation, the auto-instrumentation, the language-specific libraries, the propagation, and the production patterns.→
- Tracing + metrics + logs — the three signals integratedIntegrating tracing, metrics, and logs: the three signals, the correlation via trace IDs, the OTel Collector as the central pipeline, and the production patterns for the integration.→
Part XCI
Kubernetes Events
6 checks
- Kubernetes events — the 1-hour TTL and the event-recorderKubernetes Events: the 1-hour default TTL, the event recording, the events API, the rate limiting, and the integration with the event-exporter.→
- Event types — Normal, Warning, and the involved objectsKubernetes event types: Normal, Warning, the involved objects, the reasons, the messages, and the production patterns for the event types.→
- Event recorder — emitting events from controllersThe client-go event recorder: the recorder API, the event broadcaster, the broadcaster sinks, the integration with controllers, and the production patterns.→
- Event exporter — the long-term storage for eventsevent-exporter: shipping Kubernetes events to long-term storage, the deployment, the sinks (Loki, Elasticsearch, S3, webhook), the configuration, and the production patterns.→
- Event rate limiting — the throttle on the event floodKubernetes event rate limiting: the eventratelimit admission controller, the Server type, the per-namespace limits, the configuration, and the production patterns.→
- Event retention — the long-term storage for eventsKubernetes event retention: the 1-hour TTL, the event-exporter, the long-term storage destinations, the retention policies, and the production patterns.→
Part XCII
Alerting
6 checks
- Alertmanager — the alert routing engineAlertmanager in Kubernetes: the architecture, the routing, the receivers, the inhibition, the silences, and the production patterns for the Alertmanager.→
- Alert rules — Prometheus alerting rules for productionPrometheus alerting rules for production: the rule structure, the severity, the SLO-driven alerts, the burn rate alerts, the runbook integration, and the production patterns.→
- Inhibition and silences — the alert noise reductionAlertmanager inhibition and silences: the conditional suppression, the temporary suppression, the production patterns, and the failure modes of the noise reduction.→
- Alert routing — the routing trees and the receiversAlertmanager routing trees: the conditional routing, the receivers, the matchers, the continues, the production patterns, and the failure modes of the routing.→
- Alert receivers — Slack, PagerDuty, and webhookAlertmanager receivers: Slack, PagerDuty, email, webhook, the configuration, the templates, the production patterns, and the failure modes of the receivers.→
- Runbook links — the alert to the documentationRunbook URLs in alerting: the alert-runbook link, the runbook content, the on-call integration, the production patterns, and the failure modes of the runbook integration.→
Part XCIII
Monitoring the Monitoring
6 checks
- Observability resilience — monitoring the monitoringObservability resilience: the monitoring the monitoring, the self-monitoring, the metric reliability, the cardinality control, the alerting on observability, and the HA patterns.→
- Self-monitoring — Prometheus monitors itselfPrometheus self-monitoring: the up metric, the memory usage, the scrape duration, the evaluation duration, the active series, and the production patterns for the self-monitoring.→
- Metric reliability — the metric qualityMetric reliability: the metric quality, the consistency, the accuracy, the freshness, the label design, and the production patterns for the metric reliability.→
- Cardinality control — the high-cardinality explosion preventionCardinality control in Prometheus: the high-cardinality explosion, the metric_relabel_configs, the cardinality budget, the recording rules, and the production patterns for the cardinality control.→
- Alert on observability — the self-monitoring alertsAlerting on observability: the Prometheus alerts, the Grafana alerts, the Loki alerts, the Alertmanager alerts, the production patterns, and the failure modes of the alert on observability.→
- Observability HA — the high availability of the observability stackObservability HA: the Prometheus HA, the Grafana HA, the Loki HA, the Alertmanager HA, the OTel Collector HA, and the production patterns for the observability HA.→
Part XCIV
Audit Logging
6 checks
- Audit policy stages — RequestReceived, ResponseStarted, ResponseComplete, PanicKubernetes audit policy stages: the four stages, the audit events, the request and response metadata, the policy configuration, and the production patterns.→
- Audit log backends — log file and webhookKubernetes audit log backends: the log file, the webhook, the dynamic backends, the volume management, the rotation, and the production patterns for the audit log.→
- Audit log volume management — the rotation and archivalAudit log volume management: the rotation, the archival, the long-term storage, the retention, the disk usage, and the production patterns for the audit log.→
- Audit policy in production — the security and compliance patternsAudit policy in production: the security patterns, the compliance patterns, the per-resource rules, the secret auditing, the production guidelines, and the failure modes.→
- Audit log in production — the security investigation and complianceAudit log in production: the security investigation, the compliance audit, the SIEM integration, the retention, the GDPR, the SOC 2, and the production patterns.→
- Audit log debugging — the troubleshooting patternsAudit log debugging: the common issues, the troubleshooting patterns, the audit policy validation, the volume issues, the performance issues, and the production patterns.→
Part XCV
Backup Strategy
6 checks
- Backup strategy — what to protectKubernetes backup strategy: what to protect (cluster state, manifests in Git, etcd, persistent data, secrets, certificates, external dependencies). The 3-2-1 backup rule, the recovery time, and the production patterns.→
- etcd backup — the cluster state protectionetcd backup strategy: the snapshot, the storage, the encryption, the schedule, the verification, the recovery, and the production patterns for the etcd backup.→
- Persistent data backup — the volumesKubernetes persistent data backup: the CSI snapshots, the Velero backup, the cloud-provider snapshots, the Restic integration, the encryption, and the production patterns.→
- Manifest backup — Git as the source of truthKubernetes manifest backup: Git as the source of truth, the GitOps workflow, the manifest versioning, the rollback, the disaster recovery, and the production patterns.→
- Secret backup — the encrypted credentialsKubernetes secret backup: the encryption at rest, the secret management, the backup encrypted, the rotation, the SOPS, the external secret managers, and the production patterns.→
- Disaster recovery plan — the cluster reconstitutionKubernetes disaster recovery plan: the RTO/RPO, the recovery procedures, the cluster reconstitution, the regional failover, the backup verification, and the production patterns.→
Part XCVI
Workload Backup
6 checks
- Workload backup principles — etcd is not a backupWorkload backup principles in Kubernetes: why etcd is cluster state not workload data, what needs backing up (manifests, persistent data, application state, secrets, certificates), the 3-2-1 rule, and the operational discipline of treating backup as a first-class production concern.→
- Application-consistent backups — database quiescence and orderingApplication-consistent backups in Kubernetes: why crash-consistent snapshots are insufficient for databases, the quiesce pattern (PRE and POST hooks), the ordering problem between snapshot and quiesce, and the operational discipline of treating application state as a first-class backup concern.→
- CSI volume snapshots — the API and lifecycleCSI volume snapshots in Kubernetes: the VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass objects, the snapshot lifecycle, the relationship to PVCs and PVs, the deletion ordering, and the operational discipline of treating CSI snapshots as a primary backup mechanism.→
- The snapshot data flow — controllers, sidecars, and the CSI RPC chainThe CSI snapshot data flow: the snapshot controller, the external-snapshotter sidecar, the external-provisioner sidecar, the kubelet plugin path, the CreateSnapshot RPC chain, and the operational visibility (events, logs, metrics) that turns a black box into a debuggable system.→
- PVC, PV, and snapshot relationships — what to back up and in what orderPVC, PV, and VolumeSnapshot relationships in a backup program: the dependency graph from PVC to PV to VolumeSnapshot to backup, the deletion ordering, the restore ordering (StorageClass, CRDs, ConfigMaps, snapshots, PVCs, workloads), and the operational discipline of ordering the restore sequence correctly.→
- Object storage for backups — S3, MinIO, and the durability rulesObject storage as the destination for Kubernetes backups: S3-compatible APIs, MinIO for self-hosted, durability vs availability, encryption at rest and in transit, lifecycle policies, versioning, cross-region replication, and the operational discipline of treating object storage as production-critical infrastructure.→
Part XCVII
Kubernetes Backup Tools
6 checks
- Velero architecture — controllers, plugins, and the data pathVelero architecture: the controller, the server, the plugins (for AWS, Azure, GCP, on-prem), the Restic/Kopia daemon, the backup data path, the restore data path, and the operational visibility that turns Velero from a black box into a debuggable system.→
- Installing and configuring Velero — install paths, credentials, and namespace selectionInstalling Velero: the install command, the credentials Secret, the BSL (BackupStorageLocation), the VSL (VolumeSnapshotLocation), the namespace layout, RBAC, plugin selection, and the operational discipline of treating the install as production infrastructure.→
- Velero Restic vs Kopia vs native CSI snapshots — the consistency trade-offThe Velero volume backup trade-offs: Restic vs Kopia vs native CSI snapshots. Crash-consistent vs application-consistent backups, performance, encryption, deduplication, restore granularity, and the operational decision of which to use when.→
- Velero backup lifecycle — schedules, hooks, and resource selectionThe Velero backup lifecycle: the Backup CRD, the Schedule CRD, the resource selection (namespaces, label selectors, include/exclude), the PRE/POST hooks, the retention, the phases (InProgress, Completed, PartiallyFailed, Failed), and the operational discipline of treating the schedule as production infrastructure.→
- Velero restore — selector logic, namespace mapping, and the ordering trapsVelero restore mechanics: the Restore CRD, resource and namespace mapping, selector logic, the storage location mapping, the ordering of restore phases, common ordering traps, and the operational discipline of treating restore as a runbook-driven procedure.→
- Verify the backup is actually restorable — the principles-first disciplineThe principles-first discipline of backup verification: a backup that has never been restored is not a backup. Quarterly restore tests, automated restore-drills in sandbox clusters, the rclone/restic verify path, checksumming, the gap between backup success and restore success, and the operational discipline of treating verification as a first-class concern.→
Part XCVIII
Disaster Recovery
6 checks
- DR principles — RPO, RTO, and the cost of recoveryDisaster recovery principles: RPO (recovery point objective) and RTO (recovery time objective) as the design constraints, the cost of recovery as a function of automation, the relationship to backup program, the tier model (Tier 0 through Tier 6), and the operational discipline of defining RPO/RTO before designing the recovery architecture.→
- Recovery architecture for control-plane loss — the design choicesRecovery architecture for control-plane loss: etcd restore, control-plane rebuild, restore-in-place vs restore-to-new-cluster, the snapshot vs live etcd trade-off, the quorum-loss scenarios, and the operational discipline of treating control-plane recovery as a primary runbook.→
- Etcd recovery scenarios — snapshot, restore, and the quorum trapEtcd recovery scenarios in detail: snapshot creation with etcdctl, the restore procedure, the data directory layout, the quorum trap (2 of 3, 3 of 5), the runtime configuration (peer-urls, client-urls), and the operational discipline of treating etcd snapshots as production-critical infrastructure.→
- Replica rebuild from manifests — when Git is the backupReplica rebuild from manifests: when the cluster state is lost but Git is intact, the recovery path is to apply manifests from Git and let the controllers reconcile. The strategy, the GitOps prerequisite, the operator-installed add-ons (CNIs, ingress controllers, cert-managers), and the operational discipline of treating Git as the primary source of truth.→
- Cross-cluster restore — restoring a backup into a different clusterCross-cluster restore in Velero: restoring a backup taken in cluster A into a fresh cluster B, the storage location mapping, the namespace mapping, the CSI driver compatibility, the secret and config replication, and the operational discipline of testing cross-cluster restore quarterly.→
- DR testing and game days — the validation cadenceDR testing and game days in Kubernetes: the testing cadence (weekly, monthly, quarterly, annually), the test scenarios (single PVC restore, namespace restore, full cluster restore, cross-cluster restore), game day exercises, the post-test report, and the operational discipline of treating DR testing as production infrastructure.→
Part XCIX
Complete Cluster Loss
6 checks
- Cluster loss — the end-to-end recovery sequenceComplete cluster loss in Kubernetes: the end-to-end recovery sequence from new infrastructure to validated production service. The phases (infrastructure, control plane, networking, storage, cluster state, workers, workloads, persistent data, validation), the dependencies, the parallelism, and the operational discipline of treating the sequence as a runbook.→
- New infrastructure — provisioning nodes from scratchPhase 1 of complete cluster loss recovery: provisioning new infrastructure. The IaC approach (Terraform, Cluster API), the node specifications, the network and storage prerequisites, the time budget, and the operational discipline of treating infrastructure provisioning as the foundation of recovery.→
- Control plane rebuild — kubeadm init, HA topology, and the join sequencePhase 2 of complete cluster loss recovery: rebuilding the control plane with kubeadm init on the first node, joining the additional control-plane nodes for HA, configuring the API server endpoint, and the operational discipline of rehearsing the kubeadm init procedure quarterly.→
- Networking restoration — CNI, CoreDNS, and cluster DNS verificationPhase 3 of complete cluster loss recovery: restoring cluster networking. The CNI install (Calico, Cilium, Flannel), the pod CIDR alignment with kubeadm init, CoreDNS verification, kube-proxy configuration, and the operational discipline of testing networking before workloads are applied.→
- Storage restoration — CSI drivers, StorageClasses, and snapshot recoveryPhase 4 of complete cluster loss recovery: restoring storage. The CSI driver install, the StorageClass definitions, the IAM role configuration, the VolumeSnapshot restoration from Velero or cloud snapshots, and the operational discipline of testing storage before workloads depend on it.→
- Cluster state, workers, workloads, persistent data, and validation — phases 5-9Phases 5-9 of complete cluster loss recovery: cluster state (CRDs, add-ons, secrets), workers joining, workloads applied, persistent data restored from Velero, and end-to-end validation. The dependencies, the validation cadence, and the operational discipline of treating the final phases as the proof of recovery.→
Part C
Multi-Cluster Concepts
6 checks
- Multi-cluster concepts — when one cluster is not enoughMulti-cluster concepts in Kubernetes: when one cluster is not enough (environment isolation, blast radius, geography, compliance, scale), the topology patterns (hub-and-spoke, fleet, primary-secondary, active-active), the trade-offs, and the operational discipline of defining why multi-cluster before choosing the topology.→
- Cluster API — declarative Kubernetes cluster lifecycleCluster API (CAPI) for declarative Kubernetes cluster lifecycle: the CAPI control plane, the providers (AWS, Azure, GCP, vSphere, bare metal), the Cluster and Machine CRDs, the bootstrap and infrastructure providers, the GitOps integration, and the operational discipline of treating CAPI as production infrastructure.→
- Rancher — fleet management with downstream clustersRancher for Kubernetes fleet management: the architecture (Rancher server, downstream clusters, clusters API), the import workflow (custom, EKS/AKS/GKE, K3s, RKE2), the project and namespace model, the access control, the GitOps integration (Fleet), and the operational discipline of treating Rancher as production infrastructure.→
- Tanzu — VMware fleet management with the Tanzu Kubernetes PlatformTanzu for Kubernetes fleet management: the Tanzu Kubernetes Platform (TKG), the Tanzu Mission Control (TMC), the cluster kinds (management, workload, standalone), the GitOps integration, the policy model, and the operational discipline of treating Tanzu as production infrastructure.→
- Multi-cluster service mesh and federation — connectivity across clustersMulti-cluster service connectivity: Submariner for L3 cross-cluster Pod-to-Pod, Skupper for L7 application-level bridging, service mesh federation (Istio multi-primary, Linkerd multi-cluster), KubeFed (deprecated), and the operational discipline of treating cross-cluster connectivity as production networking.→
- Multi-cluster anti-patterns — the most common mistakesMulti-cluster anti-patterns in Kubernetes: clusters as tenants, cluster-per-app, premature active-active, inconsistent policies across clusters, missing cross-cluster networking, missing cross-cluster observability, and the operational discipline of avoiding these anti-patterns.→
Part CI
Cluster Boundaries
6 checks
- Cluster boundaries — why separate clustersCluster boundaries in Kubernetes: why separate clusters exist (environment, security, geography, failure domain, compliance, scale), the boundary types, the trade-offs, and the operational discipline of defining boundaries before designing the topology.→
- Environment isolation — dev, staging, prod, and the promotion pipelineEnvironment isolation in Kubernetes: dev, staging, prod as the standard environments, the promotion pipeline (Git → CI → dev → staging → prod), the configuration differences, the data differences (synthetic vs production), the access differences, and the operational discipline of treating each environment as a separate boundary.→
- Geography and compliance — region, residency, and the regulatory boundariesGeography and compliance in Kubernetes clusters: regional clusters for latency, data residency for GDPR/Federal, sovereignty for classified workloads, the regulatory mapping (PCI, HIPAA, GDPR, FedRAMP), the operational trade-offs, and the operational discipline of aligning cluster boundaries with regulatory boundaries.→
- Failure domain and blast radius — partitioning risk across clustersFailure domain and blast radius in Kubernetes clusters: how a regional outage or control-plane failure affects workloads, the multi-cluster partitioning of blast radius, the trade-offs of cluster-per-workload-type, the relationship to RTO/RPO, and the operational discipline of designing for failure domains before designing for scale.→
- Cluster-per-team vs cluster-per-app vs hybrid — choosing the right granularityCluster granularity choices: cluster-per-team, cluster-per-app, cluster-per-environment, and the hybrid approach. The trade-offs (operational cost, isolation, shared capacity), the decision framework (team size, app count, compliance, scale), and the operational discipline of reviewing cluster granularity quarterly.→
- Federation vs independent clusters — when to federateFederation vs independent clusters in Kubernetes: federation (single API across clusters, KubeFed deprecated), independent clusters (each managed separately), the trade-offs, when federation makes sense, the modern alternatives (Submariner, Istio multi-cluster, Argo CD), and the operational discipline of choosing the right architecture.→
Part CII
Managed vs Self-Managed Kubernetes
6 checks
- Managed vs self-managed Kubernetes — the fundamental trade-offManaged vs self-managed Kubernetes: the responsibility split (control plane, etcd, upgrades, patching, networking, storage), the trade-offs (operational burden vs control, cost vs flexibility, vendor lock-in), the hybrid approach (managed control plane + self-managed workloads), and the operational discipline of choosing based on team capacity and requirements.→
- kubeadm vs EKS — the operational comparisonkubeadm vs EKS in detail: the responsibility split (control plane, etcd, upgrades, networking), the cost comparison (per-cluster fee vs infrastructure cost), the lock-in assessment, the operational trade-offs, and the operational discipline of choosing based on team capacity.→
- AKS, GKE, OKE — the other managed Kubernetes providersAKS, GKE, and OKE in detail: the responsibility split for each, the cost comparison, the lock-in assessment (Azure-specific, GCP-specific, OCI-specific integrations), the operational trade-offs, and the operational discipline of choosing based on cloud commitment and lock-in tolerance.→
- Control plane HA — responsibility and architectureControl plane HA in managed vs self-managed Kubernetes: the responsibility split, the HA architecture (3 control-plane nodes, 5 for larger clusters), the failure modes (1 of 3 dead, 2 of 3 dead), the etcd quorum, and the operational discipline of testing control plane HA quarterly.→
- Upgrade responsibilities — who upgrades what and whenUpgrade responsibilities in managed vs self-managed Kubernetes: the control plane upgrade (managed vs kubeadm), the worker node upgrade (managed node groups vs self-managed), the version skew rules, the upgrade sequence (control plane first, workers second), the operational trade-offs, and the discipline of testing upgrades in staging before production.→
- Cost and operational trade-offs — total cost of ownershipCost and operational trade-offs in managed vs self-managed Kubernetes: the per-cluster fees, the operational burden cost, the lock-in cost, the team capacity required, the total cost of ownership calculation, and the operational discipline of modelling TCO before choosing.→
Part CIII
GitOps Introduction
6 checks
- GitOps principles — Git as the source of truthGitOps principles: Git as the source of truth, declarative desired state, controllers that reconcile actual to desired, the four principles (declarative, versioned, automatically pulled, continuously reconciled), the benefits, and the operational discipline of treating Git as the canonical cluster definition.→
- Desired state in Git — the manifest repository structureDesired state in Git: the manifest repository structure (environments, apps, base/overlay), the directory layout (clusters/, apps/, infrastructure/), the manifest format (raw YAML, Kustomize, Helm), the Git workflow (PR, review, merge), the branch strategy, and the operational discipline of treating the manifest repository as production infrastructure.→
- Reconciliation controllers — Argo CD and Flux in depthReconciliation controllers in GitOps: Argo CD (architecture, Application CR, sync waves, App of Apps), Flux (GitRepository, Kustomization, HelmRelease, Source Controller), the comparison, the operational trade-offs, and the operational discipline of treating the controller as production infrastructure.→
- GitOps vs imperative — the operational comparisonGitOps vs imperative cluster management: the push model (CI runs kubectl apply), the pull model (controller in cluster pulls from Git), the trade-offs (security, audit, drift, recovery), the hybrid model, and the operational discipline of preferring the pull model for production.→
- Drift detection — finding and correcting out-of-band changesDrift detection in GitOps: what drift is (cluster state != Git state), how the controller detects it (compare, report, optionally correct), the difference between desired and unexpected drift, the alerting and remediation, and the operational discipline of treating drift as a signal of process failure.→
- GitOps anti-patterns — the most common mistakesGitOps anti-patterns: directly pushing to main, storing Secrets in Git, missing sync waves for CRD ordering, selfHeal disabled in production, no ignoreDifferences for HPA, monolithic Application per cluster, and the operational discipline of avoiding these anti-patterns.→
Part CIV
Helm
6 checks
- Helm chart structure — Chart.yaml, values.yaml, templates, and helpersHelm chart structure: Chart.yaml (metadata, version, dependencies), values.yaml (default values), templates/ (Kubernetes manifests with Go templating), helpers (named templates), the chart layout, the rendering process, and the operational discipline of understanding chart structure before installing.→
- Helm values and templating — Go template patterns in productionHelm values and templating: values.yaml structure, Go template patterns (conditionals, loops, functions, pipelines), values merging, the --set vs -f trade-off, value validation via schema, and the operational discipline of treating values files as production configuration.→
- Helm releases and revisions — the lifecycle of an installed chartHelm releases and revisions: a release is an installed instance of a chart with a name and a history; a revision is one version of a release; the release state stored in Secrets; helm list, helm history, helm status, helm rollback, the operational trade-offs, and the discipline of treating releases as production objects.→
- Helm install, upgrade, rollback — the operational lifecycleHelm install, upgrade, and rollback in production: the lifecycle commands, the --atomic and --wait flags, the cleanup-on-fail behaviour, the dry-run via --dry-run, the CI/CD integration (helm diff, helmfile, Argo CD Helm), the operational trade-offs, and the discipline of testing every chart change in staging before production.→
- Helm repositories and OCI registries — the chart distribution modelHelm repositories and OCI registries: traditional chart repositories (index.yaml, HTTP servers, helm repo add), OCI registries (Helm 3.8+, Harbor, GHCR, ECR), the migration path, the operational trade-offs (chart signing, authentication, version discovery), and the operational discipline of pinning and signing charts in production.→
- Helm production discipline — safety, review, and the discipline of pinningHelm production discipline: pinning versions, signing charts, dry-run before apply, --atomic and --wait, helm diff in PR review, helmfile for multi-release management, Argo CD Helm integration, the operational safety nets, and the discipline of treating Helm as production deployment infrastructure.→
Part CV
Kustomize
6 checks
- Kustomize overview — declarative customisation without templatesKustomize overview: declarative customisation without templates (built into kubectl), the base + overlay pattern, the kustomization.yaml structure, the comparison with Helm (no templating, no Tiller), the use cases, and the operational discipline of understanding Kustomize before adopting it.→
- Base + overlay — the canonical Kustomize patternBase + overlay in Kustomize: the canonical pattern for environment-specific configuration. The base directory holds the common manifests; overlays per environment reference the base and apply patches. The kustomization.yaml structure (resources, patches, commonLabels, namespace, namePrefix), the inheritance rules, and the operational discipline.→
- kustomization.yaml in depth — fields, transformers, and generatorskustomization.yaml fields in depth: resources, patches, commonLabels, namespace, namePrefix, images (image transformer), replicas, components, configMapGenerator, secretGenerator, the transformer types (annotations, labels, images, replicas), the generator types (configMapGenerator, secretGenerator), and the operational discipline.→
- Kustomize patches — strategic merge vs JSON 6902 vs images transformerKustomize patches in detail: strategic merge patches (Kubernetes-native, merge by field), JSON 6902 patches (precise, path-based), the images transformer (rewrite image names/tags), when to use each, the operational trade-offs, and the discipline of choosing the right patch type for the task.→
- Kustomize vs Helm — when to use whichKustomize vs Helm comparison: when to use which tool. Kustomize for application manifests with environment overlays, no templating, built into kubectl. Helm for third-party packages with complex templating, version history, rollback. The hybrid approach (Helm for platform, Kustomize for applications), the trade-offs, and the operational discipline.→
- Kustomize production discipline — render before apply, review, pin, signKustomize production discipline: render before apply (kubectl kustomize), review the rendered output in PRs, pin image tags, sign the configuration, the operational safety nets (kubectl diff, Argo CD diff), the anti-patterns (untested overlays, commonLabels breaking selectors, no --enable-helm), and the discipline.→
Part CVI
Package Management Anti-Patterns
6 checks
- Package management anti-patterns — the most common mistakesPackage management anti-patterns in Kubernetes: giant values files, unpinned charts, blindly installing public charts, configuration drift, multiple Helm releases for the same workload, secrets in values files, and the operational discipline of avoiding these anti-patterns.→
- Giant values files — splitting for reviewabilityGiant values files anti-pattern: why they are a problem (PR review impossible, hidden coupling, change scope unclear), the fix (split by concern: image, service, resources, ingress, autoscaling, secrets reference), the helmfile approach for multi-release, the Kustomize alternative, and the operational discipline of small focused files.→
- Unpinned charts — the floating tag trapUnpinned charts anti-pattern: why floating tags are dangerous (chart upgraded between staging and production, unexpected breaking changes, supply chain risk), the fix (always --version, never latest, use OCI digests for immutability), the lock file approach (helm-secrets, helm-diff, helmfile state), and the operational discipline.→
- Blindly installing public charts — the supply chain riskBlindly installing public charts anti-pattern: why blindly installing is a supply chain risk (privileged workloads, broad RBAC, backdoors), the fix (review every chart, render with helm template, inspect the manifests, verify signature), the trusted repositories, the organisational policy, and the operational discipline.→
- Configuration drift — detecting and correcting divergenceConfiguration drift anti-pattern: the divergence between Helm release state and cluster actual state, the sources (kubectl edit, kubectl apply from laptop, CI pipeline bypassing Helm), the detection (helm diff, Argo CD selfHeal, drift alerts), the correction (helm upgrade with the correct values), and the operational discipline.→
- Helm + Kustomize together — combining the tools safelyHelm + Kustomize together: the hybrid pattern (Helm for platform packages, Kustomize for application overlays), the integration via Argo CD, the boundaries (which tool owns which resource), the conflicts to avoid (overlapping patches), the operational trade-offs, and the discipline.→
Part CVII
Namespaces and Multi-Tenancy
6 checks
- Multi-tenancy models — soft vs hard, namespaces vs clustersMulti-tenancy models in Kubernetes: soft multi-tenancy (namespaces with RBAC, NetworkPolicy, ResourceQuota, PSS — single cluster, shared control plane), hard multi-tenancy (separate clusters, separate control planes), the trade-offs (isolation vs cost), the use cases, and the operational discipline of choosing the model based on the driver.→
- Namespace as tenancy boundary — what it isolates and what it does notNamespaces as the multi-tenancy boundary in Kubernetes: what they isolate (resource names, RBAC, NetworkPolicy selectors, ResourceQuota, PSS, DNS suffix) and what they do not (network without NetworkPolicy, node failure, resource pressure, storage backend, etcd blast radius). The layered controls required for safe multi-tenancy, and the operational discipline.→
- Soft multi-tenancy with RBAC, NetworkPolicy, and Quotas — the production stackSoft multi-tenancy production stack: RBAC (Roles, RoleBindings, ClusterRoles), NetworkPolicy (default-deny + explicit allow, ingress and egress), ResourceQuota (compute, storage, object counts), PSS (privileged, baseline, restricted), the integration, the anti-patterns, and the operational discipline.→
- vCluster and Kyverno — hard multi-tenancy and per-tenant policyvCluster for hard multi-tenancy within a cluster (virtual control plane per tenant), Kyverno for per-tenant policy enforcement (admission control, mutation, generation), the integration with namespaces, the operational trade-offs, and the operational discipline.→
- Tenant onboarding and isolation testing — the operational disciplineTenant onboarding and isolation testing in Kubernetes multi-tenancy: the onboarding checklist (namespace, RBAC, NetworkPolicy, ResourceQuota, PSS, secrets), the isolation tests (cross-namespace network, cross-namespace resource access, RBAC denial, ResourceQuota enforcement, PSS enforcement), the audit cadence, and the operational discipline.→
- Multi-tenancy limits and pitfalls — what namespaces cannot doMulti-tenancy limits in Kubernetes: what namespaces cannot do (no network isolation without NetworkPolicy, no resource isolation without Quota, no kernel isolation, no workload identity isolation without ServiceAccount per tenant), the failure modes (escaped Pods, noisy neighbours, etcd blast radius, missing policies), the verification cadence, and the operational discipline.→
Part CVIII
ResourceQuota
6 checks
- ResourceQuota — the namespace resource budgetResourceQuota in Kubernetes: the per-namespace budget for compute (CPU, memory), storage, object counts, extended resources, the enforcement model (admission control, rejected when exceeded), the relationship to LimitRange, the scoping (PriorityClass, StorageClass), and the operational discipline.→
- Compute quotas — CPU and memory budgets per namespaceCompute quotas in Kubernetes: requests.cpu, requests.memory, limits.cpu, limits.memory, the sum across all Pods in a namespace, the relationship to Pod requests and limits, the overcommit strategy (requests vs limits), the observability, and the operational discipline.→
- Storage and object count quotas — bounding PVCs, secrets, and cluster objectsStorage and object count quotas in Kubernetes: requests.storage, persistentvolumeclaims, ephemeral-storage; object counts (pods, services, secrets, configmaps, jobs, cronjobs, deployments, statefulsets); the enforcement model; the operational trade-offs; and the discipline.→
- Quota scoping — PriorityClass and StorageClass-based quotasQuota scoping in Kubernetes: PriorityClass-scoped quotas (only count Pods with the specified PriorityClass), StorageClass-scoped quotas (only count PVCs with the specified StorageClass), the use cases (different priorities per workload, different storage tiers), the operational trade-offs, and the discipline.→
- Quota enforcement lifecycle — admission, observation, and the failure modesQuota enforcement lifecycle in Kubernetes: admission control (Pod rejected when quota exceeded), observation (kubectl describe, Prometheus metrics), the failure modes (rollout failures, HPA scaling failures, debug Pods blocked, cluster upgrade issues), the operational response, and the discipline.→
- Quota operations and anti-patterns — the operational disciplineQuota operations and anti-patterns in Kubernetes: setting quotas deliberately (start with observed usage, add headroom), the anti-patterns (no quota on kube-system, oversized workload, frequent quota adjustments, no LimitRange), the operational response, the cadence, and the discipline.→
Part CIX
LimitRange
6 checks
- LimitRange overview — defaults and constraints per container and PodLimitRange in Kubernetes: per-container defaults, max/min constraints, per-Pod constraints, per-PVC constraints; the relationship to ResourceQuota (defaults make the quota meaningful); the four types (Container, Pod, PersistentVolumeClaim, PersistentVolume); and the operational discipline.→
- Container defaults — the foundation of resource accountingContainer defaults in LimitRange: default (limits applied when not specified), defaultRequest (requests applied when not specified), the admission behaviour, the overcommit strategy (default vs defaultRequest ratio), the operational impact, and the discipline.→
- Min and max constraints — bounding what containers can requestMin and max constraints in LimitRange: max (highest resource a container can request), min (lowest resource a container must request), the admission behaviour, the maxLimitRequestRatio (preventing extreme overcommit), the operational impact, and the discipline.→
- Pod-level limits — bounding the sum across containersPod-level limits in LimitRange: max (sum of all containers in a Pod cannot exceed), min (sum must be at least), the use case (multi-container Pods, sidecars, init containers), the interaction with container-level limits, the operational trade-offs, and the discipline.→
- PVC constraints — bounding storage requestsPVC constraints in LimitRange: max (maximum storage request), min (minimum storage request), the use case (preventing oversized PVCs), the interaction with StorageClass quotas, the operational trade-offs, and the discipline.→
- LimitRange anti-patterns — the operational disciplineLimitRange anti-patterns in Kubernetes: no LimitRange (ResourceQuota meaningless), max too tight (rollouts fail), max too loose (no protection), min too high (small workloads rejected), no PVC max (oversized PVCs allowed), maxLimitRequestRatio extreme (CPU throttling), and the operational discipline.→
Part CX
Priority and Preemption
6 checks
- PriorityClass — workload priority in KubernetesPriorityClass in Kubernetes: the priority value (higher = more important), the globalDefault flag, the preemption policy (Never, PreemptLowerPriority), system-cluster-critical and system-node-critical, the use cases (production vs batch, critical vs non-critical), and the operational discipline.→
- System critical priority classes — protecting kube-systemSystem critical priority classes in Kubernetes: system-cluster-critical (1,000,000,000) for cluster-wide critical Pods (CoreDNS, kube-proxy), system-node-critical (1,000,000,001) for node-critical Pods (CNI agents, monitoring), the automatic installation, the use case, the protection from preemption, and the operational discipline.→
- Preemption mechanics — how the scheduler evicts lower-priority PodsPreemption mechanics in Kubernetes: when the scheduler preempts (cluster under pressure, higher priority Pod pending), the n+1 search (find a node where the higher-priority Pod fits), the PodList ordering (nominate candidates for removal), graceful termination (PDB-aware, terminationGracePeriodSeconds), the failure modes, and the operational discipline.→
- Scheduler integration — how priority affects scheduling decisionsScheduler integration with priority: the scheduling queue (activeQ, backoffQ, unschedulableQ), priority sorting (higher priority first), the preemption hook, the n+1 search, the failure modes (no preemption, infinite loop), and the operational discipline.→
- PDB interaction with preemption — budgets that block preemptionPod Disruption Budgets (PDBs) interaction with preemption: PDBs limit voluntary disruptions; preemption is voluntary; PDBs can block preemption. The failure mode (PDB too restrictive → no eviction candidates → higher-priority Pod Pending), the configuration (minAvailable, maxUnavailable), the operational discipline (configure PDBs with headroom), and the verification.→
- Priority anti-patterns — the most common mistakesPriority anti-patterns in Kubernetes: application PriorityClasses using values above 1 billion, no globalDefault, batch workloads with PreemptLowerPriority, PDBs with minAvailable: 100%, no system classes for critical components, priority based on team not workload, and the operational discipline.→
Part CXI
Kubernetes Networking Advanced Topics
6 checks
- eBPF dataplanes and Cilium — modern Kubernetes networkingeBPF dataplanes in Kubernetes: Cilium as the leading eBPF-based CNI; the architecture (eBPF programs for networking, Envoy for L7, Hubble for observability); the comparison with iptables/IPVS (faster, more observable, identity-based); the operational trade-offs; and the discipline.→
- Gateway API — the modern Ingress replacementGateway API in Kubernetes: the modern replacement for Ingress; the resources (GatewayClass, Gateway, HTTPRoute, TCPRoute, TLSRoute); the advantages over Ingress (role-oriented, portable, expressive); the implementation (Cilium, Istio, NGINX Gateway Fabric); the operational trade-offs; and the discipline.→
- BGP service advertisement — LoadBalancer IPs and BGP peersBGP service advertisement in Kubernetes: LoadBalancer Services and BGP peers (MetalLB, Cilium BGP); the use cases (bare metal, on-prem, multi-cluster), the configuration (IP pools, peers, BGP communities), the operational trade-offs, and the discipline.→
- Advanced policy — L7, FQDN, and DNS-based controlsAdvanced NetworkPolicy in Kubernetes: L7 policies (HTTP, gRPC, Kafka path/method-based), FQDN-based egress (allow specific external domains), DNS-based controls (egress to specific resolvers), Cilium's CiliumNetworkPolicy with L7 rules, the operational trade-offs, and the discipline.→
- Dual-stack and IPv6 — modern IP addressing in KubernetesDual-stack and IPv6 in Kubernetes: dual-stack networking (IPv4 + IPv6 simultaneously), the configuration (kubeadm init, CNI support, Service API), the migration path, IPv6-only clusters, the operational trade-offs, and the discipline.→
- Network anti-patterns — the most common mistakesNetwork anti-patterns in Kubernetes: default-allow NetworkPolicy (every Pod talks to every Pod), Cilium without Hubble (no L7 visibility), single CNI choice without testing, no BGP on bare metal, no IPv6 planning, custom CNI scripts, and the operational discipline.→
Part CXII
Load Balancing on Bare Metal
6 checks
- Bare metal LB problem — why cloud-provider LBs do not exist on-premThe bare metal load balancing problem in Kubernetes: cloud-provider load balancers (AWS ELB, GCP LB, Azure LB) do not exist on bare metal or on-prem clusters. The alternatives (NodePort, externalIPs, BGP with MetalLB, Cilium BGP, F5 BIG-IP, HAProxy), the trade-offs, the design choices, and the operational discipline.→
- MetalLB L2 mode — ARP/NDP-based load balancingMetalLB L2 mode in detail: how L2 mode uses ARP/NDP to advertise Service IPs; the configuration (IPAddressPool, L2Advertisement, IP allocation); the use cases (small clusters, no router config); the limitations (single-node bottleneck, slow failover); the operational discipline.→
- MetalLB BGP mode — multi-node load balancing with ECMPMetalLB BGP mode in detail: how BGP mode advertises Service IPs via BGP; the configuration (IPAddressPool, BGPPeer, BGPAdvertisement, communities); the use cases (production on-prem, multi-node); the benefits (multi-node, ECMP, fast failover with BFD); the operational discipline.→
- Integration with VyOS and BIRD — router-side BGP configurationIntegration of MetalLB BGP with VyOS and BIRD routers: router-side BGP configuration (peer, ASN, prefix-list, route-map, communities), BFD configuration, the use case (VyOS for the cluster router, BIRD for Linux-based routers), the operational trade-offs, and the discipline.→
- BGP peer configuration — IP pools, peers, and communitiesBGP peer configuration in MetalLB: IPAddressPool (the range of IPs), BGPPeer (router IP, ASN, timers, BFD), BGPAdvertisement (which pools to advertise to which peers), BGP communities (tags for routing policy), the use cases (multi-pool, multi-peer, multi-tenant), the operational trade-offs, and the discipline.→
- Bare metal LB anti-patterns — the most common mistakesBare metal LB anti-patterns in Kubernetes: LoadBalancer Service without a controller, single-node LB without failover plan, no BGP communities for routing policy, no BFD for fast failover, mixed L2 and BGP for the same IP pool, no monitoring on BGP sessions, and the operational discipline.→
Part CXIII
Kubernetes DNS Advanced Troubleshooting
6 checks
- CoreDNS advanced configuration — Corefile, plugins, and tuningCoreDNS advanced configuration: the Corefile format, plugins (cache, forward, log, errors, prometheus, ready, loop, reload, kubernetes), tuning (cache TTL, forward upstreams, ready endpoint), the operational trade-offs, and the discipline.→
- ndots and search paths — the DNS resolution chainndots and search paths in Kubernetes DNS: the ndots:5 default (a name with fewer than five dots is tried against the search domains first), the search path (namespace.svc.cluster.local, svc.cluster.local, cluster.local), the latency implications, nodelocaldns mitigation, the operational discipline, and tuning.→
- DNS latency and nodelocaldns — caching at the node levelDNS latency in Kubernetes: sources (search path overhead, upstream latency, CoreDNS load), measurement (query latency histograms), mitigation with nodelocaldns (caching at node level), the configuration, the observability, and the operational discipline.→
- Stub domains and upstream resolvers — overriding DNS for specific zonesStub domains and upstream resolvers in Kubernetes: stub domains (override DNS for specific zones like .corp or .internal), upstream resolvers (configure CoreDNS to forward to specific upstreams per zone), the use cases (corporate DNS integration, conditional forwarding), the operational trade-offs, and the discipline.→
- DNS security — DoH, response policy zones, and DNSSECDNS security in Kubernetes: DNS-over-HTTPS (DoH) for encrypted queries, response policy zones (RPZ) for blocking malicious domains, DNSSEC for verifying upstream responses, the operational trade-offs, and the discipline.→
- DNS troubleshooting flow — the diagnostic methodologyDNS troubleshooting in Kubernetes: the diagnostic flow (verify Pod can reach CoreDNS, verify CoreDNS resolves cluster names, verify upstream resolution, verify nodelocaldns, verify stub domains), the common failures (Corefile errors, upstream failures, ndots misconfig, stub domain leaks), the tools (dig, nslookup, kubectl exec), and the discipline.→
Part CXIV
Certificate and TLS Operations
6 checks
- cert-manager — Kubernetes-native certificate managementcert-manager in Kubernetes: the certificate management framework (Issuers, ClusterIssuers, Certificates), the ACME integration (Let's Encrypt, ACME servers), the internal CA integration (cert-manager CA injector), Vault PKI integration, the certificate lifecycle (issue, renew, store), and the operational discipline.→
- ACME issuers — Let's Encrypt and the ACME protocolACME issuers in cert-manager: the ACME protocol (Automatic Certificate Management Environment), Let's Encrypt staging vs production, the challenge solvers (http-01, dns-01), wildcard certificates, rate limits, the operational trade-offs, and the discipline.→
- Internal CA — the cert-manager CA injector and self-signed CAsInternal CA in Kubernetes: creating an in-cluster CA (cfssl, easyrsa, openssl), the cert-manager CA injector (cert-manager-csi-driver-spiffe or csi-driver-cert-manager), issuing certificates from the CA via Issuer, the use cases (private services, mTLS, internal PKI), the operational trade-offs, and the discipline.→
- Vault PKI — HashiCorp Vault as the certificate authorityVault PKI as the certificate authority for Kubernetes: enabling the PKI secrets engine in Vault, configuring roles and policies, the cert-manager Vault issuer with Kubernetes auth, the audit trail, the operational trade-offs (Vault operational complexity vs centralised PKI), and the discipline.→
- Application TLS secrets — using TLS material in PodsApplication TLS secrets in Kubernetes: the kubernetes.io/tls Secret type, mounting Secrets as volumes or environment variables, the CSI driver for direct certificate mounting (no Secret), the cert-manager csi-driver-spiffe, the operational trade-offs, and the discipline.→
- Certificate rotation and renewal — the operational disciplineCertificate rotation and renewal in Kubernetes: cert-manager automatic renewal (renewBefore), the renewal flow (check expiry, request new cert, store in Secret, update references), the failure modes (ACME rate limits, CA key compromised, DNS propagation), the rotation strategy (overlap period, canary, application reload), and the discipline.→
Part CXV
Image Registry Operations
6 checks
- Image registry overview — pull, push, cache, and securityImage registries in Kubernetes: how Pods pull images (imagePullPolicy, imagePullSecrets), the registry types (public Docker Hub, private registry, cloud registry), the cache (registry cache proxy, pull-through cache), the security (image signing, vulnerability scanning), and the operational discipline.→
- Private registries — Harbor, Quay, and on-prem controlPrivate container registries: Harbor (CNCF graduated; pull-through cache, vulnerability scanning, image signing with cosign, replication), Quay (Red Hat; similar features, Clair scanning), self-hosted with distribution, the operational trade-offs (storage, replication, signing), and the discipline.→
- imagePullSecrets and credential rotation — authentication to registriesimagePullSecrets in Kubernetes: the Secret types (kubernetes.io/dockerconfigjson for Docker, kubernetes.io/dockercfg for legacy), the Pod spec integration, the credential rotation, the use cases (per-namespace vs cluster-wide, IRSA for cloud), the security considerations, and the discipline.→
- Unavailable registries — graceful degradation and pull-through cacheUnavailable registries in Kubernetes: failure modes (registry down, rate-limited, network partition), the impact on Pods (ImagePullBackOff, ErrImagePull), pull-through cache as the mitigation, the registry fallback configuration, the operational discipline of testing registry outages.→
- Tag vs digest strategy — immutability and reproducibilityTag vs digest strategy in Kubernetes: tags (mutable, human-readable, e.g., :latest, :1.2.3), digests (immutable, SHA256, e.g., :sha256:abc...), the trade-offs (tags are convenient, digests are immutable), the recommendation (use tags for development, digests for production), the operational discipline.→
- Registry cache — pull-through proxy and local kubelet cacheRegistry cache in Kubernetes: pull-through proxy (distribution, Harbor proxy cache), local kubelet image cache, the configuration, the operational trade-offs (storage, sync latency, garbage collection), and the discipline.→
Part CXVI
Maintenance Windows
6 checks
- Maintenance windows — the contract between uptime and changeKubernetes maintenance windows as the risk-governed contract between runtime change and operational uptime: definitions, ownership, prerequisites, what stays out of scope, and the difference between a window and a freeze.→
- Snowflakes and phoenixes — the topology of replaceable infrastructureKubernetes node personality as a spectrum from snowflake to phoenix: configuration drift, golden images, image-based rollouts, and how to design a node fleet so workers are cattle, not pets.→
- Pre-change gates — what must be true before a window opensThe pre-change gate of a Kubernetes maintenance window: etcd snapshot, tested rollback, drained replicas, capacity headroom, communication plan, and the order in which the gate must be satisfied.→
- Cordon and drain orchestration — the eviction choreographyKubernetes drain orchestration: the difference between cordon and drain, eviction API and PDB interaction, the order of node retirement, and the automation patterns that ship with kubeadm, Cluster-API, and Karpenter.→
- Notify, freeze, and rollout windows — the social contract of changeKubernetes change communication: pre-announce, freeze windows, change windows, and rollout windows. The relationship between the cluster, the on-call, the customer, and the change calendar.→
- Post-change validation and rollback gates — closing the window with evidenceKubernetes post-change validation: the evidence a window must produce to close, the validation checks that confirm the change worked, the rollback if it did not, and the post-incident review that pays down the lesson.→
Part CXVII
Change Management
6 checks
- Change management as risk governance — the goal of a changeKubernetes change management as the risk-governed discipline of planning, classifying, reviewing, executing, and verifying a change. The goal is not zero changes; the goal is reversible, well-understood change.→
- CAB, peer review, and four-eyes — the social contract of governanceKubernetes change review governance: the four-eyes principle, the Change Advisory Board (CAB), the peer review for routine changes, and the social contract that makes governance effective.→
- Risk classification and CHG documents — the cost of a changeKubernetes change risk classification: classes 1, 2, and 3, the change document (CHG), the fields that must be filled, and the cost of a change that is misclassified.→
- Pre-prod gates and canary fleets — paying down risk before productionKubernetes pre-production change gates: staging, canary, progressive delivery, blue-green, and the relationship between the gate and the production rollout.→
- Communication and customer notifications — the social contract of changeKubernetes change communication: pre-announce, in-flight, post-change, the customer-facing notification, the internal escalation, and the discipline that makes communication effective.→
- Post-change verification and PIR — the artefact that pays down the lessonKubernetes post-change review: the PIR template, the post-change review (PCR), the metrics that the review surfaces, and the discipline that turns the change into an improvement to the next window.→
Part CXVIII
Kubernetes Troubleshooting Methodology
6 checks
- The 11-step methodology — the canonical troubleshooting workflowThe canonical Kubernetes troubleshooting 11-step methodology: define symptom → determine impact → inspect object → events → logs → dependencies → identify component → hypothesis → test → restore → validate. The discipline that makes panics into procedure.→
- Symptom definition and impact — the first two stepsKubernetes troubleshooting steps 1 and 2: define the symptom and determine the impact. The discipline of writing the symptom in one sentence and the impact in business terms, before any kubectl command.→
- Inspect object → events → logs — the evidence-gathering arcKubernetes troubleshooting steps 3, 4, and 5: inspect the object, read the events, read the logs. The evidence-gathering arc that converts a symptom into a candidate cause.→
- Dependencies and component identification — the network of causeKubernetes troubleshooting steps 6 and 7: dependencies (the network of cause) and component identification (the piece of software that is failing). The discipline that turns evidence into a hypothesis.→
- Hypothesis, test, restore, validate — the closing arcKubernetes troubleshooting steps 8–11: hypothesis, test, restore, validate. The closing arc that converts a candidate cause into a closed incident.→
- The sysadmin troubleshooting posture — habits, tools, and ergonomicsThe operations posture of a Kubernetes sysadmin: the tooling stack, the alias set, the on-call ergonomics, the muscle memory for the 11-step methodology, and the discipline that makes panic into procedure.→
Part CXIX
Pod Troubleshooting
6 checks
- Pending pods — the scheduling diagnostic gridKubernetes Pending pod troubleshooting: the scheduling diagnostic grid, the events that surface the failure, the unschedulable conditions, and the systematic approach to finding the cause.→
- CrashLoopBackOff and ImagePullBackOff — the image and startup failuresKubernetes CrashLoopBackOff and ImagePullBackOff: the diagnose path, the events, the previous logs, and the systematic approach to recovering a workload that cannot start.→
- CreateContainerConfigError and OOMKilled — the misconfiguration and memory failuresKubernetes CreateContainerConfigError and OOMKilled: diagnosing missing ConfigMaps, Secrets, and volume references, and the kubelet's OOM killer terminating the container for exceeding memory limits.→
- Probe failures — readiness, liveness, and startupKubernetes probe failures: readiness, liveness, and startup probes. The diagnostic arc, the failure modes, and the systematic approach to diagnosing a workload that is running but not Ready.→
- Stuck Terminating — the eviction and shutdown diagnosticKubernetes Pod stuck in Terminating: the diagnostic arc, the finalizer, the grace period, the preStop hook, and the systematic approach to recovering a workload that will not shut down.→
- Pod field reference — the canonical `kubectl describe pod` excerptsKubernetes Pod troubleshooting field reference: the canonical `kubectl describe pod` excerpts for Pending, ImagePullBackOff, CreateContainerConfigError, CrashLoopBackOff, OOMKilled, probe failures, and Terminating. The diagnosis cheat sheet.→
Part CXX
Deployment Troubleshooting
6 checks
- Rollout stuck — the progression blockKubernetes Deployment rollout stuck: the progression block, the rollout status, the rollout history, and the systematic approach to diagnosing a Deployment that will not converge.→
- Unavailable replicas and maxUnavailable — the rolling update guardKubernetes Deployment unavailable replicas: the maxUnavailable surge, the rolling update math, the PDB interaction, and the systematic approach to diagnosing a Deployment that has lost replicas.→
- Broken selectors and orphan services — the routing failureKubernetes Deployment broken selector and orphan service: the diagnostic arc, the events that surface the failure, the systematic approach to finding a Service that no longer routes to its Deployment.→
- Readiness failures cascading — the probe and the rolloutKubernetes Deployment readiness failures cascading through a rollout: the probe misconfiguration, the new Pods not becoming Ready, and the systematic approach to diagnosing a Deployment that is stuck on readiness.→
- Bad images and rollout deadlock — the recovery pathKubernetes Deployment bad image and rollout deadlock: the diagnostic, the recovery path, and the systematic approach to recovering a Deployment that cannot roll forward or back.→
- Rollback, undo, and history — the recovery toolkitKubernetes Deployment rollback toolkit: kubectl rollout undo, kubectl rollout history, kubectl rollout pause, kubectl rollout restart, and the systematic approach to recovering a Deployment.→
Part CXXI
Service Troubleshooting
6 checks
- DNS → Service → EndpointSlice → Pod IP → application port — the canonical flowKubernetes Service troubleshooting systematic flow: DNS → Service → EndpointSlice → Pod IP → application port. The five layers of the canonical flow and the diagnostic at each layer.→
- Service has no endpoints — the empty EndpointSliceKubernetes Service has no endpoints: the diagnostic for an empty EndpointSlice, the selector mismatch, the readiness probe, and the systematic approach to recovering a Service that has no routing.→
- ClusterIP unreachable — the kube-proxy and routing failureKubernetes ClusterIP unreachable: the diagnostic for the kube-proxy, the iptables/IPVS rules, the routing, and the systematic approach to recovering a Service that has endpoints but does not route traffic.→
- NodePort and LoadBalancer — the external reachability pathKubernetes Service NodePort and LoadBalancer: the diagnostic for external reachability, the node port, the cloud provider's load balancer, and the systematic approach to recovering Services with external traffic.→
- Headless and ExternalName — the DNS-only ServicesKubernetes headless Services and ExternalName Services: the diagnostic for the DNS-only path, the EndpointSlice publishing the Pod IPs, and the systematic approach to recovering Services that bypass the kube-proxy.→
- kube-proxy and iptables/IPVS — the routing engineKubernetes kube-proxy and iptables/IPVS: the mode, the rules, the reconciliation, and the systematic approach to diagnosing the cluster's routing engine.→
Part CXXII
DNS Troubleshooting
6 checks
- CoreDNS architecture and role — the cluster's DNS resolverKubernetes CoreDNS architecture and role: the CoreDNS Pod, the ConfigMap, the plugins, the deployment, and the systematic approach to diagnosing the cluster's DNS resolver.→
- NXDOMAIN, no endpoints, and stub-domain — the DNS resolution failuresKubernetes DNS resolution failures: NXDOMAIN, no endpoints, stub-domain, and the systematic approach to diagnosing the cluster's DNS resolution path.→
- Search path and ndots — the DNS client configurationKubernetes DNS search path and ndots: the Pod's resolv.conf, the ndots:5 option, the search path expansion, and the systematic approach to diagnosing DNS query failures caused by the client configuration.→
- CoreDNS scale and tuning — the resolver at scaleKubernetes CoreDNS scale and tuning: the replicas, the cache, the forward, the health, and the systematic approach to tuning the cluster's DNS resolver for production scale.→
- DNS outage and mitigations — the cluster wide failureKubernetes DNS outage and mitigations: the cluster-wide failure, the recovery strategies, the fallback to upstream DNS, and the systematic approach to surviving a CoreDNS outage.→
- External DNS and split-horizon — the cluster to enterprise DNSKubernetes External DNS and split-horizon: the external-dns controller, the split-horizon DNS, the cluster-to-enterprise DNS integration, and the systematic approach to publishing cluster Services to the enterprise DNS.→
Part CXXIII
NetworkPolicy Troubleshooting
6 checks
- Default-allow vs default-deny — the cluster network's postureKubernetes NetworkPolicy default-allow vs default-deny: the cluster's default network posture, the NetworkPolicy's role in changing it, and the systematic approach to diagnosing a cluster's network policy failures.→
- Selector mismatch diagnosis — the policy that misses the PodKubernetes NetworkPolicy selector mismatch: the diagnostic for a policy that does not match the Pods, the labels, the namespace, and the systematic approach to recovering a network policy that is not enforced.→
- Egress and DNS rules — the outbound traffic pathKubernetes NetworkPolicy egress rules and DNS: the outbound traffic path, the DNS server access, the kube-system namespace, and the systematic approach to diagnosing egress failures.→
- Cross-namespace policy — the multi-tenant networkKubernetes NetworkPolicy cross-namespace: the namespaceSelector, the multi-tenant network, the cluster-wide policy, and the systematic approach to diagnosing cross-namespace policy failures.→
- CNI enforcement validity — the policy engine checkKubernetes NetworkPolicy CNI enforcement validity: the CNI's policy engine, the policy engine's health, the policy audit, and the systematic approach to verifying the CNI is enforcing the policy.→
- Policy testing and CI gates — the prevention strategyKubernetes NetworkPolicy testing and CI gates: the policy as code, the CI pipeline, the policy validation, and the systematic approach to preventing NetworkPolicy failures.→
Part CXXIV
Node Troubleshooting
6 checks
- NotReady and unknown nodes — the heart of the workerKubernetes node NotReady and unknown: the diagnostic for the node heart, the kubelet, the runtime, and the systematic approach to recovering a node that has lost its heartbeat.→
- DiskPressure and PIDPressure — the resource exhaustionKubernetes node DiskPressure and PIDPressure: the diagnostic for the node's resource exhaustion, the kubelet's eviction, the disk usage, and the systematic approach to recovering a node under pressure.→
- kubelet logs and auth — the node's voiceKubernetes kubelet logs and authentication: the diagnostic for the kubelet's logs, the kubelet's authentication, the TLS certificates, and the systematic approach to recovering the kubelet.→
- containerd and runtime recovery — the container engineKubernetes containerd and runtime recovery: the diagnostic for the container engine, the runtime's logs, the image registry, and the systematic approach to recovering the node's container engine.→
- Node-level kernel and network — the host underneathKubernetes node-level kernel and network: the diagnostic for the kernel, the network interfaces, the routes, and the systematic approach to diagnosing the node's host.→
- Node replacement and rollback — the recovery pathKubernetes node replacement and rollback: the diagnostic, the recovery path, the cluster-api or kubeadm join, and the systematic approach to recovering a node that cannot be repaired.→
Part CXXV
Control Plane Troubleshooting
6 checks
- API server health and 503s — the cluster's front doorKubernetes API server health and 503 responses: the diagnostic for the API server, the etcd, the auth, and the systematic approach to recovering the cluster's front door.→
- controller-manager loops — the cluster's reconciliationKubernetes controller-manager loops: the diagnostic for the controller-manager, the reconciliation loops, the cluster's state, and the systematic approach to recovering the cluster's reconciliation.→
- Scheduler backlogs — the cluster's placement engineKubernetes scheduler backlogs: the diagnostic for the scheduler, the pending Pods, the scheduler's filters, and the systematic approach to recovering the cluster's placement engine.→
- kube-apiserver HA and load balancers — the cluster's front door at scaleKubernetes kube-apiserver HA and load balancers: the HA topology, the load balancer, the multi-master, the keepalived, and the systematic approach to a cluster's front door at scale.→
- cert rotation and clock skew — the cryptographic driftKubernetes certificate rotation and clock skew: the diagnostic for the certificates, the clock, the cryptographic drift, and the systematic approach to recovering the cluster's cryptographic health.→
- Control plane failure recovery — the cluster's worst dayKubernetes control plane failure recovery: the disaster recovery, the etcd snapshot, the kubeadm reset, the cluster rebuild, and the systematic approach to surviving the cluster's worst day.→
Part CXXVI
etcd Incident Response
6 checks
- etcd alarm and no-space — the cluster's disk exhaustionKubernetes etcd alarm and no-space: the diagnostic for the etcd's disk usage, the alarm, the defrag, and the systematic approach to recovering the etcd from disk exhaustion.→
- Quorum loss triage — the cluster's brain failureKubernetes etcd quorum loss: the diagnostic for the etcd's quorum, the failed members, the network partition, and the systematic approach to recovering the cluster's brain.→
- etcd snapshot restore — the cluster's ultimate recoveryKubernetes etcd snapshot restore: the diagnostic, the recovery path, the snapshot verification, and the systematic approach to restoring the cluster from an etcd snapshot.→
- Replace failed member — the cluster's recoveryKubernetes etcd replace failed member: the diagnostic, the new member, the cluster rebuild, and the systematic approach to replacing a failed etcd member.→
- Performance incident on etcd — the cluster's slownessKubernetes etcd performance incident: the diagnostic for the etcd's latency, the fsync, the disk, and the systematic approach to recovering the etcd from a performance incident.→
- Post-mortem and runbook exercises — the artefact that pays down the lessonKubernetes etcd post-mortem and runbook exercises: the diagnostic, the post-mortem, the runbook exercises, and the systematic approach to turning the etcd incident into a runbook improvement.→
Part CXXVII
Storage Troubleshooting
6 checks
- PVC pending and dynamic-provision — the storage runtimeKubernetes PVC Pending and dynamic provisioning: the diagnostic for the PVC, the StorageClass, the CSI driver, and the systematic approach to recovering a workload that cannot get storage.→
- CSI driver crash and force-detach — the storage runtime failuresKubernetes CSI driver crash and force-detach: the diagnostic for the CSI driver, the controller plugin, the node plugin, and the systematic approach to recovering from a storage runtime failure.→
- Filesystem remount and read-only — the storage integrityKubernetes filesystem remount and read-only: the diagnostic for the filesystem, the remount, the read-only state, and the systematic approach to recovering the storage integrity.→
- Volume expansion and quota — the storage growthKubernetes volume expansion and storage quota: the diagnostic for the volume expansion, the storage quota, the StorageClass, and the systematic approach to recovering the storage growth.→
- Snapshot and restore at CSI — the storage recoveryKubernetes CSI snapshot and restore: the diagnostic for the snapshot, the VolumeSnapshot, the restore, and the systematic approach to recovering data from a snapshot.→
- Performance IOPS and throttling — the storage bottleneckKubernetes storage performance: IOPS, throttling, the backend's limits, the cgroup's limits, and the systematic approach to recovering from a storage performance incident.→
Part CXXVIII
Application Performance Troubleshooting
6 checks
- Latency and saturation — the application performanceKubernetes application performance troubleshooting: latency, saturation, the four golden signals, and the systematic approach to diagnosing a slow application.→
- CPU throttling and limits — the compute bottleneckKubernetes CPU throttling and limits: the diagnostic for the CPU throttling, the cgroup's CFS quota, the CPU limit, and the systematic approach to recovering from a CPU bottleneck.→
- Memory limits and OOM semantics — the memory bottleneckKubernetes memory limits and OOM semantics: the diagnostic for the OOM, the memory limit, the kernel OOM killer, and the systematic approach to recovering from a memory bottleneck.→
- Network saturation and retries — the network bottleneckKubernetes network saturation and retries: the diagnostic for the network, the packet drops, the TCP retries, and the systematic approach to recovering from a network bottleneck.→
- Storage IOPS and tail latency — the storage performanceKubernetes storage IOPS and tail latency: the diagnostic for the storage performance, the 99th-percentile latency, the IOPS, and the systematic approach to recovering from a storage performance bottleneck.→
- Continuous profiling and tracing — the performance insightKubernetes continuous profiling and tracing: the diagnostic for the profiling, the tracing, the flame graphs, the spans, and the systematic approach to performance insight.→
Part CXXIX
Security Incident Response
6 checks
- Detection and triage — the security incident's first hourKubernetes security incident detection and triage: the alerts, the triage, the containment, and the systematic approach to the first hour of a security incident.→
- Containment and isolation — the security incident's first hourKubernetes security incident containment and isolation: the network policy, the cordon, the eviction, the credential rotation, and the systematic approach to containing the impact.→
- Container forensics — the security incident's investigationKubernetes container forensics: the diagnostic for the container, the audit logs, the file system, the network, and the systematic approach to investigating a security incident.→
- Cluster-wide credential rotation — the security incident's recoveryKubernetes cluster-wide credential rotation: the ServiceAccount, the certificate, the API server, the kubelet, and the systematic approach to rotating every credential in the cluster.→
- Disclosure and notification — the security incident's social contractKubernetes security incident disclosure and notification: the audience, the channel, the timing, the regulatory, and the systematic approach to disclosing a security incident.→
- Recovery and post-mortem — the security incident's closureKubernetes security incident recovery and post-mortem: the recovery, the post-mortem, the lessons, the action items, and the systematic approach to closing the security incident.→
Part CXXX
Production Anti-Patterns
6 checks
- Top 20 production anti-patterns — the cluster's hidden failuresKubernetes top 20 production anti-patterns: latest tags, no requests, no limits, no probes, no PDB, privileged workloads, default ServiceAccount, secrets in manifests, no NetworkPolicies, no backup, no etcd test, manual edits, no monitoring, all replicas on one node, and the systematic approach to identifying and fixing the cluster's anti-patterns.→
- Image and compute anti-patterns — the workload's foundationKubernetes image and compute anti-patterns: latest tags, no requests, no limits, no probes, and the systematic approach to fixing the workload's foundation.→
- Availability anti-patterns — the workload's resilienceKubernetes availability anti-patterns: no PDB, all replicas on one node, no anti-affinity, no topology spread, and the systematic approach to fixing the workload's resilience.→
- Security anti-patterns — the cluster's protectionKubernetes security anti-patterns: privileged containers, default ServiceAccount, secrets in manifests, no NetworkPolicies, no Pod Security Standards, and the systematic approach to fixing the cluster's security.→
- Operational anti-patterns — the cluster's runbookKubernetes operational anti-patterns: no backup, no etcd test, manual edits, no monitoring, and the systematic approach to fixing the cluster's operational hygiene.→
- CI gates and Polaris — the prevention strategyKubernetes CI gates and Polaris: the CI pipeline, the Polaris audit, the policy as code, and the systematic approach to preventing anti-patterns from being shipped to production.→
Part CXXXI
Production Reference Architecture
6 checks
- Production reference architecture — the canonical clusterKubernetes production reference architecture: the HA control plane, the multi-worker pools, the production networking, the cluster DNS, the persistent storage, the application Deployment, the stateful workload, the ConfigMaps, the Secrets, the requests/limits, the probes, the affinity/spread, the PDB, the Services, the Ingress/Gateway, the TLS, the NetworkPolicies, the RBAC, the workload security, the monitoring, the logging, the tracing, the backup, the etcd backup, the restore, the rolling deployment, the worker maintenance, the Kubernetes upgrade, the failure recovery.→
- HA control plane topology — the cluster's brain at scaleKubernetes HA control plane topology: the 3 control-plane nodes, the load balancer, the etcd cluster, the certificates, and the systematic approach to deploying the HA control plane.→
- Multi-worker pools and topology spread — the cluster's computeKubernetes multi-worker pools and topology spread: the general, memory-optimized, compute-optimized node pools, the topology spread constraints, the taints and tolerations, and the systematic approach to the cluster's compute.→
- Production networking and ingress — the cluster's connectivityKubernetes production networking and ingress: the CNI, the CoreDNS, the NetworkPolicy, the Ingress / Gateway API, the cert-manager, the TLS, and the systematic approach to the cluster's connectivity.→
- Stateful workload — the cluster's data layerKubernetes stateful workload reference: the StatefulSet, the PVC, the StorageClass, the headless Service, the init containers, the readiness probes, and the systematic approach to the cluster's data layer.→
- The reference architecture in one diagram — the cluster's complete storyKubernetes production reference architecture in one diagram: the HA control plane, the multi-worker pools, the production networking, the cluster DNS, the persistent storage, the application Deployment, the stateful workload, the ConfigMaps, the Secrets, the requests/limits, the probes, the affinity/spread, the PDB, the Services, the Ingress/Gateway, the TLS, the NetworkPolicies, the RBAC, the workload security, the monitoring, the logging, the tracing, the backup, the etcd backup, the restore, the rolling deployment, the worker maintenance, the Kubernetes upgrade, the failure recovery.→