Kubernetes for Production Sysadmins — Final Assessment
The final assessment has fourteen sections. Each section opens with production reasoning the student should be able to demonstrate from memory, followed by questions that exercise it. A passing score is 80%.
The theory is closed-book. The scenarios are open-book and open-shell: what matters is the evidence you capture and the reasoning you show. Every scenario answer should include:
- the symptom and its impact
- the evidence you collected, in order
- the most likely root cause, with justification
- the remediation you applied
- the verification you used to confirm recovery
- the rollback you kept ready if the fix did not work
A high-quality answer is specific, references the API objects involved, and identifies a concrete change in the operator’s practice. An answer that names a controller without naming the object it watches is incomplete; an answer that names the object without naming the controller is incomplete.
Section 1 — Control Plane and Reconciliation
The control plane is the cluster’s brain: kube-apiserver holds every object and is the only component that talks to etcd; kube-scheduler filters and scores nodes for every unscheduled Pod; kube-controller-manager runs the ReplicaSet, Node, EndpointSlice, ServiceAccount, Job, and dozens of other control loops. Reconciliation is the operating model — observed state drives toward desired state via control loops, and every production action is best understood as “set the desired state correctly and let the controllers converge”. HA means three (or five) control-plane nodes behind a load balancer; quorum loss in etcd is unrecoverable without restoring from a snapshot.
A passing student reasons about the control plane as one system: a Deployment manifest does not “create” Pods — it changes the desired state, the Deployment controller creates a ReplicaSet, the ReplicaSet controller creates Pods, the scheduler binds them, the kubelet runs them, kube-proxy programs the data plane, and CoreDNS publishes the Service endpoints. Every failure in that chain shows up as Pods not Ready, not as an “error” anywhere.
Section 2 — Workloads and Pod Lifecycle
Pods have phases (Pending, Running, Succeeded, Failed, Unknown) and containers have states (Waiting, Running, Terminated). A Pending Pod is a scheduling problem; a CrashLoopBackOff is an application or configuration problem; an OOMKilled is a memory problem. Three probe kinds matter: liveness (restart the container when stuck), readiness (route traffic only when the container can serve it), startup (give the container time to initialise before liveness kicks in). Probes are per-container, not per-Pod.
Deployments are stateless; StatefulSets carry stable identity (ordinal, DNS, per-Pod storage); DaemonSets run one Pod per node; Jobs run to completion; CronJobs schedule Jobs. Each workload has a different rollout discipline: Deployments do rolling updates, StatefulSets do ordered updates (with PDB guards), DaemonSets do node-by-node rollouts, Jobs do retry policies. Production anti-patterns include running stateful workloads on Deployments, missing requests, missing probes, and missing PodDisruptionBudgets.
Section 3 — Networking and Services
The Kubernetes networking model promises: every Pod gets a routable IP; Pods can reach every other Pod without NAT; agents on a node can reach Pods without NAT; Pods in a cluster can reach every other Pod without NAT; the IP a Pod sees for itself is the IP others see for it. CNI implements the first link; kube-proxy implements Services (iptables, IPVS, or eBPF); CoreDNS implements name resolution. The Service API has four flavours: ClusterIP (in-cluster), NodePort (each node, high port), LoadBalancer (cloud-provisioned), and ExternalName (DNS CNAME).
Gateway API is the modern replacement for Ingress: explicit
roles (infrastructure provider, cluster operator, application
developer), CRDs instead of annotations, cross-namespace
routing, and traffic splitting as first-class primitives.
NetworkPolicy is namespace-scoped L3/L4 policy: default
behaviour with no policy is allow-all, and the smallest
default-deny is one empty podSelector policy. NetworkPolicy
depends on the CNI enforcing it — Flannel does not.
Section 4 — Storage
Storage in Kubernetes flows from CSI through StorageClasses
to PVCs to Pods. The StorageClass names a provisioner and
parameters (fsType, type, iopsPerGB, encrypted); the
StorageClass default flag avoids storageClassName on every
PVC. A PVC is bound to a PV by the PV controller; the Pod
mounts the PVC. Volume expansion is a StorageClass capability
(allowVolumeExpansion: true) and a CSI driver capability
(online or offline). Snapshots are CSI snapshots — they are
crash-consistent by default and application-consistent only if
the application (or an operator helper) flushes.
Stateful workloads (databases, message queues, key-value
stores) belong on StatefulSets with per-Pod PVCs. A
StatefulSet alone is not a backup — volumeClaimTemplates
create PVCs, but the application data inside them needs an
application-consistent backup. Backups of stateful workloads
usually combine CSI snapshots (for fast recovery) with
application-level dump tools (for point-in-time and logical
restores).
Section 5 — Security and RBAC
Authentication is who you are (certificates, bearer tokens,
OIDC, ServiceAccount tokens, webhooks); authorization is what
you can do (always RBAC in production, never ABAC or
legacy-Authorizer); admission is what we let you change it to
(built-in plugins plus webhooks). Pod Security Standards
replaces PodSecurityPolicy with three namespace profiles
(privileged, baseline, restricted); restricted is the
goal. Secrets deserve their own discipline: encryption-at-rest
in etcd, RBAC on get/list of Secrets, no Secrets in
environment variables, External Secrets Operator for the
things that should never live in etcd.
ServiceAccounts are the Pod’s identity. Default ServiceAccounts
carry no rights by default but still expose a token via
automount; production workloads should disable automount
(automountServiceAccountToken: false) when not needed and
otherwise bind narrowly scoped Roles. Supply chain is the
last leg: tag → digest pinning, vulnerability scanning, image
signing (Cosign, Notary v2), SBOM, admission policies that
reject unsigned images.
Section 6 — Observability
A production cluster emits three signals: metrics, logs, and
traces — and three object-state streams: Kubernetes events,
audit logs, and controller-runtime logs from kube-scheduler,
kube-controller-manager, and kubelet. Metrics Server feeds
kubectl top and HPA. kube-state-metrics turns Kubernetes
object state into Prometheus metrics (Deployment replicas,
Node Ready, PVC phase, Pod restart count). Prometheus scrapes
kube-state-metrics, the kubelet, CoreDNS, and the application
workloads. Loki ships and indexes logs; Tempo stores traces;
Grafana dashboards tie them together.
Alerting discipline: alert on user-visible symptoms and recovery actions, not on internal-state noise. Top-tier alerts on a cluster: API server 5xx, etcd cluster size less than quorum, kubelet certificate expiry < 30 days, Node NotReady
15 minutes, Deployment AvailableReplicas < DesiredReplicas, PVC Phase Pending > 10 minutes. Avoid alert storms on per-Pod restarts; aggregate.
Section 7 — HA and Upgrades
HA for the control plane means three (or five) nodes behind a load balancer, etcd quorum = 2 (or 3) of 3 (or 5), and leader-elected controller-manager and scheduler. A worker node failure is a pod-replacement problem, not a control-plane problem. Loss of one etcd member in a 3-member cluster is survivable; loss of two is not (quorum loss); recovery requires restoring from snapshot.
Upgrades follow kubeadm’s strict sequence: control plane
first (one node at a time, with etcd backup before each), then
workers (drain, upgrade kubelet, uncordon; surge a new node
and decommission the old for zero-impact rollouts). Version
skew is bounded: kubelet must be within three minor versions
of the API server, kubectl within one. API deprecation is
caught with pluto and kubent against the manifests before
the upgrade; CI gates prevent the deprecated API from
shipping in the first place.
Section 8 — Disaster Recovery
A backup strategy covers three things: cluster state (etcd snapshot, ideally daily with hourly incrementals and 30-day retention); workload configuration (the Git repo, plus a Velero backup of namespace resources); persistent data (CSI snapshots and application-consistent exports, ideally to a separate bucket and account). Velero is the canonical tool for namespace-scoped backup of resources; it does NOT backup PV contents unless paired with a CSI snapshotter or Restic / Kopia.
Recovery is two distinct exercises. Restore means
rebuilding the same cluster onto the same hardware. DR means
rebuilding onto different hardware (or a different region)
when the primary is gone. Etcd restore: stop the control
plane on every node, run etcdutl snapshot restore on each
member with its own --name and
--initial-advertise-peer-urls plus the shared
--initial-cluster, restart etcd, restart kube-apiserver,
verify. Validate the restored cluster: API
reachable, all nodes Ready, Pods Scheduled, no PVCs Pending.
Schedule game days monthly; an untested backup is a wish.
Section 9 — Scheduling, Placement and Node Lifecycle
Scheduling is a two-phase decision: filtering removes the nodes that cannot host the Pod (insufficient allocatable capacity, an unmatched nodeSelector or nodeAffinity, an untolerated taint, a PVC bound to the wrong zone), then scoring ranks whatever survives. Placement policy is written with nodeAffinity (which nodes), podAffinity and podAntiAffinity (which neighbours), and topologySpreadConstraints (how evenly across a domain). maxSkew is an imbalance budget measured against the least populated domain, and whenUnsatisfiable: DoNotSchedule turns that budget into a hard admission rule, so an empty or unschedulable domain will wedge an entire rollout.
Node lifecycle is the other half. The kubelet renews its Lease every ten seconds; the node controller declares Ready=Unknown after —node-monitor-grace-period (40s by default) and taints the node node.kubernetes.io/unreachable:NoExecute; Pods leave 300 seconds later under the toleration that admission gave them. Under memory, disk, or PID pressure the kubelet evicts locally instead, ranked by QoS class and by usage above requests. A passing student can look at a Pending Pod, a NotReady node, and an evicted Pod and say which of those three systems produced it, and therefore which knob changes the outcome.
Section 10 — Multi-Tenancy and Resource Governance
A namespace is a naming and policy scope, not a security boundary. Soft multi-tenancy is four things working together: RBAC per tenant, a default-deny NetworkPolicy, ResourceQuota for the budget, and LimitRange for the defaults that make the budget enforceable. A quota on requests.cpu rejects every Pod that does not declare a CPU request, so the LimitRange that supplies the default is what keeps the tenant’s existing manifests working. PriorityClass decides who wins when the budget runs out, and preemption is bounded by the PodDisruptionBudgets of the candidate victims.
What a namespace does not isolate matters as much as what it does: the node kernel, the container runtime, and every cluster-scoped object — CustomResourceDefinitions, PersistentVolumes, StorageClasses, ClusterRoles — are shared, along with the control plane itself. Hard multi-tenancy means separate clusters, or virtual control planes such as vCluster, plus policy enforcement with Kyverno or a ValidatingAdmissionPolicy and a restricted Pod Security Standard. A passing student can state which risks the namespace stack retires and which it merely documents.
Section 11 — Packaging, GitOps and Declarative Delivery
Helm renders YAML from values and records each release as a revision stored in Secrets in the release namespace, which is what helm rollback reads and what a Secret-pruning job destroys. Kustomize patches plain YAML with no template language, and its configMapGenerator appends a content hash to the generated name so that changing configuration changes the Pod template and therefore triggers a rollout — the exact failure a hand-written ConfigMap has, where the data changes and nothing restarts. helm upgrade —atomic rolls a failed release back instead of leaving it half-applied; unpinned chart versions and floating image tags are the two packaging habits that destroy reproducibility.
GitOps closes the loop: Git holds the desired state and Argo CD or Flux reconciles the cluster toward it. Drift is the central operational concept — an out-of-band kubectl edit leaves the Application OutOfSync, and self-heal decides whether the controller reverts it or waits for a human. Every emergency change made with kubectl is a debt against the repository, and it is paid back by committing the same change before the incident is closed. A passing student treats an uncommitted hotfix as an open incident action, not as a fix.
Section 12 — DNS, TLS and the Cluster Edge
CoreDNS answers from the Corefile plugin chain, and the Pod resolv.conf ships ndots:5, so any name with fewer than five dots walks the search path first — several NXDOMAIN round trips for one external hostname. The mitigations are NodeLocal DNSCache as a DaemonSet, fully-qualified names or a trailing dot, dnsConfig options that lower ndots, and enough CoreDNS replicas with the cache plugin tuned. Watch conntrack as well: UDP DNS at scale exhausts the node’s table before CoreDNS itself struggles. On bare metal there is no cloud controller to allocate a LoadBalancer address, so MetalLB in L2 mode elects one node to answer ARP for the VIP (failover, not distribution) and BGP mode peers with the routers for ECMP across nodes.
TLS at the edge is managed, not manual. cert-manager turns a Certificate into a CertificateRequest, an Order, and a Challenge, and writes the result into a Secret that the Ingress controller or Gateway reads; the issuer may be ACME, an internal CA, or Vault PKI. Renewal rewrites the same Secret in place, so a workload that loads its key material once at start-up needs a watcher or a reload path or it will serve an expired certificate. Registries are the other edge: pull-through caches for availability, namespace-scoped imagePullSecrets, and digests rather than tags for reproducibility.
Section 13 — Capacity, Autoscaling and Performance
Three autoscalers act on three different signals. The HPA changes the replica count from a metric, computing desired replicas as the current count multiplied by the ratio of current to target and rounded up, immediately on the way up and after a 300-second stabilisation window on the way down. The VPA changes requests and limits from observed usage, and applying them means evicting the Pod. The Cluster Autoscaler changes the node count, and it reasons about the sum of Pod requests against allocatable — never about measured usage, which is why an over-requested, idle fleet never shrinks.
Performance work starts where capacity planning ends. A CPU limit is a CFS quota granted every 100 ms, so a container can be throttled hard while its 30-second average looks idle; container_cpu_cfs_throttled_seconds_total is the evidence, not kubectl top. Memory limits have no such elasticity — the kernel OOM-kills at the limit and the restart is the only symptom. Storage appears as tail latency rather than as errors, and the network as retransmits, dropped conntrack entries, and retries. A passing student names the metric that proves a bottleneck before changing a single field in a manifest.
Section 14 — Troubleshooting Method and Production Architecture
The method is the deliverable: state the symptom and its impact, inspect the object, read the events, read the logs, map the dependencies, name the component responsible, form one hypothesis, test it, restore service, validate against the user-visible path, and write it down. Evidence is collected before anything is changed, and restoration precedes root cause whenever users are affected. The production anti-patterns are the inverse of that method: no requests, no probes, no PodDisruptionBudget, floating tags, cluster-admin everywhere, manual changes with no repository behind them, and a backup nobody has ever restored.
Production architecture is the sum of the preceding thirteen sections: three control-plane nodes across three failure domains behind a load balancer, separate worker pools for infrastructure and application workloads, topology spread on every critical Deployment, a CNI that actually enforces NetworkPolicy, CSI-backed storage with tested snapshots, an observability stack that alerts on symptoms rather than on internal state, and a change process with pre-change gates, canary fleets, and a post-incident review that produces an artefact. A passing student can defend each of those choices and say what it costs.