Linux final assessment - production-ready sysadmin

A final assessment for the Production Linux Sysadmin course. It covers the whole curriculum: foundation, administration, supply chain and patching, security, performance, HA, backup and DR, hardware and out-of-band access, TLS and secrets, change and drift, containers, and incident response.

Format

• Part A - 39 auto-scored questions, in the set above this text. Machine-marked, with a published explanation for every item.
• Part B - 96 written short-answer prompts in the sections below. Each carries a model answer directly underneath.
• Part C - 8 scenarios, each with a rubric of required elements. The rubrics are in the frontmatter of this assessment and summarised in the scenario section below.
• Open-book, timed (2.5 hours).
• Pass: 70% overall, 60% per domain, 6 of 8 scenarios.

Domains and how they are scored

The 60% floor applies per domain. These are the domains:

A domain score combines its Part A questions and its written section. A domain below 60% is a fail regardless of the overall total, because an overall pass built on a hole in one domain is exactly the credential this assessment exists to prevent.

Section 1: Foundation (10 written)

1. What does the kernel’s OOM killer do?

   Model answer: When the kernel cannot reclaim memory it selects a task and kills it. Selection is by oom_score, which is driven by memory footprint and adjusted by oom_score_adj. The kill is recorded in the kernel log. An OOM kill is a symptom of overcommit or an unbounded working set, not a cause; the fix is a memory limit on the unit or a smaller working set.

2. What is the difference between a process and a thread?

   Model answer: A process owns an address space; threads within it share that address space and file descriptors. Linux schedules both as tasks, which is why ps -L shows threads. Operationally: a crash or an OOM kill takes the whole process, and a stuck thread can hold a lock the rest of the process needs.

3. What is the difference between ext4 and XFS?

   Model answer: ext4 is mature, can be shrunk, and behaves well on small filesystems. XFS scales better for large files and parallel I/O and is the RHEL default, but it cannot be shrunk at all. That one asymmetry decides the capacity plan, because an over-provisioned XFS volume can only be reclaimed by creating a new volume and copying.

4. How does systemd manage service dependencies?

   Model answer: Requirement and ordering are separate. Requires= and Wants= state what must or should also be active; Before= and After= state the order. Declaring Requires= without After= starts both at once, which is the most common unit bug. Verify with systemctl list-dependencies and systemd-analyze verify.

5. What is the difference between journald and rsyslog?

   Model answer: journald stores structured, indexed, binary records with per-boot separation and rich metadata, locally. rsyslog handles plain text, filtering and network forwarding. On most hosts journald is the collector and rsyslog is the shipper. Retention is configured separately in each, and a host can lose logs at either point.

6. What is the difference between /etc and /var?

   Model answer: /etc is configuration: static, small, owned by configuration management, and reproducible. /var is variable state: logs, spools, caches and databases. It grows without limit if unattended, which is why it belongs on its own filesystem with monitoring on both space and inodes.

7. How does SSH key authentication work?

   Model answer: The client proves possession of the private key by signing a challenge bound to the session, and the server checks the signature against authorized_keys. The private key never leaves the client. Host key verification is the mirror image and protects you from connecting to an impostor - which is why duplicate host keys from a golden image break the guarantee entirely.

8. What is the difference between TCP and UDP?

   Model answer: TCP is connection-oriented, ordered and retransmits lost segments; UDP is a datagram service with no delivery guarantee. The choice sets the failure mode. TCP turns loss into latency; UDP turns loss into missing data, which is why UDP syslog drops messages silently.

9. What is the kernel’s role in the operating system?

   Model answer: It mediates access to hardware, schedules tasks, manages memory, implements filesystems and the network stack, and enforces isolation boundaries. Everything above it is a process. Containers share it, which is why a kernel upgrade or a kernel panic is a fleet-wide event on a container host.

10. What is systemd’s responsibility as PID 1?

    Model answer: It starts and supervises units, reaps orphaned processes, mounts filesystems, manages the shutdown sequence and owns the cgroup hierarchy. If PID 1 exits, the kernel panics. That is why PID 1 in a container matters too: a process that does not reap children leaks zombies.

Section 2: Single-server administration (14 written)

Part A questions d2-q1 to d2-q4 score the same domain. They test destructive-command discipline specifically — verifying the target before an irreversible operation, and knowing which flag inspects and which one writes.

1. How do you find the process using a specific port?

   Model answer: ss -lntp (or -lnup for UDP), or lsof -i :PORT, or fuser -n tcp PORT. Run it as root or the process name is hidden. If nothing is listening, the question changes: check whether the service failed to bind, and read the journal for the bind error.

2. How do you extend an LVM volume?

   Model answer: Confirm free extents with vgs, then lvextend -L +100G /dev/vg0/lv, then grow the filesystem with resize2fs or xfs_growfs. Both can be done online. Growing is the safe direction; nothing is lost if you stop halfway.

3. How do you configure a static IP with NetworkManager?

   Model answer: nmcli con mod <name> ipv4.method manual ipv4.addresses <addr/prefix> ipv4.gateway <gw> ipv4.dns <servers>, then nmcli con up <name>. Set ipv6.gateway too if you configure IPv6 manually. Verify with ip a and ip r, and keep a console path open before applying.

4. How do you add a new user with sudo?

   Model answer: useradd -m -s /bin/bash name, set authentication, then grant privilege - group membership (sudo on Debian, wheel on RHEL) or a file in /etc/sudoers.d. Validate any sudoers file with visudo -c -f before installing it; a syntax error there locks everyone out of sudo.

5. How do you find files by size or modification time?

   Model answer: find /path -size +1G and find /path -mtime +30. -mtime counts whole 24-hour periods, which surprises people at boundaries; use -newermt "2026-08-01" for precise windows. Add -printf '%s %p\n' and sort when you need the largest.

6. How do you check disk space and inode usage?

   Model answer: df -h and df -i, always together. Inode exhaustion presents exactly like a full disk - writes fail with ENOSPC - but df -h shows free space. The fix is different too: delete small files or rebuild the filesystem with more inodes.

7. How do you configure systemd resource limits for a service?

   Model answer: A drop-in under /etc/systemd/system/<unit>.d/ with MemoryMax=, MemoryHigh=, CPUQuota= and TasksMax=, then systemctl daemon-reload and restart. Confirm the limit is real with systemctl show <unit> -p MemoryMax rather than trusting the file - a typo silently yields no limit.

8. How do you find a process consuming CPU?

   Model answer: top or pidstat -u 1 to identify it, then split user time from system time. High system time points at syscalls or the kernel; high user time points at the application. top -H or pidstat -t narrows it to a thread, which is what you need before profiling.

9. How do you configure SSH passwordless login?

   Model answer: Generate a key, install the public half with ssh-copy-id, and check permissions: 700 on ~/.ssh, 600 on authorized_keys, and a home directory not group writable. Verify the key works before disabling password authentication, and keep a console path while you do it.

10. How do you update packages safely on a production host?

    Model answer: Know the exact version you are moving to, stage it somewhere that is not production, patch in waves with a hold time and an abort trigger, restart the services that map replaced libraries, and verify with evidence rather than with the package version alone. Section 7 covers each element.

11. You have edited /etc/fstab on a live production host. How do you validate the change so that the next reboot cannot fail?

    Model answer: Never leave it untested — an fstab typo is a boot failure that needs console access to fix. Validate before you walk away: findmnt --verify --verbose parses the file and reports bad options, missing devices and unknown filesystem types. systemctl daemon-reload then mount -a proves the entries actually mount; systemctl daemon-reload matters because systemd generates a .mount unit per entry and will otherwise act on the stale generation. Confirm any UUID with blkid rather than trusting the one you copied. For a filesystem the host can survive without, add nofail (and a short x-systemd.device-timeout=) so a missing device degrades instead of dropping the host to emergency mode.

12. An ext4 filesystem and an XFS filesystem each fail to mount after an unclean shutdown. How does the repair differ, and what is the last-resort option on XFS?

    Model answer: Both must be unmounted first — running a repair on a mounted filesystem corrupts it. On ext4, e2fsck -f replays the journal and repairs metadata, and it is routine. On XFS the tool is xfs_repair, which will refuse to run while the log is dirty and tell you to mount and unmount the filesystem so the log replays cleanly. The last resort is xfs_repair -L, which zeroes the log: it discards every metadata change the log had not yet committed, so it is a data-loss operation and belongs behind a backup and a decision someone recorded. Take an image of the device first if the data matters. Note the asymmetry with resize while you are here: ext4 grows and shrinks, XFS grows only.

13. A production host’s root filesystem remounted read-only at 02:00. Walk through your diagnosis before you consider remounting read-write.

    Model answer: A read-only remount is the kernel protecting the filesystem after an I/O or metadata error, so treat the remount as a symptom and find the error first. Read dmesg -T and journalctl -k for the original error — the first one, not the cascade — and for EXT4-fs error or XFS ... Corruption lines naming the device. Check the hardware path: smartctl -a, the RAID controller, multipath state, and whether the backing device disappeared and returned. Establish whether the filesystem is merely read-only or actually damaged. Only then decide: remounting read-write on a filesystem with live metadata corruption spreads the damage, and it destroys the read-only state that is currently keeping the data intact. If the device is failing, the correct move is to copy data off, not to keep the host running. Note that space and inode monitoring will not catch this — usage stops changing, so alert on the read-only state itself.

14. How do you shrink an LVM volume, and which filesystem cannot be shrunk at all?

    Model answer: XFS cannot be shrunk. On ext4: back up, unmount, e2fsck -f, resize2fs to the smaller size, then lvreduce. The order is the whole answer - lvreduce first removes extents the filesystem still believes it owns, and the filesystem is destroyed. There is no undo; recovery is restore from backup. For XFS, create a new smaller volume, copy, verify, switch, then remove the old one.

Section 3: Security (10 written)

1. What is SELinux enforcing mode?

   Model answer: Policy denials are enforced rather than only logged. Permissive mode logs the same denials without blocking, which makes it a diagnostic tool, not a destination. Read denials with ausearch -m avc and fix them with a context correction or a targeted policy module - never by disabling SELinux.

2. How does nftables differ from iptables?

   Model answer: One unified framework with a single syntax across families, sets and maps as first-class objects, and atomic ruleset replacement. iptables splits IPv4 and IPv6 and applies rules incrementally. The operational consequence: with nftables the ruleset either applies whole or not at all, so you cannot end up half-firewalled.

3. A binary is not setuid, yet it runs with privileges the invoking user does not have. How is that possible, and how do you audit for it?

   Model answer: File capabilities. setcap attaches a specific privilege to the binary — cap_net_raw=ep on ping, cap_net_bind_service=ep on a service that binds port 443 — without the all-or-nothing escalation of setuid root. find / -perm /6000 will never show them, which is why a setuid-only audit is incomplete: getcap -r / is the other half. Treat cap_setuid, cap_sys_admin and cap_dac_override on any binary as equivalent to setuid root, because each is a route back to full privilege.

4. How do you configure SSH key-based authentication?

   Model answer: Install the public key in authorized_keys, fix the permissions, then set PubkeyAuthentication yes, PasswordAuthentication no and PermitRootLogin prohibit-password. Validate with sshd -t and keep an open session while reloading. Restrict by key options or Match blocks where the key is for automation.

5. What is the principle of least privilege?

   Model answer: Every actor gets exactly the access its job needs and no more, for no longer than it needs it. In practice: specific sudo commands rather than ALL, dropped capabilities on units, per-service accounts, and scoped API credentials. It limits blast radius; it does not prevent compromise.

6. How does fail2ban work?

   Model answer: It watches logs for a failure pattern and adds a temporary firewall rule for the offending source. It reduces brute-force noise. It does not replace key-only authentication, and a badly written jail will lock out your own monitoring or, behind a proxy, ban the proxy address for everyone.

7. What is the recommended password storage in /etc/shadow?

   Model answer: A modern slow hash with a per-user salt - yescrypt ($y$) where available, otherwise sha512crypt ($6$) with a high round count. MD5 ($1$) is obsolete. An empty field means no password required and ! or * means the account cannot authenticate by password; the difference matters when auditing.

8. How do you use sudo to grant specific commands?

   Model answer: A file in /etc/sudoers.d naming the exact binaries, validated with visudo -c -f. Use absolute paths, avoid wildcards that permit argument injection, and never grant an editor, a shell escape or anything that runs arbitrary commands. sudo -l shows what a user actually has.

9. What is the role of auditd in security?

   Model answer: It records security-relevant kernel events - syscalls, file access, privilege changes - to a log designed for investigation rather than debugging. It answers “who changed this and when”. It is only useful if the rules are targeted, the log is shipped off the host, and someone actually reads it.

10. How do you configure AppArmor for a service?

    Model answer: Write or install a profile for the binary, load it in complain mode, exercise the service, collect the denials with aa-logprof, refine, then set enforce mode. Check the state with aa-status. The discipline is the same as SELinux: observe, then constrain, never disable.

Section 4: Performance and monitoring (10 written)

1. What does the USE methodology check?

   Model answer: For every resource: Utilisation, Saturation and Errors. Saturation is the one people skip and the one that predicts trouble - a device at 70% utilisation with a deep queue is in worse shape than one at 95% with none. Working the list resource by resource is what stops a diagnosis becoming a tool tour.

2. How do you identify the bottleneck in a slow system?

   Model answer: Start from the symptom and a timeline of what changed, then work USE across CPU, memory, disk and network on each tier. Name a saturated resource with evidence before changing anything. “No resource is saturated” is a valid result and redirects you to dependencies, locks and the application itself.

3. What is the difference between iostat await and %util?

   Model answer: await is the average time a request spends waiting plus service time - it is what the application feels. %util is the fraction of time the device had at least one request in flight. On modern parallel devices %util can sit near 100% while the device is nowhere near its limit, so await and queue depth are the meaningful signals.

4. How do you read load average?

   Model answer: Against core count, and as three numbers. On Linux it counts runnable and uninterruptible tasks, so heavy disk or NFS waiting inflates it without any CPU pressure. Load 16 on 32 cores is unremarkable; load 16 on 4 cores with low CPU utilisation means you are blocked on I/O.

5. What is the difference between RSS and VSZ?

   Model answer: VSZ is the size of the address space the process has mapped, including mappings never touched. RSS is what is actually resident in physical memory, and it double-counts shared pages across processes. Neither is a clean measure of a process’s true cost; the cgroup memory accounting is.

6. How does the OOM killer select victims?

   Model answer: By oom_score, driven mainly by memory footprint and adjusted by oom_score_adj. The largest consumer usually loses, which is often the database rather than the leaking process. Protect what matters with oom_score_adj, and bound the leaker with MemoryMax= instead of relying on the kernel to choose correctly.

7. What is the difference between Prometheus and Grafana?

   Model answer: Prometheus scrapes, stores and evaluates time series and generates alerts. Grafana queries data sources and draws them. Alerting can live in either, and deciding which one owns it is a real operational choice - split it and you get alerts nobody can find.

8. How do you configure Loki for log aggregation?

   Model answer: Ship with Promtail or the Grafana agent, label streams carefully, and let Loki index the labels rather than the content. High-cardinality labels such as request ID are the way to destroy a Loki install. Retention and object storage must be sized against the ingest rate before it is load-bearing.

9. What is eBPF used for in performance analysis?

   Model answer: Running verified programs in kernel context to observe syscalls, scheduling, block I/O and network events with low overhead and without patching anything. It answers questions sampling profilers cannot, such as per-request block latency distribution. The cost is kernel version sensitivity. Where you do want a sampling profile — perf record -F 99 -g or a BPF profiler, folded into a flame graph — the prerequisite is symbols: without frame pointers or debug information the result is a wall of unknown addresses.

10. A host shows load average 40 on 16 CPUs, %us 5 and %wa 60. Is the CPU saturated? What is your next command, and why?

    Model answer: No. Load average counts runnable (R) tasks and tasks in uninterruptible sleep (D), and the latter are blocked on I/O consuming no CPU at all. %us of 5 says the CPUs are almost idle; %wa of 60 says they are waiting on storage. Next command is vmstat 1: the r column is the real run queue and b counts tasks blocked in D state. Expect r low and b high, which confirms storage rather than CPU. Then iostat -x 1 for await and queue depth, and ps -eo state,comm | awk '$1 ~ /D/' for who is stuck. Adding CPUs to this host changes nothing.

Section 5: HA and clustering (10 written)

1. What is the difference between active-passive and active-active clusters?

   Model answer: Active-passive runs the workload on one node and holds the others in reserve, so capacity planning is simple and failover has a visible gap. Active-active runs it everywhere, which needs shared-state handling and gives you the two-writer problem to solve. Most “active-active” deployments are active-passive with a load balancer.

2. What is quorum and why does it matter?

   Model answer: The majority a partition needs before it may act. It exists so that only one partition can ever act, which is what stops two halves of a cluster serving the same data. Quorum is floor(votes / 2) + 1. It is necessary and not sufficient: on two nodes with two_node: 1 both sides are quorate and only fencing saves you.

3. How does Pacemaker decide where to run resources?

   Model answer: By scoring every node for every resource using constraints, stickiness and health, then running the allocation with the highest score. Location constraints bias placement, colocation ties resources together, and ordering sequences them. crm_simulate shows the decision before you commit to it.

4. What is fencing and why is it required?

   Model answer: Forcibly removing a node whose state is unknown, usually by cutting its power or its storage access. It is required because “unresponsive” and “still writing” look identical from outside. Without it, recovering a resource elsewhere risks two nodes writing the same data. It must be tested, not merely configured.

5. How does keepalived provide a floating IP?

   Model answer: VRRP elects a master among peers, and the master claims the virtual IP and announces it with a gratuitous ARP. On failure a backup takes over. It moves an address; it knows nothing about your data. Combined with shared storage and no fencing, it will happily give you two writers.

6. What is the difference between Corosync and Pacemaker?

   Model answer: Corosync is the membership and messaging layer - who is in the cluster, and quorum via votequorum. Pacemaker is the resource manager that decides what runs where. Corosync answers “who is here”, Pacemaker answers “what should be running”. Diagnose in that order.

7. How do you configure constraints in Pacemaker?

   Model answer: pcs constraint location, colocation and order, with scores. INFINITY makes a constraint mandatory; a finite score makes it a preference. Over- constraining is a common cause of resources that will not start anywhere; crm_simulate -sL explains why.

8. What is the role of a quorum device in a 2-node cluster?

   Model answer: It supplies a third vote from outside the pair, so one surviving node holds a genuine majority instead of relying on two_node: 1. It must sit on an independent failure domain, or it fails with the thing it was meant to arbitrate.

9. How do you test cluster failover?

   Model answer: Deliberately, in stages, with the expected outcome written down first: clean stop, then resource failure, then hard power loss, then interconnect partition with fencing enabled. Measure the recovery time and check afterwards that nothing is running twice. A failover test that was not fenced has proved very little.

10. What is split brain and how is it prevented?

    Model answer: Two partitions both believe they own the resource and both act, which on shared storage corrupts data with no route back. Prevention is quorum plus fencing: quorum stops a minority acting, fencing makes the state of the other side certain. Two-node clusters depend on fencing alone.

Section 6: Backup and DR (10 written)

1. What does the 3-2-1-1-0 backup rule mean?

   Model answer: Three copies, on two media types, one off-site, one immutable or offline, and zero errors on a verified restore. The last two digits are the ones that matter now: immutability defeats ransomware that deletes backups, and the zero means the restore has actually been tested.

2. What is the difference between file backup and application backup?

   Model answer: A file backup copies bytes and is crash-consistent at best. An application backup uses the application’s own mechanism - a dump, a log-shipping checkpoint, a quiesced snapshot - and is consistent by construction. Copying live database files is the classic way to obtain a backup that restores into a corrupt database.

3. How does BorgBackup work?

   Model answer: Content-defined chunking with deduplication across the repository, compression and authenticated encryption, kept in an append-oriented repository with borg check for verification. Deduplication is the strength and the risk: one repository is a single point of failure, so it needs its own copy.

4. What is the role of immutable backups?

   Model answer: To survive an attacker who holds your credentials. Object lock or write-once media means a copy cannot be deleted or altered before its retention expires, even by an administrator. Without it, a ransomware operator deletes the backups first and the retention policy is whatever they choose.

5. How do you test a restore procedure?

   Model answer: Restore to a clean host from the backup alone, with no access to the original, and verify the data by an application-level check rather than by file count. Time it and record the number, because that is your real RTO. Untested restores fail on the missing pieces: keys, credentials, and the runbook itself.

6. What is RPO and how is it calculated?

   Model answer: The maximum data loss you accept, measured in time. It is set by backup or replication frequency plus the time to detect a failure, not by the schedule alone. An hourly backup that is discovered broken after a day gives an RPO of a day. Monitor backup success, or your stated RPO is an aspiration.

7. How do you design a DR runbook?

   Model answer: Write it for someone who was not there: preconditions, the declaration decision and who makes it, ordered steps with expected output, verification, and a documented failback. Include credentials and their location, dependency order, and contacts. Then rehearse it, because an unrehearsed runbook is a document, not a capability.

8. What is the role of a witness in DR?

   Model answer: To break the tie between two sites so that only one takes over. Placed in a third failure domain, it turns “both sites think they are primary” into a decision. A witness in one of the two sites is not a witness; it is a vote for that site.

9. How do you encrypt backups at rest?

   Model answer: Encrypt in the backup tool, with the key held outside the systems being backed up, and an escrow copy held somewhere you can reach during a disaster. Test a restore using only the escrowed key. A backup you cannot decrypt during the incident is indistinguishable from no backup.

10. How do you handle full cluster loss in DR?

    Model answer: Rebuild from the image and configuration management first, then restore data, then re-form the cluster, and only then return traffic. Decide the surviving source of truth explicitly before restoring anything. Do not restore cluster state blindly, because a restored CIB can start resources against storage that has already moved.

Section 7: Supply chain, patching, scheduling and config management (10 written)

Part A questions d3-q1 to d3-q10 score the same domain.

1. How do you establish that a package you are about to install came from who you think it did?

   Model answer: The repository is signed and you have verified the signing key’s fingerprint out of band. Trust is scoped to that repository with Signed-By or a per-repo gpgkey. gpgcheck=1 and repo_gpgcheck=1 are on. HTTPS protects the transport only; it says nothing about who built the package.

2. What is wrong with curl … | bash as an install method, and what do you do when a vendor offers nothing else?

   Model answer: You execute code you have not read, from a server that can serve different content to your review than to your shell, with no signature, no version pinning and no uninstall. If it is the only option: download to a file, check the checksum against a separately published value, read it, pin the version, and re-run the review when the version changes.

3. What is an SBOM for, operationally?

   Model answer: To answer “which of my hosts contain this component” in minutes rather than days when an advisory lands. It is an inventory question, not a compliance artefact. It only works if it is generated at build time and kept queryable.

4. How do you prioritise a set of CVEs?

   Model answer: By exposure, exploitability and blast radius in your environment, not by CVSS ordering. An internet-facing service with a public exploit outranks a higher-scored issue on a host reachable only from inside. Record the reasoning; the order is a decision you will defend.

5. What is a wave rollout and what must be defined before it starts?

   Model answer: Patching in growing batches with a hold between them. Before the first host: batch sizes, hold time covering a full traffic cycle, a measurable abort trigger, the exact version, and a rollback that has been executed at least once.

6. How do you roll back a regressed package on Debian and on RHEL?

   Model answer: RHEL: dnf history undo <id>, limited to versions still available in a repository. Debian: apt install pkg=version from the cache or a snapshot repository, because there is no transaction history. Neither reverses migrations or rewritten configuration files, which is the limitation that matters.

7. What is the risk of apt-mark hold and dnf versionlock?

   Model answer: The host stops receiving updates for that package, including security updates, and reports itself fully patched. Holds need an owner, an expiry and a review. Put apt-mark showhold and dnf versionlock list into the drift check.

8. After patching a library, how do you prove the fix is actually in effect?

   Model answer: Enumerate processes still mapping the replaced object - lsof +c 0 | grep -E 'DEL|deleted', needs-restarting -s, or checkrestart - restart them in a controlled order, and re-run until the list is empty. The empty list is the evidence, not the package version.

9. What does a systemd timer give you that cron does not?

   Model answer: Overlap prevention (systemd will not start a unit that is already running), OnUnitInactiveSec so the interval is measured from the end of the last run, Persistent=true for missed runs, structured logs per invocation, resource control from the unit, and OnFailure= for notification. Cron gives you none of these without flock and a working mail path.

10. How do you detect configuration drift, and what is the trap?

    Model answer: Run configuration management in check mode on a schedule and alert on the changed count. The trap is the exit code: ansible-playbook --check exits 0 whether or not it found drift, because non-zero is reserved for failures. A check that only tests $? never fires.

Section 8: Hardware, out-of-band access, TLS and secrets (8 written)

Part A questions d8-q1 to d8-q6 score the same domain.

1. Which SMART signals justify replacing a drive?

   Model answer: Rising Reallocated_Sector_Ct or Current_Pending_Sector, any Offline_Uncorrectable, or a failed self-test. Trend beats absolute value, and the overall PASSED verdict is nearly meaningless on its own. Behind a RAID controller use smartctl -d megaraid,N.

2. What does nvme smart-log tell you that a filesystem-level check cannot?

   Model answer: Device health from the controller: critical_warning bits, available_spare against its threshold, percentage_used against endurance, media errors, and temperature. Alert on the spare margin shrinking, not on the warning bit appearing - by then the device is at the edge.

3. What is a BMC and what is the operational rule about its network?

   Model answer: A management controller with its own power and network that gives you console, power control and sensors independently of the host OS. It must sit on a network isolated from both production and the cluster interconnect, because it is your recovery path and your fencing path - and it must not fail with the thing it is there to recover.

4. Why must a server certificate carry a subjectAltName?

   Model answer: Hostname verification is defined against SAN entries, and Common Name fallback has been removed from Go and the browsers. The dangerous part is that the chain still verifies, so a check based on openssl s_client output looks clean while every real client refuses to connect.

5. openssl s_client prints Verification: OK. What has not been checked?

   Model answer: The hostname, unless you passed -verify_hostname, and revocation, which needs -crl_check or OCSP. -connect supplies a destination, not an identity to match.

6. What must exist before you can revoke a certificate from your internal CA?

   Model answer: The CA index database and serial file alongside the CA key, plus the issued certificate. Afterwards you must regenerate the CRL, publish it where clients fetch it, and confirm clients are configured to check it. Revocation nobody checks is a paperwork exercise.

7. How do you pass a secret to a command without leaking it?

   Model answer: Never on the command line - /proc/<pid>/cmdline is world-readable while the process runs, and it reaches shell history and process accounting. Use a permission-controlled file (ipmitool -f), the environment (-E), or the tool’s own credential helper.

8. A secret has been committed to a repository. What is the remediation?

   Model answer: Rotate it first. Removing the commit does not un-disclose it, and history rewriting is slow, partial and does not reach clones, forks or CI caches. Rotate, then clean history, then add detection so the next one is caught before it merges.

Section 9: Change, drift, immutable infrastructure and containers (8 written)

Part A questions d9-q1 to d9-q6 score the same domain.

1. What must a change plan contain?

   Model answer: Scope and blast radius, the procedure, verification with expected output, and a rollback with a measurable trigger and a named decider. The trigger and the decider are the parts usually missing, and they are the parts you need when the change has half worked.

2. What is the difference between a rollback and a fix forward, and how do you choose?

   Model answer: Rollback returns to a known state; fix forward moves to a new one. Choose rollback whenever the known state is reachable and safe, because it is the option with a tested outcome. Fix forward is correct when rollback is impossible - a completed migration, for example - and that fact should be in the plan before you start.

3. Why must a golden image be stripped before capture, and of what?

   Model answer: Because anything left becomes identical across every instance. Remove SSH host keys, /etc/machine-id, persistent network rules, logs, cached credentials and any DHCP leases. Duplicate host keys destroy host key verification for the whole fleet; a duplicate machine-id breaks journald and DHCP identity subtly.

4. What does “immutable infrastructure” change about operations?

   Model answer: Hosts are replaced rather than modified, so the image becomes the unit of change and drift has nowhere to accumulate. In exchange you need a fast, reliable image pipeline, and you need to stop fixing production by hand - a manual fix on an immutable host is lost at the next replacement and is the reason the fix “keeps coming back”.

5. Which cloud-init modules run once, and which run every boot?

   Model answer: Per-instance modules run once, keyed on the instance ID and recorded under /var/lib/cloud; per-boot modules run every time. Changing user-data on an existing instance therefore appears to do nothing. Rebuild the instance rather than clearing state to make a change stick.

6. What actually isolates a container?

   Model answer: Namespaces for what it can see (PID, mount, network, UTS, IPC, user) and cgroups for what it can consume. There is no guest kernel. Everything else - capabilities, seccomp, LSM policy - narrows the syscall surface it shares with the host.

7. What are lowerdir, upperdir, workdir and merged in OverlayFS?

   Model answer: lowerdir is the read-only base (possibly many layers), upperdir takes all writes, workdir is private scratch on the same filesystem as upperdir, and the merged view is the mount point. Container writes live in upperdir, which is why they vanish when the container is removed.

8. Is the capability bounding set per-process or system-wide?

   Model answer: Per-process, inherited across fork and exec, and reducible only. CapabilityBoundingSet= in a unit sets a ceiling a compromised service cannot climb back over, even via a setuid binary. Verify with systemctl show -p CapabilityBoundingSet.

Section 10: Incident response (6 written)

Part A questions d10-q1 to d10-q3 score the same domain.

1. What is the troubleshooting loop?

   Model answer: Observe, hypothesise, test the hypothesis with a check that can disprove it, then change one thing and measure. Repeat. The discipline is the falsifiable test and the single change; without them you accumulate changes and lose the ability to attribute the outcome.

2. Distinguish trigger, contributing factor and root cause.

   Model answer: The trigger started the clock. Contributing factors made it harmful or delayed detection. The root cause is the systemic gap that allowed the trigger to matter. Stopping at the trigger produces “be more careful”, which changes nothing.

3. How is incident severity assigned?

   Model answer: By user-visible impact and its scope, against a definition published in advance, decided at declaration time under uncertainty, and re-graded openly as the picture changes. Host counts mislead in both directions.

4. What must you capture before restarting a failing service?

   Model answer: Process state (stacks, open descriptors, memory maps), the journal tail for the unit, socket and queue state, and the exact restart time for correlation. Restarting destroys the evidence of why it failed, which is why restart-only responses recur.

5. When do you stop investigating and restore service?

   Model answer: When the impact of continuing exceeds the value of the evidence still to be gained - and the decision is explicit, announced, and recorded with what was captured first. Capture what is cheap even when you cannot capture everything.

6. What makes a post-incident review useful?

   Model answer: A timeline built from evidence rather than memory, the trigger and contributing factors separated from the root cause, action items with owners and dates that change the system rather than the people, and a blameless framing so the next person reports early instead of hiding the problem.

Section 11: Networking (8 written)

Networking is taught across Parts XIX–XXV (55 lessons). The assessment covers the production disciplines: prove the layer that is actually broken before changing the layer that might be, and treat every “the network is fine” assertion as a hypothesis to be tested, not a fact.

1. A user reports ssh db1 times out from the bastion but works from the office workstation. List, in order, the evidence you would collect on the bastion before changing anything on the server.

   Model answer: Capture the failed handshake with tcpdump -i any -n host db1 and port 22 -w /tmp/ssh.pcap while reproducing. Run ss -ltn '( sport = :22 )' and look at the Recv-Q on the listener. Check ip route get <db1> from the bastion and ip neigh get <db1>. journalctl -u sshd --since "5 min ago" on db1 for any rejection messages. Look for a stateful middlebox on the path (conntrack -L | grep db1 on any in-path router). Only after the layer that is broken is named is any change made on that layer.

2. A service reports DNS resolution works for the first hour after boot and then fails intermittently. Name the most likely cause and the evidence that distinguishes it from “the upstream resolver is intermittent”.

   Model answer: A stale cache and a resolver loop. systemd-resolved in its default configuration caches for Cache=yes and the negative TTL is honoured; a stale A record can survive a backing-service IP change. Distinguish from upstream intermittent by querying the configured resolver directly with dig +short @<resolver> <name> and comparing against resolvectl query. If the resolver returns a stale record but the upstream returns a fresh one, the cause is local cache, not the upstream. If both return the stale record, the upstream is the cause and TTL/serial issues are likely.

3. mtr -rwnbz -c 100 db1 from the bastion shows a single hop with 35% loss, and every hop after that hop shows the same 35% loss. What does this mean?

   Model answer: ICMP rate-limiting at the loss hop. Routers often de-prioritise ICMP under load, so a traceroute probe elicits a slow reply while the actual forwarded traffic passes through normally. Verify with mtr -T -P 22 db1 (TCP-based, port 22) or a port-specific probe (tcping, hping3 --tcp -p 22) and compare loss. If TCP-mode shows 0% loss where UDP-mode shows 35%, the network is healthy and the apparent loss is ICMP handling. Do not “fix” a router that is not the cause.

4. chronyc tracking on a server shows Last offset -22.3 sec. What is the immediate operational risk, and what is the correct response?

   Model answer: TLS certificates, Kerberos tickets and signed log timestamps all assume a sane clock. A 22-second offset means the host is failing TLS handshake validation against time-sensitive peers (notably OCSP stapling, short-lived cert chains and KDC pre-auth) and producing log records whose timestamps disagree with the rest of the fleet by hours. The correct response is to force a step (chronyc makestep) on the affected host, then investigate why the previous step failed: unreachable upstream NTP source (chronyc sources -v), a makestep threshold (chronyc makestep only steps below the threshold, otherwise slews), or an NTP daemon stopped by the package manager.

5. A 1500-byte packet reaches the host but the application reads only 1480 bytes per read. List three possible causes and the single command that distinguishes them.

   Model answer: (a) Generic Routing Encapsulation or an overlay network subtracting bytes from MTU, (b) a TSO/GSO offload fragmenting at the NIC differently from the application’s view, or (c) the application using a smaller receive buffer than the MSS. The single command that distinguishes them: ip route get <dst> shows the path MTU; tcpdump -i eth0 -nnvv -s0 'host <dst> and tcp' shows the actual on-wire packet sizes and the TCP MSS negotiated in the SYN; and ss -i shows the PMTU/ MSS the kernel is using for that flow. The application’s read size is unrelated to any of these unless the application is reading from a smaller socket buffer than the kernel’s advertised MSS.

6. A user reports that a TCP connection from a home network to a corporate application fails after exactly 60 seconds. List the single most common cause and the evidence that distinguishes it from a server-side failure.

   Model answer: A stateful middlebox aging out the flow after a 60-second idle. Many consumer-grade firewalls and ISP gateways have a 60-second TCP idle timeout. The TCP keepalive interval on most Linux defaults is 7200 seconds, so the connection looks alive to the kernel but is dead in the middlebox. Distinguish by capturing with tcpdump on the server: if the server sees SYN, SYN-ACK and then nothing for 60 seconds, the middlebox is the cause; if the server sees a RST from its own side, the application is closing. The fix is to enable TCP keepalives on both ends (sysctl net.ipv4.tcp_keepalive_time=60) or shorten the application’s idle timeout, not to increase the middlebox state.

7. A user reports that ping6 2001:db8::1 works but connecting to https://[2001:db8::1]/ does not. Name two distinct causes and the single command that distinguishes them.

   Model answer: (a) IPv6 routing in the application is correct but a firewall on the path blocks TCP/443 while allowing ICMPv6; (b) the application does not have a route for the destination but the OS does and responds to ICMPv6. Distinguish with ip -6 route get 2001:db8::1 (should show a route), then tcpdump -i any -nn -s0 'host 2001:db8::1 and tcp' while reproducing. If the SYN never appears on the wire, the issue is on the client side (routing or firewall); if the SYN appears but no SYN-ACK returns, the path is the issue. curl -6 -v https://[2001:db8::1]/ shows which side closes the connection.

8. A network capture shows retransmissions, out-of-order segments and a steadily growing tcpi_rtt. What does this combination mean, and what is the operational implication?

   Model answer: A path that is dropping or reordering packets while round-trip is increasing. This is a precursor to a flow stalling or being reset by a middlebox timeout. The operational implication is that the connection’s effective throughput has collapsed (TCP backs off exponentially under loss); the user sees the application as slow or hung. The investigation path: mtr -T -P <port> to the destination to localise loss, check the switches / WAN for errors (ifconfig eth0 or ethtool -S eth0 for rx_crc_errors and rx_missed_errors), and look at the recent change window (firmware, routing, optical level).

Section 12: Scenarios (8, rubric-marked)

The eight scenarios are defined with full rubrics in this assessment’s frontmatter, and each lists the evidence, remediation and rollback elements expected. Items marked REQUIRED are pass/fail on their own.

Mark a scenario as passed only if every REQUIRED element appears in your answer. A scenario answer that omits a REQUIRED data-loss or fencing element fails that scenario however good the rest is, because in production it would have destroyed data.

Pass criteria

• 70% overall: Part A auto-score, plus Part B self-marked at one point per prompt, plus 5 points per passed scenario.
• 60% per domain: every domain in the table above must reach 60% on its own. A hole in one domain fails the whole assessment.
• 6 of 8 scenarios passed against their rubrics.
• No REQUIRED data-loss or fencing element missed in any scenario answer.

The assessment tests the application of the curriculum to realistic scenarios. Memorisation is not enough; the candidate must demonstrate the discipline.