Linux · Self-assessment
Knowledge checks
Every knowledge check in this course, in curriculum order. Each link opens the page at its quiz. The questions are auto-graded in the browser and nothing is recorded — a wrong answer costs you only the explanation, which is the part worth reading.
- Knowledge checks
- 542
- Parts covered
- 83
- Of all lessons
- 100%
Part I
Foundations
6 checks
- Welcome to Linux for Production SysadminsA hands-on, production-oriented Linux course for systems administrators.→
- History and ecosystem of UNIX and LinuxFrom Bell Labs through GNU, the kernel, distributions, and licensing.→
- Kernel vs userland — the layered modelHardware, kernel, libraries, services, shell, applications, users.→
- The Filesystem Hierarchy StandardWhat lives where on a Linux system and why.→
- /proc, /sys, /dev, /run — the kernel-exposed filesystemsPseudo-filesystems that let userland talk to the kernel without ioctl.→
- Linux distributions and support lifecyclesDebian vs RHEL families, LTS releases, and how to plan for end of life.→
Part II
Shell and Command-Line Operations
10 checks
- Shell anatomy — commands, options, arguments, exit codesHow the shell parses what you type and reports what happened.→
- Quoting, expansion, and globbingHow the shell turns what you type into what the command receives.→
- Environment variables, PATH, and shell initialisationWhere variables come from, why PATH matters, and which startup files matter.→
- Redirection, pipes, and three-channel I/OConnecting commands and files through stdin, stdout, and stderr.→
- Exit codes, conditionals, and testsUsing $? to drive if, while, and logical operators.→
- Core text tools — grep, cut, sort, uniq, tr, wcThe high-frequency text manipulation toolkit.→
- sed and awk in productionStream editors for text reshaping and field-based extraction.→
- find, locate, and xargs — bulk operations on filesSelecting files and applying commands to them safely.→
- Shell history, aliases, and functionsProductivity patterns that pay back every day.→
- jq and structured text — JSON, YAML, and CSV at the command lineReading, querying, and reshaping structured data without a script.→
Part III
Filesystems and Files
6 checks
- Linux file types and the seven file kindsRegular, directory, symlink, device, socket, pipe, and what each means to a sysadmin.→
- Inodes, dentries, and the VFS — how Linux tracks filesWhat an inode actually stores and why `df` and `du` can disagree.→
- Hard links and symbolic linksTwo ways to make one file reachable from two paths; what each can and cannot do.→
- Permissions, ownership, and umaskThe read/write/execute bits, owners and groups, and the default-permission mask.→
- POSIX ACLs — fine-grained permissions beyond owner/group/otherWhen the standard three-class permissions are not enough.→
- Extended attributes, immutable attributes, and capabilitiesThe file-level metadata that lives outside the standard mode bits.→
Part IV
Users, Groups and Identity
6 checks
- /etc/passwd, /etc/shadow, /etc/groupHow Linux stores identity, what every field means, and what should never be edited by hand.→
- UIDs, GIDs, and supplementary groupsHow Linux identity numbers and group memberships actually work.→
- Password policies and account lockingchage, PAM, password aging, account expiration, and lockout.→
- Service accounts and system accountsDesigning accounts for daemons, CI, and shared services.→
- Name Service Switch (NSS) and getentHow Linux decides where to look for users, groups, and hosts.→
- Account lifecycle and offboarding - creating, changing and removing usersuseradd, usermod and userdel in production: what each tool touches, what it deliberately does not, and why a reused UID hands one person another person files.→
Part V
sudo and Privileged Access
6 checks
- Root vs sudo — the production privilege modelWhy sudo replaced direct root login and what it gives you.→
- sudoers syntax — rules, aliases, and defaultsThe grammar that decides who can do what.→
- Drop-in files, package-installed rules, and #includedirHow to keep sudoers modular, version-controlled, and package-aware.→
- Least-privilege sudo designDesigning sudoers rules that grant exactly the access required and nothing more.→
- Sudo logging, I/O capture, and SIEM integrationTurning every sudo invocation into an actionable audit trail.→
- Shell escapes, NOEXEC, and sudoeditWhy a sudoers rule that names a command without arguments is equivalent to ALL, and the three controls that actually bound a grant.→
Part VI
Processes
6 checks
- The Linux process model — PIDs, PPIDs, sessions, and groupsHow the kernel tracks running work and how a sysadmin observes it.→
- Process lifecycle — fork, exec, exit, and waitHow a process is created, what happens during execution, and how it terminates.→
- Signals — SIGTERM, SIGKILL, SIGHUP, and the restHow processes receive asynchronous notifications and what each signal means.→
- Threads and cgroup viewsThreads, thread groups, and how the kernel tracks many execution contexts in one process.→
- Nice, renice, and CPU scheduling prioritiesHow to politely ask the scheduler to give your process more or less CPU.→
- ps, pgrep, pkill, top, htop — the production toolkitThe commands that answer every "what is running on this host?" question.→
Part VII
systemd and Service Management
8 checks
- systemd architecture — PID 1 and the unit modelWhy systemd is PID 1, what units are, and how the dependency graph works.→
- systemctl and unit managementStart, stop, enable, mask, and inspect units.→
- Writing systemd unit filesAuthoring your own service, socket, and timer units from scratch.→
- Targets, dependencies, and orderingHow systemd's transaction model decides what runs in what order.→
- systemd timers — replacing cron with calendar and monotonic schedulesWhen systemd timers are better than cron, and how to write them.→
- systemd sockets, paths, and activationOn-demand service activation, file watchers, and the activation magic of systemd.→
- Drop-in overrides — modifying units without rewritingThe right way to make a partial change to a vendor-provided unit.→
- systemd-analyze, security review, and boot diagnosticsFind slow boot units, audit hardening, and verify security profiles.→
Part VIII
Logging and journald
6 checks
- Linux logging architecture — kernel, journald, syslogHow Linux captures, stores, and queries logs from every source.→
- journalctl — querying the systemd journalThe full power of journal filtering: by unit, time, priority, and field.→
- rsyslog, syslog-ng, and traditional syslogThe text-log pipeline that predates journald and still ships logs to /var/log.→
- logrotate — the rotation policyHow Linux manages text-log file growth, retention, and post-rotation actions.→
- Persistent journals and disk pressureWhy the journal sometimes fills the disk and how to size it.→
- Centralised logging — rsyslog forwarding and journald remoteShip every log to a central store before the host dies.→
Part IX
Boot Process
6 checks
- Firmware, UEFI, and BIOSThe pre-OS layer that hands control to the bootloader.→
- GRUB — the bootloaderHow GRUB loads the kernel and initramfs.→
- Kernel, initramfs, and the first userlandWhat the kernel does after GRUB hands off, and how the root filesystem is mounted.→
- systemd targets and rescue modeBoot into specific states for recovery and maintenance.→
- Emergency and rescue mode — when normal boot failsThe diagnostic paths when the system cannot reach multi-user.target.→
- Boot failure diagnosis from the OOB consoleReading the console output for the four common boot failure modes.→
Part X
Kernel Management
6 checks
- Kernel versions, releases, and lifecycleHow kernel versions work, what LTS means, and how to plan kernel updates.→
- Kernel modules — load, list, dependencies, blacklistHow Linux extends the kernel at runtime and how to manage the extensions.→
- sysctl — runtime kernel tuningHow to read and modify runtime kernel parameters.→
- Kernel logs, taint flags, and oops decodingReading dmesg output, recognising taint flags, and understanding kernel oops.→
- Live kernel patching — kpatch, kgraft, and livepatchHow to apply security kernel patches without rebooting.→
- Kernel command line parameters - the settings you can only make at bootThe three places a kernel knob can live, how the kernel splits its command line between itself, modules and init, and how to prove a parameter actually took effect.→
Part XI
Package Management
7 checks
- Package management overview — apt vs dnfHow package managers work, why they exist, and the production discipline around them.→
- apt and dpkg — Debian-family package managementThe day-to-day commands for installing, querying, and upgrading on Debian and Ubuntu.→
- dnf and rpm — RHEL-family package managementThe day-to-day commands for installing, querying, and upgrading on RHEL, Rocky, Alma.→
- Repositories, signing, and trusted sourcesHow packages are signed, why GPG keys matter, and how to configure a trusted repo set.→
- Package pinning, holds, and version selectionHow to pin specific versions in production and why.→
- Security updates and CVE responseHow to receive, test, and deploy security updates on a fleet.→
- Package cache, cleanup, and rollbackHow to manage disk space, package cache, and when things go wrong.→
Part XII
Repository Security and Supply Chain
6 checks
- Repository trust and GPG signingHow trust flows from a vendor key to an installed package, and what can break it.→
- Third-party repository riskWhen to trust a non-vendor repository, when not to, and how to contain the risk.→
- Package provenance and SBOMHow to know what is inside a package and where it came from.→
- Mitigating curl | bash and other supply-chain footgunsWhy the "curl | bash" pattern is dangerous and what to do instead.→
- Malicious packages and maintainer compromiseWhat a valid signature does and does not prove, how upstream compromise reaches a signed repository, and how to respond when a package you installed turns out to be backdoored.→
- Language package managers on production hostspip, npm and gem install code outside the distribution trust model. How dependency confusion works, why lockfiles are not hashes, and how to contain install-time code execution.→
Part XIII
Disks and Block Devices
6 checks
- Block devices, naming, and disk identificationHow Linux names disks and partitions, and what each name means.→
- Partitions, GPT, and MBRHow disks are divided and what each partition holds.→
- Filesystem signatures and wipefsHow the kernel detects and wipes filesystem signatures.→
- Persistent naming - UUIDs, labels, and stable pathsHow to reference disks in fstab without breaking under hardware changes.→
- Sector sizes and partition alignment - 512n, 512e, and 4KnLogical versus physical sector size, why a misaligned partition doubles every write, and how to read the topology a device reports.→
- Rescanning and growing block devices - when the kernel still sees the old sizeA LUN grew, a virtual disk grew, and lsblk has not noticed. The layers that each have to be told separately, and the ones where a mistake is unrecoverable.→
Part XIV
Filesystems
6 checks
- Filesystem concepts - journaling, allocation, and durabilityWhat a filesystem does and why ext4 / XFS / btrfs make different choices.→
- ext4 deep dive - features, tuning, and operationsHow ext4 works, what its tuning knobs do, and how to operate it.→
- XFS deep dive - features, tuning, and operationsHow XFS works, what its tuning knobs do, and how to operate it on large filesystems.→
- btrfs where it fits - snapshots, subvolumes, and operationsWhen to use btrfs and how to operate its advanced features.→
- Inode exhaustion - the filesystem that says noThe disk is not full, but the filesystem refuses to create files. Why, and how to fix it.→
- Filesystem check, repair, and recovery with fsck and xfs_repairWhen the filesystem itself is broken: how to check, repair, and recover safely.→
Part XV
/etc/fstab and Mount Management
6 checks
- fstab syntax - fields, options, and the dump/pass columnsHow /etc/fstab actually works and what each column means.→
- Mount options and systemd integrationHow mount options interact with systemd-managed mounts and the fstab generator.→
- Recovering from a broken fstab without a live USBHow to recover a host that will not boot because of an fstab entry.→
- Network filesystems in fstab - _netdev, automount, and the boot you cannot watchHow a network mount in /etc/fstab becomes a boot-time dependency on another host, and how to write the entry so a filer outage does not take the host down.→
- fstab entries that are not disks - bind mounts, tmpfs, and swapThe three fstab entry types that have no block device behind them, and the specific ways each of them fails.→
- Mount units by hand, and verifying a mount change before you rebootWhen to write a .mount unit instead of an fstab line, what the unit form loses, and how to check a mount change before the reboot that tests it for real.→
Part XVI
LVM
7 checks
- LVM concepts - PV, VG, LV, and the storage stackWhat LVM does, why it exists, and how the layers fit together.→
- LVM creation, extension, and snapshotsHow to create, extend, and snapshot LVM volumes safely.→
- LVM snapshots and rollback - the local safety netHow to use LVM snapshots for atomic rollback of changes.→
- LVM monitoring and capacity planningHow to monitor LVM capacity and plan growth before it becomes a problem.→
- LVM troubleshooting - common failures and how to recoverWhen LVM does not work as expected: missing PV, full VG, broken LV, and how to recover.→
- Shrinking an LV - the highest-risk LVM operationHow to reclaim space from a logical volume without destroying the filesystem, and why XFS cannot shrink at all.→
- LVM thin provisioning and thin-pool monitoringHow thin pools over-commit storage, why metadata exhaustion is worse than data exhaustion, and the two percentages you must alert on.→
Part XVII
Software RAID
6 checks
- mdadm RAID concepts - levels, layouts, and failure modesWhat RAID does, the common levels, and how to choose the right one.→
- RAID1 mirroring - simple and reliableHow RAID1 works and when to use it for production data.→
- RAID5 and RAID6 - parity for capacity and fault toleranceHow RAID5 and RAID6 work, when to use them, and the rebuild trade-offs.→
- RAID10 - mirrored stripes for performance and redundancyHow RAID10 combines striping and mirroring, and when to choose it.→
- Degraded arrays and rebuilds - replacing a failed diskHow to identify a degraded array, replace a failed disk, and monitor the rebuild.→
- md RAID monitoring and failure detectionMake an mdadm array tell you when a disk dies: persistence, mdadm --monitor alerting, periodic scrubs, write-intent bitmaps and node_exporter metrics.→
Part XVIII
Enterprise Storage
6 checks
- SAN vs NAS vs object storage - which one to chooseThe taxonomy of enterprise storage and how to choose the right one for the workload.→
- iSCSI - block storage over IPHow iSCSI works, how to connect, and how to operate it for production.→
- Fibre Channel - high-speed block storageHow Fibre Channel works and when to choose it over iSCSI.→
- NFS client - mounting and operating network filesystemsHow to mount, mount with options, and operate NFS for production.→
- Multipath and device-mapper - redundant paths to SAN LUNsHow multipath works, when to use it, and how to configure it for production.→
- Shared storage failure paths - reading a host whose storage has gone awayThe four shapes a shared-storage failure takes on the host, how to tell them apart when df itself hangs, and the order recovery has to happen in.→
Part XIX
Networking Foundations
8 checks
- OSI and TCP/IP models - what each layer is forHow the seven-layer OSI model and the four-layer TCP/IP model map to Linux sysadmin work.→
- Ethernet and MAC addressing - what is on the wireHow Ethernet frames carry MAC addresses, what those addresses look like, and how a frame traverses a LAN.→
- IPv4 addressing, subnetting, and CIDRHow IPv4 addresses are structured, how subnet masks and CIDR notation work, and how to size a subnet for production use.→
- IPv6 addressing and subnettingHow IPv6 addresses are structured, how the prefix length works, and why IPv6 matters in production.→
- ARP and IPv6 neighbour discoveryHow hosts map IP addresses to MAC addresses on a LAN, and why this is the layer where most "cannot reach" starts.→
- TCP and UDP - transport-layer essentialsHow TCP provides reliable streams, how UDP provides datagrams, and how Linux exposes both via ss and /proc.→
- ICMP and path MTU - control messages and fragmentationHow ICMP messages signal errors and reachability, and how path MTU discovery prevents fragmentation problems.→
- DNS and DHCP conceptsHow DNS resolves names to addresses, how DHCP assigns addresses, and why both are foundational for production.→
Part XX
Linux Network Configuration
7 checks
- The ip command - the sysadmin network toolkitHow ip replaced ifconfig and route, what its subcommands do, and the workflow for inspecting and changing state.→
- Linux interface configuration - addresses, MTU, offloadsHow to assign addresses, set MTU, and tune offloads on a Linux network interface.→
- Routing and default gateways on LinuxHow the routing table works, what a default gateway is, and how to add and debug routes on a Linux host.→
- Netplan on Ubuntu - declarative network configurationHow Netplan reads YAML and configures the network via NetworkManager or systemd-networkd.→
- NetworkManager - flexible configuration for desktops and laptopsHow NetworkManager handles wired, wireless, and VPN connections, and how to control it from the command line.→
- systemd-networkd - predictable configuration for serversHow systemd-networkd reads .network files and applies them, and why it is the right default for headless servers.→
- Distribution comparison - network configuration across distrosHow Debian, Ubuntu, RHEL, and SUSE differ in default network tooling, and what to expect when you move between them.→
Part XXI
Advanced Linux Networking
8 checks
- VLANs and 802.1Q - segmenting a network with tagsHow VLANs work at layer 2, what the 802.1Q tag does, and how to configure VLAN interfaces on Linux.→
- Linux bridges and bridge VLANsHow a Linux bridge connects layer-2 segments, and how bridge VLANs let you filter 802.1Q tags on a per-port basis.→
- Bonding and LACP - aggregating links for redundancy and throughputHow Linux bonding aggregates multiple links into one logical interface, and how LACP negotiates with the switch.→
- NIC teaming - the userspace alternative to bondingHow teamd differs from the kernel bonding driver, and when (rarely) it is the right choice.→
- MTU and jumbo frames - when 1500 is not enoughHow MTU affects throughput, why jumbo frames help, and what to do when a tunnel forces a smaller MTU.→
- Multiple routing tables on LinuxHow Linux supports multiple routing tables, and how the kernel picks which table to use.→
- Policy routing - choosing routes by source, mark, or interfaceHow Linux policy routing selects a route based on packet metadata, and the real-world cases where it is the right tool.→
- IPv6 advanced - privacy, SLAAC details, and operational gotchasHow IPv6 privacy extensions work, what SLAAC actually does, and the operational pitfalls unique to IPv6.→
Part XXII
Network Troubleshooting
9 checks
- Network troubleshooting methodology - the systematic approachHow to debug network problems systematically: define the symptom, isolate the layer, test the hypothesis.→
- ss and socket state - inspecting TCP and UDP from the kernelHow ss reads kernel state directly, what every TCP state means, and how to spot trouble.→
- ping, traceroute, and mtr - path diagnosisHow to use ping, traceroute, and mtr to find where a network path is broken, and how to interpret the output.→
- dig and nslookup - DNS diagnosis in detailHow to use dig and nslookup to query DNS, read answers, and trace resolution chains.→
- tcpdump and packet capture - seeing the bytesHow to capture packets with tcpdump, filter for what matters, and read the output to diagnose problems.→
- ethtool and offloads - link-layer diagnosticsHow to use ethtool to read and tune NIC settings, identify offload problems, and diagnose link-layer issues.→
- arping and ARP inspection - forcing layer-2 discoveryHow to use arping, arp-scan, and ip neigh to inspect and trigger ARP resolution on the local link.→
- nc and openssl s_client - probing TCP and TLSHow to use nc and openssl s_client to test TCP connectivity, send raw data, and probe TLS endpoints.→
- curl as a diagnostic tool - HTTP, headers, and timingHow to use curl to debug HTTP/HTTPS problems: verbose output, header inspection, timing, redirects, and TLS.→
Part XXIII
DNS
6 checks
- DNS resolver architecture - how Linux finds namesHow the Linux resolver library, NSS, and systemd-resolved interact, and how to configure them correctly.→
- /etc/resolv.conf - the resolver configuration fileHow /etc/resolv.conf works, who owns it, and how to configure it correctly without getting overwritten.→
- systemd-resolved - the modern stub resolverHow systemd-resolved provides per-interface DNS, caching, DNSSEC, and DNS-over-TLS on modern Linux hosts.→
- Recursive resolution - how a name becomes an IPHow the recursive DNS lookup chain works, from the root servers to the authoritative answer, and what each step does.→
- Authoritative DNS, TTLs, and caching behaviourHow authoritative DNS works, what TTLs control, and how to design TTL strategy for production zones.→
- Cluster DNS failure modes - what breaks when DNS is downHow DNS outages affect clusters, the cascading failures that follow, and how to design for DNS resilience.→
Part XXIV
Time Synchronisation
6 checks
- Time concepts - UTC, monotonic, and the wall clockHow Linux represents time, the difference between UTC and monotonic clocks, and why it matters for production.→
- chrony architecture - the modern NTP client and serverHow chrony works: sources, selection, slewing, and how to configure it as both client and server.→
- chrony client configuration - tuning for accuracy and resilienceHow to configure a chrony client for production: sources, makestep, drift, and what each directive does.→
- systemd-timesyncd - the lightweight NTP clientHow systemd-timesyncd provides basic NTP client functionality, and when to prefer it over chrony.→
- Time skew operational impact - what breaks when clocks driftHow time skew breaks TLS, Kerberos, databases, logs, and other production systems, and how to detect it.→
- NTP fleet design - hierarchy, falsetickers and leap secondsWhy the number of time sources matters more than their accuracy, how chrony decides one of them is lying, and what a leap second does to a fleet that has not decided how to handle it.→
Part XXV
Firewalls
9 checks
- Firewall concepts - what a Linux firewall actually doesHow Linux firewalls work at the kernel level, what hooks exist, and the difference between host and network firewalls.→
- nftables architecture - tables, chains, and rulesHow nftables is structured: tables, chains, hooks, and the syntax for rules.→
- nftables chains, tables, and hooks in depthHow to choose chains, priorities, and the right hook for each rule.→
- nftables stateful filtering and conntrackHow conntrack tracks connections, and how to use stateful matches in nftables.→
- nftables NAT - source NAT, destination NAT, and masqueradeHow Network Address Translation works in nftables: SNAT, DNAT, masquerade, and port forwarding.→
- iptables legacy and migration to nftablesHow iptables works, why it is deprecated, and how to migrate to nftables.→
- firewalld - zone-based firewall managementHow firewalld works: zones, services, and the relationship with nftables.→
- ufw and distribution firewall wrappersHow ufw and other distribution firewall wrappers work, and when to use them versus direct nftables.→
- Validating actual firewall behaviourHow to test that your firewall actually blocks what you think it blocks: external scans, nmap, and rule validation.→
Part XXVI
SSH
8 checks
- SSH architecture - how Secure Shell actually worksHow SSH provides confidentiality, integrity, and authentication: the protocol layers, key exchange, and channel model.→
- SSH keys and known_hosts - the key-based authentication modelHow SSH keys work, how to generate them, and how known_hosts protects against man-in-the-middle attacks.→
- ssh-agent and forwarding - keeping keys safeHow ssh-agent holds decrypted keys, how agent forwarding works, and the security implications of each.→
- sshd configuration - hardening the serverHow to configure sshd for security and reliability: authentication, ciphers, access controls, and logging.→
- SSH certificates and bastionsHow SSH certificates replace TOFU for large fleets, and how bastions protect production hosts from direct internet exposure.→
- ProxyJump and jump hosts - secure access patternsHow ProxyJump works, how to use it for bastion hosts, and how to combine it with port forwarding for complex topologies.→
- SSH hardening - production checklistA comprehensive SSH hardening checklist: ciphers, authentication, banners, timeouts, logging, and rate limiting.→
- SSH MFA and 2FA conceptsHow multi-factor authentication works with SSH: TOTP, FIDO2/U2F, hardware tokens, and the operational trade-offs.→
Part XXVII
Authentication and Enterprise Identity
8 checks
- PAM architecture - pluggable authentication on LinuxHow PAM (Pluggable Authentication Modules) works: stacks, modules, control flags, and the configuration files.→
- NSS - Name Service Switch for users, groups, and hostsHow NSS (Name Service Switch) lets the resolver look up users, groups, and hosts from multiple sources.→
- LDAP and Active Directory - directory services fundamentalsHow LDAP and Active Directory work: schemas, binds, searches, and how Linux integrates with them.→
- Kerberos concepts - tickets, realms, and principalsHow Kerberos authentication works: realms, principals, tickets, and the role of the KDC.→
- SSSD architecture - the system security services daemonHow SSSD provides NSS, PAM, and Kerberos integration for central identity on Linux.→
- SSSD and Active Directory integrationA complete guide to joining Linux to AD with SSSD, configuring Kerberos, and verifying single sign-on.→
- Central identity failure modes - what breaks when LDAP or AD is downHow to design central identity for resilience: cache lifetimes, fallback paths, offline auth, and what to do when the central server is down.→
- PAM and NSS troubleshooting - fixing auth that breaksA systematic approach to debugging PAM and NSS failures: stack inspection, log analysis, common errors, and recovery.→
Part XXVIII
SELinux and AppArmor
6 checks
- MAC vs DAC - mandatory versus discretionary access controlThe difference between Discretionary Access Control (DAC) and Mandatory Access Control (MAC), and why MAC matters for production.→
- SELinux architecture - contexts, policy, and enforcementHow SELinux works: labels, policy types, enforcement modes, and the tools to manage it.→
- SELinux contexts and labels - inspecting and fixingHow to read, set, and restore SELinux contexts on files, processes, and ports.→
- SELinux policy and modes - enforcing, permissive, and custom policiesHow SELinux policies work, how to switch modes safely, and when to write custom policy modules.→
- AppArmor concepts - path-based MAC for Debian-family systemsHow AppArmor works: profiles, modes, and how to manage them on Ubuntu and Debian.→
- MAC denial investigation - diagnosing SELinux and AppArmor denialsA systematic approach to diagnosing MAC denials: from denial to fix, using audit logs, AVC, and profile modes.→
Part XXIX
Linux Security Hardening
7 checks
- Linux hardening principles - defence in depthThe principles of host hardening: defence in depth, least privilege, fail-safe defaults, and the CIS Controls.→
- SSH and sudo hardening - the privileged access surfaceHow to harden SSH and sudo: the two most common paths to root on Linux and the controls that limit them.→
- Kernel hardening with sysctl - the runtime network and security parametersHow to harden the Linux kernel runtime parameters with sysctl: networking, ASLR, kernel pointers, and other security settings.→
- Mount options for security - nodev, nosuid, noexecHow filesystem mount options defend against privilege escalation and execution of untrusted code: nodev, nosuid, noexec, and more.→
- Service minimisation - reducing the attack surfaceHow to minimise the services running on a host: disable what is not needed, run what is needed in containers, and audit the listening surface.→
- Package update discipline - patching as a security controlHow package updates are a security control: frequency, automation, validation, and the trade-offs of unattended upgrades.→
- CIS benchmarks conceptually - mapping controls to the frameworkHow CIS Benchmarks are structured, what they cover, and how to apply them to a Linux host without breaking applications.→
Part XXX
Linux Capabilities and Privilege
6 checks
- Traditional root model - why capabilities existWhy the all-or-nothing root model is dangerous and how Linux capabilities split root powers into discrete pieces.→
- Linux capabilities overview - the full list and categoriesThe complete Linux capabilities, organised by category, and how to use them in practice.→
- File capabilities - replacing setuid with fine-grained privilegesHow to grant capabilities to binaries via file xattrs, replacing setuid for service privilege.→
- capsh and capability debugging - inspecting and dropping capabilitiesHow to inspect, decode, and manipulate process capabilities with capsh, getpcaps, and prctl.→
- Process capability sets and the exec transitionThe five per-thread capability sets, the kernel algorithm that recomputes them at execve, and why an inheritable capability on its own does nothing.→
- Root-equivalent capabilities - when dropping to one changes nothingWhich individual capabilities are a complete path back to root, how each one gets there, and how to review a capability grant instead of counting it.→
Part XXXI
Audit and Security Logging
6 checks
- auditd architecture - the Linux audit frameworkHow auditd works: rules, watch points, the audit log, and ausearch / aureport tools.→
- audit rules and watch points - what to recordHow to write audit rules for files, syscalls, and users; what to record and what to skip.→
- ausearch and aureport - querying the audit logHow to use ausearch and aureport to find specific events and summarise the audit log.→
- Authentication and sudo logs - what gets recorded and how to read itHow authentication events are logged in auth.log, secure, journald, and sudo.log; how to read and correlate them.→
- Central security logging - shipping logs to a SIEMHow to centralise auth.log, journald, sudo.log, and audit.log to a SIEM or central log server.→
- Audit record anatomy and attribution - answering whoReading a raw audit event across its several records, and using auid and ses to attribute an action to a human rather than to a shared account.→
Part XXXII
Vulnerability and Patch Management
6 checks
- Vulnerability concepts - CVEs, severity, and exploitabilityHow vulnerabilities are disclosed, classified by severity, and prioritised for patching.→
- CVE severity vs context - prioritising for the environmentHow to apply CVSS scores in context: asset value, exposure, compensating controls, and patch availability.→
- Scanning and detecting vulnerabilitiesTools and techniques for vulnerability scanning: package scanners, CVE matching, and continuous detection.→
- Patch priority decisions - timing and riskHow to decide when to patch, what risk to accept, and how to balance security against availability.→
- Emergency patching - reacting to critical CVEs under pressureHow to react to a critical CVE with active exploitation: triage, communicate, deploy, verify, post-mortem.→
- Reboot and restart requirements - finishing the patchInstalling the package does not stop the vulnerable code running. How to detect deleted-but-mapped libraries, pending reboots and stale kernels, and how to decide what actually has to restart.→
Part XXXIII
Fleet Patch Management
6 checks
- Fleet patch strategy - waves, canaries, and rollbackHow to patch a fleet of hosts safely: dev/test/canary/production waves, health validation, and automated rollback.→
- Canary and wave deployments - phased rollout with health checksHow to deploy changes to a small subset first, validate, then expand - the canary and wave patterns.→
- Rollback strategies - safe deployment reversalsHow to roll back a deployment safely: package rollback, configuration revert, container redeploy, and database migration reversal.→
- Pre-patching and post-patching validationHow to validate a patching operation before, during, and after - health checks, smoke tests, and rollback triggers.→
- Automated patching in a fleet - what to automate and what never toThe trade-off automation makes against the wave strategy, how to keep automated patching staged, and the guardrails that stop it becoming a fleet-wide simultaneous change.→
- Repository snapshots - making a wave reproducibleWhy a canary that validated version N tells you nothing about a production wave that installs N+1, and how point-in-time repository snapshots turn promotion into a metadata change.→
Part XXXIV
Configuration Management
6 checks
- Why manual does not scale - the case for configuration managementHow manual configuration breaks down at scale: drift, snowflake servers, error-prone changes, and the discipline that fixes it.→
- Desired state and idempotency - the foundations of CMHow desired-state configuration works, why idempotency matters, and how to write idempotent CM code.→
- Ansible for sysadmins - the modern CM toolHow Ansible works: inventory, modules, playbooks, and the most common patterns for sysadmins.→
- Inventory and facts - knowing your fleetHow Ansible inventory and facts work: static inventory, dynamic inventory, and gathered host facts.→
- Config drift detection - keeping reality aligned with declared stateHow to detect configuration drift: comparing hosts, scheduled checks, alerting on deviation, and remediation.→
- Safe CM rollout - blast radius, check mode, and the control nodeConfiguration management applies a mistake to the whole fleet at machine speed. The pre-flight ladder that contains it, what check mode genuinely proves, and why the control node is the most valuable host you own.→
Part XXXV
Shell Scripting for Sysadmins
7 checks
- Script structure and conventions - the production shell scriptHow to write production shell scripts: structure, conventions, error handling, and documentation.→
- Variables and quoting - the foundation of safe shell scriptsHow variables and quoting work in bash: expansion, special variables, quoting rules, and common pitfalls.→
- Conditions and loops - controlling script flowHow if/case/for/while work in bash, with patterns for production scripts.→
- Functions and arguments - reusable, testable script componentsHow to write functions in bash: arguments, return values, local variables, and testing.→
- Error handling - set -euo pipefail and trapHow to handle errors in production shell scripts: set -euo pipefail, trap, and exit codes.→
- Traps and temporary files - clean exit on signal or errorHow to use trap for cleanup, mktemp for safe temporary files, and atomic operations.→
- ShellCheck and static analysis - catching bugs before runtimeHow to use ShellCheck to find common shell bugs and style issues, and integrate it into your workflow.→
Part XXXVI
Scheduled Operations
6 checks
- cron syntax and anacron - the classic schedulerHow cron works: syntax, crontab fields, anacron for missed jobs, and common production patterns.→
- systemd timers replacing cron - the modern schedulerHow systemd timers work: calendar and monotonic timers, OnCalendar syntax, and replacing cron jobs.→
- Locking and concurrency - preventing duplicate jobsHow to prevent two instances of a job from running simultaneously: flock, lock files, and atomic operations.→
- Job logging and failure detection - knowing what happenedHow to log scheduled jobs so that success and failure are visible: stdout/stderr capture, exit codes, and alerting.→
- Timeouts, retries and jitter - scheduled jobs that fail safelyBound a scheduled job with RuntimeMaxSec, retry it deliberately, and spread a fleet-wide schedule with RandomizedDelaySec so 400 hosts do not fire at once.→
- Auditing the schedule - what actually runs on this hostEnumerate every scheduled job on a host across cron, systemd timers, at and per-user crontabs, then reconcile the inventory against what the team believes is running.→
Part XXXVII
Resource Management
6 checks
- cgroups v2 architecture - the Linux resource control subsystemHow cgroups v2 works: controllers, hierarchies, and how systemd uses them to manage resources.→
- ulimits and limits.conf - per-user resource limitsHow ulimits work, how /etc/security/limits.conf sets per-user limits, and how to apply them in production.→
- File descriptors and /proc/fd - understanding open filesHow file descriptors work on Linux, how to inspect them, how deleted-but-open files hold disk space, and how to diagnose "too many open files" errors.→
- systemd resource controls - applying cgroups via unitsHow systemd uses cgroups to control CPU, memory, IO, and tasks for every service.→
- OOM killer behaviour - what happens when memory runs outHow the OOM killer works: triggers, victim selection, configuration via sysctl, and how to investigate OOM kills.→
- cgroup accounting - proving which limit was actually hitRead cpu.stat, memory.events, memory.peak, pids.current and io.stat to show which resource limit a service hit, instead of inferring it from a symptom.→
Part XXXVIII
Linux Performance Fundamentals
6 checks
- USE methodology - the framework for performance investigationBrendan Gregg's USE method: check Utilisation, Saturation, and Errors for every resource, every time.→
- top, htop, and vmstat - the basic performance toolsHow to use top, htop, and vmstat to inspect CPU, memory, and process state on a running host.→
- mpstat and pidstat - per-CPU and per-process statisticsHow to use mpstat for per-CPU stats and pidstat for per-process stats, with focus on context switches and CPU usage.→
- iostat and sar - storage and historical performanceHow to use iostat for disk I/O stats and sar for historical performance data over time.→
- free, slabtop, and /proc/meminfo - memory diagnosticsHow to use free, slabtop, and /proc/meminfo to understand Linux memory: cache, anonymous memory, slab, and reclaim.→
- Measure, change one thing, measure again - the tuning disciplineRED for services and USE for resources, why a single before/after pair proves nothing, and the change record that makes a tuning session reversible.→
Part XXXIX
CPU Performance
6 checks
- CPU utilisation and load average - measuring the workloadHow to interpret CPU utilisation, %user vs %system vs %iowait, and load average as a saturation metric.→
- Run queue and context switching - measuring contentionHow the kernel scheduler manages runnable processes, what context switches mean, and how to spot contention.→
- Interrupts, softirqs, and steal time - hardware and VM signalsHow interrupts work on Linux, what softirqs are, and how steal time signals VM overcommit.→
- CPU affinity and NUMA - controlling where processes runHow CPU affinity and NUMA topology affect performance, and how to use taskset and numactl.→
- CPU frequency, governors and thermal throttling - the clock is not constantRead the cpufreq driver, governor and achieved frequency, distinguish a power policy from a thermal limit, and explain why the same benchmark returns a different number every run.→
- perf - finding the code that is burning the CPUSample a running system with perf top, perf record and perf stat, read a call graph, and fix the missing symbols and permission failures that make the first attempt useless.→
Part XL
Memory Performance
6 checks
- Virtual memory and pages - how Linux manages memoryHow virtual memory works: pages, page tables, the MMU, and how Linux maps virtual to physical addresses.→
- Page cache and anonymous memory - the two memory kindsThe difference between page cache (file-backed) and anonymous memory, and how the kernel reclaims each.→
- Swap and memory pressure - when the system runs out of RAMHow swap works, when to use it, how to tune swappiness, and how to detect memory pressure before OOM.→
- OOM killer decisions - who gets killed and whyHow the OOM killer selects victims: heuristics, oom_score, oom_score_adj, and how to influence the decision.→
- RSS, PSS and shared pages - what a process really costsWhy summing RSS across processes over-counts memory, how to read smaps_rollup for a defensible per-process figure, and how to estimate a working set.→
- Kernel memory, huge pages and the tuning knobs that backfireDiagnose unreclaimable slab growth, read the THP counters in /proc/vmstat, and understand why drop_caches, swappiness and overcommit rarely do what the tuning guide claims.→
Part XLI
Storage Performance
6 checks
- I/O latency, throughput, and IOPS - the three storage metricsThe three primary storage performance metrics: latency (how long?), throughput (how much?), IOPS (how many?).→
- Queue depth and utilisation - storage saturation signalsHow queue depth and device utilisation indicate storage saturation, and how to read them in iostat.→
- iotop and pidstat - per-process I/O statisticsHow to use iotop and pidstat to identify which process is causing storage I/O.→
- fio and safe benchmarking - measuring storage baselineHow to use fio for safe storage benchmarking: setting up a test, choosing parameters, and interpreting results.→
- The I/O stack - which layer is adding the latencyApplication, page cache, filesystem journal, block layer and device each add latency, and iostat only sees one of them. Read f_await, /sys/block stat and per-layer evidence.→
- Writeback and filesystem tuning - dirty pages, mount options and fstrimSize the dirty page limits in bytes rather than percent, choose atime and journal options from measurement, and run discard as a weekly batch instead of inline.→
Part XLII
Network Performance
6 checks
- Bandwidth and latency - the two network metricsThe two primary network metrics: bandwidth (how much?) and latency (how long?), and how to measure them.→
- Packet loss and retransmits - when the network dropsHow to detect packet loss and TCP retransmits, what causes them, and how to fix the underlying problem.→
- Socket queues and connection states - TCP tuningHow TCP buffers work, what the listen backlog means, and how to tune for high-connection workloads.→
- iperf3 and throughput testing - measuring bandwidthHow to use iperf3 to measure network bandwidth, latency, and jitter between two hosts.→
- The bandwidth-delay product - why a fast link runs slowCompute the BDP, read cwnd, rtt and delivery_rate from ss -ti, and decide whether a long fat link needs bigger windows or a different congestion control.→
- sar -n - network history and the counters nobody collectedRead historical network data from sysstat, compute interface utilisation against link speed, and check that the counters you need were actually being recorded.→
Part XLIII
eBPF and Advanced Observability
6 checks
- eBPF overview - in-kernel observability for sysadminsWhat eBPF is, how it works, and the sysadmin use cases that matter: tracing, networking, security.→
- Tracepoints, kprobes, and uprobes - eBPF attachment pointsHow eBPF programs attach to kernel and user events: tracepoints, kprobes, uprobes, and their use cases.→
- bpftrace and bcc - sysadmin use cases and examplesPractical bpftrace and bcc scripts for common sysadmin questions: latency, packet drops, process activity.→
- BPF maps and state - communicating with userspaceHow BPF maps work: storing data, communicating between kernel and userspace, and patterns for production eBPF tools.→
- Running eBPF safely - kernel requirements, permissions and lockdownCheck kernel config, BTF, capabilities and lockdown state before an incident, and read the specific errors bpftrace, bcc and bpftool produce when one of them is missing.→
- The cost of tracing - eBPF overhead, dropped events and production safetyProbe types differ in cost by orders of magnitude. Aggregate in the kernel, bound what you attach, read the dropped-event warnings, and know which probes not to put on a hot path.→
Part XLIV
Central Monitoring
6 checks
- Monitoring design - what to monitor, how to alert, and whyHow to design a monitoring system: the four golden signals, RED method, USE method, and what to monitor on Linux.→
- Prometheus and node_exporter - the standard Linux monitoring stackHow Prometheus and node_exporter work: the pull model, metric format, and exposing host metrics.→
- Grafana and alerting - dashboards and notificationHow to set up Grafana for dashboards and configure alerting with Alertmanager.→
- What to monitor on Linux - the production checklistA practical checklist of what to monitor on every Linux host: CPU, memory, disk, network, services, security events.→
- Hardware monitoring sensors - temperature, voltage, fanHow to read hardware sensors: temperature, voltage, fan speed, and how to alert on hardware problems.→
- Operating the monitoring stack - retention, cardinality and who watches the watcherPrometheus as a production service: sizing the TSDB, finding the metric that exploded, the scrape limits that protect it, safe reloads, maintenance silences and the dead man switch.→
Part XLV
Central Logging
6 checks
- Central logging design - the architecture and decisionsHow to design a central logging system: sources, transport, processing, storage, and access.→
- rsyslog forwarding - traditional syslog shippingHow to configure rsyslog to forward logs to a central system, with TLS, queues, and reliability.→
- Fluent Bit and Vector - modern log shippersHow to use Fluent Bit and Vector for modern log shipping: structured data, parsing, and routing.→
- Loki and Elasticsearch - the modern log storeHow to deploy Loki and Elasticsearch as the central log store, with retention and search tuning.→
- Log cardinality and capacity - sizing and tuningHow to design log cardinality for performance, plan capacity for log volume, and tune the store for production.→
- Where log messages go missing - and how to prove they did notEvery hop in a central logging pipeline can drop a message for a different reason and report it in a different place. The seven drop points, how to see each one, and the canary that verifies the whole path end to end.→
Part XLVI
OpenTelemetry
6 checks
- OpenTelemetry three pillars - metrics, logs, tracesHow OpenTelemetry unifies metrics, logs, and traces under one framework for observability.→
- OTel collectors and agents - the deployment modelHow to deploy OpenTelemetry collectors and agents: the agent/gateway pattern, OTLP, and the host receiver.→
- OTel host receiver and process metricsHow to use the OTel host receiver for host metrics and the process receiver for per-process metrics, replacing node_exporter.→
- OTel into Grafana and Tempo - the visual stackHow to ship OTel data to Grafana for visualisation and Tempo for traces: integration, dashboards, and queries.→
- OTel logs on a Linux host - the journald and filelog receiversHow logs actually reach an OpenTelemetry pipeline on a Linux host: the journald and filelog receivers, parsing and severity, offset checkpointing, and the rotation traps that duplicate or lose lines.→
- The cost of telemetry - cardinality, sampling and collector limitsWhy an OTel pipeline is a capacity decision: how metric cardinality multiplies, how to drop attributes at the agent, head versus tail sampling for traces, and the collector metrics that tell you it is dropping data.→
Part XLVII
Backup Strategy
6 checks
- Backup concepts - what to back up and howThe fundamentals of backup: file vs application, full vs incremental, retention, and the difference between backup and DR.→
- Application-consistent vs crash-consistent backupsThe difference between application-consistent and crash-consistent snapshots, and when each matters.→
- 3-2-1 and modern backup strategiesThe classic 3-2-1 backup rule and its modern variations: 3-2-1-1-0, immutable backups, and air-gapped copies.→
- Immutable and offline copies - the last line of defenceWhy immutable and offline backups matter, and how to implement them with S3 Object Lock, tape rotation, and air-gapped disks.→
- Encryption and key management - protecting backups at restHow to encrypt backups at rest, manage encryption keys, and recover them in a DR scenario.→
- Backing up what is not a file - partition tables, LVM metadata, LUKS headers and package stateThe restore that fails rarely fails on file data. It fails on the structure the files were living in, which nothing backed up because it was never a file.→
Part XLVIII
Backup Tools
6 checks
- rsync and tar - the classic backup toolsHow to use rsync and tar for backup: incremental, network-efficient, and the building blocks of backup strategies.→
- BorgBackup and Restic - the modern deduplicated backup toolsHow to use BorgBackup and Restic for modern, deduplicated, encrypted backups with snapshots and efficient storage.→
- Filesystem snapshots as backup - ZFS and btrfsHow to use ZFS and btrfs snapshots for backup: instant, space-efficient, but crash-consistent.→
- LVM snapshots for backups - the classic Linux approachHow to use LVM snapshots for backup: instant creation, copy-on-write, but with a watch-out for production impact.→
- Enterprise backup integration - Veeam, NetBackup, and similarHow enterprise backup tools integrate with Linux: Veeam Agent for Linux, Veritas NetBackup, Rubrik, and the agent-based model.→
- Backup tool selection - choosing the right tool for the workloadHow to choose the right backup tool for your workload: size, application type, retention, compliance, and operational fit.→
Part XLIX
Restore
6 checks
- Restore matters more than backup - the discipline of restoreWhy restore is more important than backup, and the discipline that makes restores reliable when they matter most.→
- Restore testing discipline - the regular validation routineHow to run restore tests regularly, what to look for, and how to use the results to improve the backup system.→
- Restore files and ownership - the practical detailsHow to restore individual files, preserve ownership and ACLs, and verify the result.→
- Restore services and configuration - the full-system recoveryHow to restore an entire service or host: configuration, application data, dependencies, and verification.→
- Partial and point-in-time restores - one file, one directory, one momentMost restores are not a rebuild. They are one file from Tuesday, and the hard parts are finding the right version and not damaging the live data while you do it.→
- Restore drills - turning an asserted RTO into a measured oneThe RTO in the DR plan is a number somebody chose. This is how to replace it with a number you observed, and why the observed one is always larger.→
Part L
Disaster Recovery
6 checks
- RPO and RTO - modelling recovery objectivesHow to model Recovery Point Objective (RPO) and Recovery Time Objective (RTO) and design a backup strategy that meets them.→
- Disaster scenarios - what to plan forCommon disaster scenarios for Linux hosts: hardware failure, ransomware, datacenter loss, and how to plan for each.→
- Bare-metal recovery - restoring to a fresh hostHow to recover a Linux host from bare metal: installation, configuration, data restore, and validation.→
- Dependencies: DNS, certificates, identity - the cascade risksHow DNS, certificates, and identity dependencies cascade in DR: if one is down, the whole recovery can fail.→
- Full cluster loss recovery - the worst-case scenarioHow to recover from total cluster loss: data centre failure, ransomware, or region outage. The runbook and the discipline.→
- Declaring a disaster, failing over, and failing backWho decides it is a disaster, which step makes failover irreversible, how to stop both sites serving at once, and the failback nobody plans for.→
Part LI
Linux Fleet Architecture
6 checks
- Fleet management plane - how a Linux fleet is operatedThe architecture of a Linux fleet: management plane, configuration, identity, monitoring, and how they work together.→
- Naming and inventory - knowing your fleetHow to name, identify, and inventory Linux hosts: hostname conventions, host IDs, and the inventory database.→
- Environment separation - dev, staging, and productionHow to separate environments: dev, staging, production. Why it matters and how to enforce it.→
- Central identity and secrets - the foundation of fleet securityWhy central identity and secrets management are critical: where to put them, how to operate, and the common pitfalls.→
- Host enrolment and decommissioning - joining and leaving the fleetThe transaction that connects a host to every part of the management plane, the bootstrap trust problem, and why decommissioning is the half that rots.→
- Scaling the management plane - what breaks between 50 and 5000 nodesThe specific walls a fleet architecture hits at each order of magnitude: push fan-out, catalogue compilation, metric cardinality, log volume, credential issuance rate, and the recovery herd.→
Part LII
High Availability Fundamentals
6 checks
- Availability and reliability - the metrics that matterHow to define and measure availability: the 9s, MTBF, MTTR, and the cost of each.→
- Redundancy and fault tolerance - building for failureHow to design for failure: redundancy at each layer, fault tolerance patterns, and the trade-offs.→
- Failure domains - what fails togetherHow to identify failure domains and design across them: power, network, hardware, software, and human.→
- N+1 capacity - the math of headroomHow N+1 capacity planning works: failure capacity, maintenance capacity, growth, and the 90%-on-3-nodes trap.→
- Host availability vs service availability - measuring the right thingWhy a host that was up 99.99% of the month can front a service that missed its target, and how to compose availability across a dependency chain.→
- Redundancy levels - N+1, N+2, 2N and 2N+1What each redundancy level costs, what it survives, and why raising the level on a small cluster is disproportionately expensive.→
Part LIII
Quorum and Split Brain
6 checks
- Quorum concepts - the math of agreementHow quorum works in distributed systems: majority, partition, and the requirement for an odd number of nodes.→
- Membership and partitions - the failure mode that breaks quorumHow membership changes and network partitions affect quorum: split brain, isolated nodes, and the recovery.→
- Split brain explained - the cluster failure modeHow split brain happens, why it is dangerous, and how quorum plus fencing prevents it.→
- Witness and quorum devices - breaking geographic tiesHow witness and quorum devices break ties when a cluster is split across sites with even node counts.→
- Failure detection - how a cluster decides a node has failedToken timeouts, monitor operations and no-quorum-policy: how a cluster infers failure from silence, and what it costs to infer it faster.→
- Two-node clusters and tie-breaking - the honest versionWhat two_node, wait_for_all, auto_tie_breaker and last_man_standing actually do, why a two-node cluster depends entirely on fencing, and how to break the fence race.→
Part LIV
Fencing and STONITH
6 checks
- Why fencing exists - the cluster disciplineWhy fencing is required for any healthy cluster: preventing data corruption from the minority group.→
- Fencing devices and agents - the practical implementationHow fencing devices work: IPMI, BMC, iLO, iDRAC, PDU, and the agents that drive them.→
- STONITH and data integrity - the shoot-the-other-node-in-the-head patternHow STONITH works, what it guarantees, and how it integrates with Pacemaker for production clusters.→
- Unresponsive is not dead - the evidence problem fencing solvesWhy a node that stopped answering cannot be assumed to have stopped writing, what counts as evidence that it stopped, and how the zombie writer corrupts a filesystem.→
- SBD and watchdog fencing - when the node fences itselfDisk-based and diskless SBD, the hardware watchdog that enforces it, why softdog is not a fencing device, and the timeout relationships that make it safe.→
- Fencing design principles - evaluating any platformThe four properties a fencing method must have, how to write a fence path per failure mode, and the anti-patterns that pass design review.→
Part LV
Pacemaker and Corosync
6 checks
- Corosync architecture - the cluster membership layerHow Corosync works: membership, quorum, and the transport for cluster communication.→
- Pacemaker architecture - the resource managerHow Pacemaker works: resource agents, constraints, and the policy engine that decides where resources run.→
- Pacemaker resources and resource agentsHow Pacemaker resource agents work: the operations, the standard agents, and how to write a custom one.→
- Pacemaker constraints: location, colocation, orderHow Pacemaker constraints shape resource placement: location, colocation, and order, with examples.→
- Pacemaker fencing integration - tying STONITH to resourcesHow Pacemaker uses STONITH: location, topology, and the relationship between fencing and resource placement.→
- Pacemaker troubleshooting - the systematic approachA systematic approach to troubleshooting Pacemaker clusters: from symptom to root cause, with the common failure modes.→
Part LVI
Keepalived and VRRP
6 checks
- VRRP concepts - virtual router redundancy protocolHow VRRP works: virtual router, master and backup, and the election when the master fails.→
- Keepalived configuration - the Linux VRRP implementationHow to configure keepalived on Linux: VRRP instance, virtual IP, priorities, scripts, and the production-grade setup.→
- Keepalived health checks - service-aware failoverHow keepalived monitors service health and triggers failover: HTTP checks, TCP checks, and custom scripts.→
- Keepalived limitations - when VRRP is not enoughThe limits of keepalived: same-subnet requirement, basic health checks, no application awareness, and when to use a load balancer instead.→
- Virtual IP mechanics - what actually moves during a failoverHow a VIP moves at layer 2: gratuitous ARP, neighbour caches, virtual MACs, why clients keep talking to the dead node, and the conntrack state that does not follow.→
- Sync groups and multiple VIPs - keeping a service togetherWhen one VRRP instance is enough, when you need a sync group, why dual-stack needs two instances, and how a sync group amplifies a flapping check.→
Part LVII
Linux Load Balancing
6 checks
- Load balancing concepts - distributing traffic across backendsHow load balancing works: layer 4 vs layer 7, algorithms, health checks, session persistence, and the common patterns.→
- HAProxy configuration - the production-grade layer 7 LBHow to configure HAProxy for production: frontends, backends, health checks, session persistence, and the production discipline.→
- nginx as a load balancer - the versatile optionHow to use nginx as a layer 7 load balancer: upstream, health checks, and the trade-offs vs HAProxy.→
- IPVS and the kernel load balancer - the layer 4 workhorseHow IPVS works inside the Linux kernel: layer 4 load balancing, the algorithms, and when to use it over HAProxy.→
- Health checks and persistence - the safety net of load balancingHow health checks and session persistence make load balancing reliable: active vs passive checks, sticky sessions, and the trade-offs.→
- Connection draining - taking a backend out without dropping requestsThe drain, maint and ready states in HAProxy, weight zero in IPVS, graceful reloads in nginx, and why session persistence makes a drain take far longer than the in-flight requests suggest.→
Part LVIII
Clustered Service Architecture
6 checks
- Stateless services - the simplest cluster architectureWhy stateless services are the foundation of scalable clusters: properties, examples, and how to design them.→
- Shared state and replicated state - when stateless is not enoughHow to handle state in clustered services: shared storage, replication, leader election, and the patterns that work.→
- External state and databases - the stateful backendHow to keep cluster state in external systems: databases, caches, and object stores, and the patterns that work.→
- Application architecture constraints - the limits of clusteringHow application design limits what a cluster can do: statefulness, dependencies, and the patterns that work.→
- Identifying the right cluster pattern - a decision procedureFive questions that classify any service into stateless, active-passive, active-active, external-state or leader-elected, with worked examples and the case for not clustering at all.→
- Failover as the client sees it - the outage the cluster does not measureWhy a twenty-second failover produces a fifteen-minute client outage, what happens to in-flight requests, and how to measure the RTO that users actually experience.→
Part LX
Distributed Storage Concepts
6 checks
- Ceph architecture intro - the modern distributed storageHow Ceph works: RADOS, OSDs, MONs, MDS, and the components that make up a modern distributed storage system.→
- Gluster concepts intro - the other distributed filesystemHow GlusterFS works: volumes, bricks, translators, and when to use it over Ceph or SAN.→
- Object and block storage tradeoffs - the right tool for the dataHow to choose between object storage and block storage: access patterns, performance, consistency, and the use cases for each.→
- Replication, quorum, and consistency - the distributed storage disciplineHow distributed storage systems replicate data and maintain consistency: replication factor, quorum, and CAP theorem in practice.→
- Failure domains in distributed storage - where the replicas actually landHow CRUSH rules and Gluster brick order decide which physical failures you survive, and the arithmetic that makes a three-replica pool undersized forever.→
- Recovery and rebalancing - what a distributed store does after a failureDegraded versus misplaced, the out timer, recovery throttles that modern Ceph ignores, and resolving a Gluster split brain by naming the survivor.→
Part LXI
DRBD Concepts
6 checks
- DRBD architecture - the distributed replicated block deviceHow DRBD works: block-level replication between two hosts, primary/secondary, and when to use it for HA.→
- DRBD primary/secondary and split-brain handlingHow DRBD handles primary/secondary roles, how split brain is really detected on reconnect, and how to recover with --discard-my-data.→
- DRBD in a cluster - integration with Pacemaker for HAHow to integrate DRBD with Pacemaker for a complete HA solution: resource configuration, constraints, and failover testing.→
- DRBD resync and online verification - reading the replication stateHow to read DRBD disk and connection states, tune the resync controller, and prove the two copies are actually identical with online verification.→
- Dual-primary DRBD - what it actually requiresWhen both DRBD nodes may be Primary at once, why that only works with a cluster filesystem, and the three preconditions that are not optional.→
- When DRBD materially helps HA - and when it does notChoosing between DRBD, shared storage, distributed storage and application-level replication, with the write-latency and RPO arithmetic that decides it.→
Part LXII
Cluster Networking
6 checks
- Cluster network design - management, application, and storageHow to design a cluster network: separate management, application, and storage networks, and the production considerations.→
- Management vs application vs storage - the network rolesWhy the three cluster networks (management, application, storage) must be separate: the consequences of mixing, and the production patterns.→
- Redundant interfaces and switches - the cluster network resilienceHow to add redundancy at the network layer: NIC bonding, switch stacking, and dual uplinks for HA clusters.→
- VLANs and failure domains in cluster networksHow to use VLANs in cluster networks: trunk ports, access ports, and the failure domain design.→
- Multicast and unicast on the cluster networkWhy Corosync moved off multicast, what an IGMP querier has to do with a cluster that dies after five minutes, and what the knet transport needs from the network.→
- Latency, jitter and cluster membershipHow the totem token budget turns network delay into fenced nodes, why jitter matters more than average latency, and how to measure and tune it without making things worse.→
Part LXIII
Cluster Time, DNS and Identity Dependencies
6 checks
- DNS cluster failure impact - what happens when DNS is downHow DNS failures cascade in a cluster: services that depend on DNS, the symptoms, and the mitigation.→
- NTP skew cluster impact - the silent time bombHow NTP skew affects cluster services: Kerberos, TLS, database replication, and the symptoms of a skewed cluster.→
- LDAP and certificate expiry cluster impact - the silent credential failuresHow LDAP and certificate expiry affect cluster services: authentication failures, expired services, and the recovery patterns.→
- Mapping cluster dependencies - finding what you depend onHow to discover a cluster external dependencies from the running system rather than from memory, classify them as hard or soft, and record the blast radius of each.→
- Designing around dependency failures - remove, cache, degrade, break glassThe four strategies for surviving a dependency outage, how to choose fail-open or fail-closed per dependency, and why circular dependencies only surface during a cold start.→
- Testing dependency failures - injecting the outage safelyHow to exercise a dependency outage with a bounded blast radius and an automatic rollback, why grey failures matter more than hard ones, and what to measure.→
Part LXIV
Rolling Maintenance
6 checks
- Rolling maintenance workflow - the loop that keeps a cluster servingThe go/no-go gate before the first node, the drain-change-validate-return loop, and the rule that the previous node must be fully back before the next one is touched.→
- Maintenance mode and drain - the safe cluster updateHow to put a node in maintenance mode, drain traffic, and bring it back. The safe cluster update pattern.→
- Health validation after a change - the safety netHow to validate that a cluster change is healthy: smoke tests, monitoring checks, and the discipline of verified changes.→
- Batch sizing and rollback - the change control disciplineHow to choose batch size for cluster changes and have a tested rollback plan. The change control discipline for HA clusters.→
- Draining behind a load balancer - rolling maintenance without a cluster managerHow to take a node out of rotation when the traffic director is HAProxy, nginx or keepalived rather than Pacemaker, and how to know the drain actually finished.→
- Orchestrating the rolling loop - automation that stops on evidenceExpressing drain, change, validate and return as an Ansible play with serial batches, a real health gate, and abort conditions that fire before the whole fleet is broken.→
Part LXV
Rolling Kernel Upgrades
6 checks
- Kernel package install - the kernel upgrade workflowHow to install a new kernel on a cluster node: package management, reboot requirement, and validation.→
- Reboot and boot validation - the kernel upgrade verificationHow to verify that a new kernel boots correctly and the system is fully functional after the reboot.→
- Live kernel patching concepts - patching without rebootingHow live kernel patching works: kpatch, kgraft, and Livepatch; the trade-offs and when to use it.→
- Kernel rollback - getting back to the kernel that workedRunning versus installed kernels, BLS boot entries, one-shot versus persistent default changes, initramfs rebuilds, and version-locking the kernel that broke the host.→
- Cluster capacity during a kernel campaign - the reboot budgetHow long a node is really out for a kernel reboot, why the 24-hour hold turns a patch into a multi-week campaign, and what that does to the capacity you are running on the whole time.→
- Proving a kernel campaign worked - CVEs, backports and vulnerability stateWhy comparing kernel version strings against upstream is meaningless on enterprise distributions, how to map a CVE to a vendor kernel build, and what /sys/devices/system/cpu/vulnerabilities actually tells you.→
Part LXVI
Capacity Planning for Clusters
6 checks
- Capacity utilisation bands - the 90%-on-3-nodes trapHow to design capacity utilisation bands, the 90%-on-3-nodes trap, and the trade-offs of headroom.→
- Normal and peak - measuring the load you actually haveHow to establish a capacity baseline from sysstat history: normal versus peak, why the average hides the number that matters, and the peak-to-mean ratio.→
- Utilisation is not saturation - what the queue tells you that the percentage cannotWhy a utilisation percentage stops predicting service quality long before it reaches 100%, and how to measure saturation directly with run queue, await, and pressure stall information.→
- Forecasting growth - turning a trend into a dateHow to convert a growth rate into the date a resource runs out: linear versus compound growth, doubling time, filesystem fill dates, and subtracting the procurement lead time.→
- Maintenance capacity - what a change window costs the clusterWhy planned maintenance consumes the same headroom as a failure, how to size for concurrent maintenance and failure, and when N+2 is the honest answer.→
- Writing the capacity plan - finding the binding constraintHow to turn per-resource measurements into one capacity plan: the six resources that run out, which one binds first, and the dated actions the plan must produce.→
Part LXVII
Cluster Monitoring
6 checks
- Cluster monitoring design - what to monitor in a clusterHow to design cluster monitoring: quorum, membership, failovers, resources, and the actionable alerts.→
- Monitoring failovers and resources - the cluster activityHow to monitor cluster failovers and resource state: tracking events, alerting on anomalies, and the actionable signals.→
- Actionable cluster alerts - the alert that leads to actionHow to design cluster alerts that are actionable: the right signal, the right threshold, the right runbook, and the right action.→
- Monitoring cluster storage and replication - watching the standby tooWhy monitoring the active node hides the failure that matters, how to watch replication health and redundancy, and which storage signals page rather than ticket.→
- Monitoring cluster network and time - the leading indicatorsWatching corosync links, the redundant ring that has been down for months, and why node-to-node clock comparison matters more than agreement with an upstream server.→
- What cluster monitoring cannot tell youSeven questions no monitoring system can answer about a cluster, what to put in place instead, and a decision procedure for what is worth waking someone for.→
Part LXVIII
Cluster Incident Response
6 checks
- Cluster IR: node unreachable - the most common incidentA runbook for the most common cluster incident: a node becomes unreachable. The diagnostic, the recovery, the prevention.→
- Cluster IR: network partition - deciding which side is realRunning an incident when the cluster has split: recognising a partition from inside it, choosing the surviving side, and why restarting the cluster service destroys the evidence.→
- Cluster IR: quorum loss - the arithmetic and the decision recordReading the vote arithmetic during a quorum-loss incident, diagnosing a failed quorum device, and recording the decision when you have to relax quorum by hand.→
- Cluster IR: fencing failure and the fence loopThree fencing incidents - the fence that never completes, the fence loop, and the node that comes back and is fenced again - and what a fenced node means for the incident timeline.→
- Cluster IR: shared storage loss - when a stop cannot succeedThe cluster incident where storage disappears, resource stops hang in uninterruptible sleep, and the cluster starts fencing healthy nodes because it cannot prove anything has stopped.→
- Cluster IR: building one timeline from nodes whose clocks disagreeCollecting evidence across cluster nodes during an incident: measuring clock offset before trusting timestamps, anchoring on a shared event, and merging several journals into one defensible timeline.→
Part LXIX
Hardware Health
6 checks
- SMART and NVMe health - disk failure predictionHow to use SMART and NVMe health monitoring to predict disk failures before they happen.→
- RAID controller monitoring - the storage layer belowHow to monitor RAID controllers: BBU, cache, RAID state, and the vendor-specific tools.→
- IPMI and BMC basics - the lights-out management foundationHow IPMI and BMC work: the out-of-band management interface, common commands, and the security considerations.→
- Thermal and ECC monitoring - the memory and CPU healthHow to monitor thermal and ECC memory: temperature, fan speed, ECC errors, and the predictive replacement discipline.→
- Hardware inventory and the SEL - turning an alert into a part numberHow to map a health alert onto a physical component with dmidecode, lsblk and lshw, and how to read the BMC System Event Log that records what the OS never saw.→
- Firmware lifecycle - the layer your package manager cannot seeAuditing and updating BIOS, BMC, drive and NIC firmware with fwupd and vendor tools, why firmware version drift is a real defect, and why firmware rollback usually does not exist.→
Part LXX
Out-of-Band Management
6 checks
- OOB architecture - the out-of-band management layerHow out-of-band management is designed: separate network, dedicated hardware, and the security considerations.→
- Redfish and IPMI APIs - the standards for hardware managementHow Redfish and IPMI work as management standards: the API surface, the typical operations, and when to use which.→
- OOB and DR - the out-of-band layer in disaster recoveryHow out-of-band management fits into disaster recovery: the role of BMC, the relationship to data center DR, and the testing discipline.→
- Serial console and SOL - the text lifeline into a hostHow to wire a serial console end to end - firmware, GRUB, kernel and getty - and reach it over IPMI serial-over-LAN when the host has no network.→
- OOB power control - the commands that change stateRead-only BMC queries first, then the power verbs and what each one actually does to a running host, plus the targeting discipline that stops you rebooting the wrong node.→
- OOB credentials and break-glass - access that survives the outageBMC account management, the circular dependency that makes a password vault useless during an incident, and a break-glass path you can actually test.→
Part LXXI
TLS and PKI
6 checks
- TLS and PKI concepts - the foundation of encrypted communicationHow TLS and PKI work: certificates, certificate authorities, chains of trust, and the renewal lifecycle.→
- OpenSSL for sysadmins - the practical toolkitHow to use openssl for the common sysadmin tasks: generating keys, creating CSRs, signing certificates, checking expiry.→
- Certificate authorities and chains - the trust model in practiceHow certificate authorities work in practice: public CAs, private CAs, intermediate CAs, and the trust chain.→
- Certificate renewal and expiry - the lifecycle disciplineHow to manage certificate renewal and expiry: monitoring, automation, and the production discipline.→
- TLS troubleshooting - diagnosing failures from the wireWhy the certificate on disk is not evidence about what a server serves, and an ordered diagnostic for chain, name, date, trust store, SNI and protocol failures.→
- Certificate revocation in practice - CRL, OCSP and short lifetimesHow CRL and OCSP actually behave, why soft-fail makes web revocation weak, what stapling fixes, and why short-lived certificates are the control that works.→
Part LXXII
Secrets
6 checks
- Secrets - the credential management problemWhy secrets management is a problem in production: where secrets live, how they leak, and the principles of secret management.→
- Secrets at rest - ownership, modes and the copies you forgotWhere credential files actually live on a Linux host, the modes and ownership they need, the window between creating a file and securing it, and the copies that outlive the original.→
- Secrets in scripts, logs and shell historyHow a correctly written script still leaks its credentials: xtrace into the journal, verbose clients in CI logs, per-tool history files, and the masking that does not mask.→
- Private keys - passphrases, agents and the keys you cannot rotateHandling SSH and TLS private keys as secrets: what a passphrase actually buys, agent lifetime and forwarding, matching a key to its certificate, and the keys that get baked into an image.→
- Rotation - making a credential change a routine operationWhy rotation is a change-management problem rather than a cryptographic one: consumer inventories, the two-credential overlap window, proving the old credential is unused, and emergency rotation after an exposure.→
- External secret managers - the bootstrap and availability problemsWhat changes when secrets move to a central store: how the host authenticates to it without a chicken-and-egg credential, what happens when the store is unreachable, and the failure modes a store introduces.→
Part LXXIII
Change Management
6 checks
- Change plans and peer review - the discipline of safe changesWhy change plans and peer review are the foundation of safe production changes: pre-checks, rollback, and validation.→
- Change classes and blast radius - sizing the process to the changeHow to classify a change as standard, normal or emergency, how to estimate blast radius honestly, and why reversibility matters more than the size of the diff.→
- Pre-checks and state capture - what to record before you touch anythingWhy a change must start from a known-good state, how to prove the host is healthy before you change it, and how to capture the before picture that validation and rollback both depend on.→
- Staged rollout - canaries, cohorts and the health gateHow to roll a change across a fleet in stages: choosing a representative canary, sizing cohorts, setting soak time, and gating on health rather than on a timer.→
- Abort criteria and the point of no return - rollback as part of the procedureWhy rollback belongs in the procedure rather than after it: writing abort criteria before the change, finding the point of no return, and choosing between rolling back and fixing forward.→
- Maintenance windows, freezes and the observation periodHow to size and defend a maintenance window, when a change freeze helps and when it makes things worse, and why the change is not finished when the window closes.→
Part LXXIV
Configuration Drift
6 checks
- Drift remediation - bringing hosts back to desired stateHow to remediate drift: the runbook for bringing hosts back, the prevention pattern, and the documentation.→
- How drift accumulates - the eight ways hosts stop matchingWhere configuration drift actually comes from: build-time divergence, package upgrades rewriting configs, emergency changes, partial rollouts, and the drift a configuration management tool creates itself.→
- Detecting drift independently of the CM toolThree sources of truth that do not come from your configuration management tool: package verification, a file integrity baseline, and node-to-node comparison - and what each one cannot see.→
- Package and version drift across a fleetDrift that lives in installed packages rather than configuration files: version divergence, package holds, third-party repositories on a subset of hosts, and code that is installed but not running.→
- Runtime versus on-disk drift - when the file is right and the system is notDrift between the configuration on disk and the state actually loaded: firewall rules, sysctl values, systemd overrides and kernel modules - and why the reboot is when it all arrives at once.→
- The limits of drift detection - measuring what nothing managesWhy "no changes" only proves the tool checked what it manages, how to measure management coverage on a real host, and how to decide what should be declared.→
Part LXXV
Immutable vs Mutable Infrastructure
6 checks
- Mutable server traditions - the legacy approachWhat mutable server traditions are, why they were the norm, and the limitations that led to the immutable approach.→
- Golden images and immutable patterns - the modern approachHow golden images and immutable patterns work: bake once, deploy many, replace on change.→
- Cloud images and the image build pipelineWhat a distribution cloud image actually is, how to verify one before you build on it, and how to run an image pipeline that produces reproducible, traceable, retirable images rather than an unlabelled pile of AMIs.→
- Configuration management and image baking - choosing the seamConfiguration can be applied at bake time, at launch time or at run time. Choosing deliberately is what separates an immutable fleet from one that has a golden image and a convergence loop fighting each other.→
- State in an immutable fleet - what survives replacement and what does notReplacing an instance deletes everything on its root disk. Finding every piece of state a Linux host holds, deciding where each piece belongs, and running stateful services under a model built for disposable machines.→
- The trade-offs, and the emergency change that has to existWhat immutable infrastructure costs, where it is the wrong choice, and how to design the emergency in-place change so the fix does not silently disappear at the next replacement.→
Part LXXVI
Virtualisation and Linux
6 checks
- Linux as a VM - virtual hardware, guest tools and the clockWhat changes when Linux runs as a guest: virtio devices and how to confirm them, guest agents and what breaks without one, memory ballooning and misleading free output, and why a hypervisor snapshot is not a backup.→
- Guest agents - the out-of-band channel into a running VMHow a guest agent actually talks to the hypervisor, why it works when the network does not, the freeze hooks that make a snapshot application-consistent, and the root-level command channel it opens.→
- Converting a guest to virtio - the migration that makes a VM unbootableHow to verify which device model a guest is really using, and how to move an existing guest from emulated devices to virtio without losing the boot.→
- CPU topology in a guest, and what the guest cannot seeThe socket and core layout a VM reports is a configuration choice with licensing and performance consequences, the CPU model constrains migration, and steal time is the only window a guest has onto its host.→
- Timekeeping in a guest - clock sources, steps and the resumed VMWhy a descheduled vCPU loses time, how kvm-clock and the TSC differ across a migration, what chrony does with a large offset by default, and the failure modes of a guest that resumes with the wrong clock.→
- Growing a guest online - disk, filesystem, memory and CPUThe hypervisor changes the virtual hardware and the guest keeps using the old numbers until you tell it not to: rescanning a resized disk, the full grow chain, hot-added memory that arrives offline, and why hot-added CPUs often do nothing.→
Part LXXVII
Linux in the Cloud
6 checks
- cloud-init and the instance metadata serviceHow a cloud instance configures itself at first boot, how to debug cloud-init when it does not, the per-provider metadata endpoints, and why that endpoint must be blocked from application containers.→
- Ephemeral and persistent storage - the cloud storage modelHow cloud storage works: ephemeral vs persistent volumes, snapshots, encryption, and the right pattern for the workload.→
- Writing user-data - payload formats, module frequency, and the test loopThe user-data payload formats cloud-init accepts, which modules run in which stage, the per-instance/per-boot/per-once frequency model, and how to test a configuration before it reaches a real instance.→
- Instance lifecycle - reboot, stop/start, terminate, replaceWhat each instance state transition does to the Linux system underneath: which disks survive, which addresses change, what cloud-init does next, and why "the instance was replaced" is a data-loss event you have to design around.→
- Security groups and the host firewall - two layers that fail differentlyThe platform firewall and the guest firewall are separate enforcement points with different failure signatures. How to tell which one dropped the packet, why stateless rules break return traffic, and why address-based rules break at the next instance replacement.→
- Instance identity and cloud IAM from the Linux sideHow a process on an instance proves who it is to a cloud API: instance roles versus static keys, the credential resolution chain that silently shadows them, and the places on a Linux box where a secret leaks.→
Part LXXVIII
Containers from the Linux Perspective
6 checks
- Container primitives - namespaces, cgroups, OverlayFS, capabilitiesHow containers are built on Linux: namespaces for isolation, cgroups for resources, OverlayFS for layers, capabilities for permissions.→
- OverlayFS for containers - the layered filesystemHow OverlayFS works for containers: lowerdir, upperdir, workdir, and the operational implications.→
- Network namespaces by hand - building container networking with ipConstruct the network a container runtime builds for you: a namespace, a veth pair, a bridge, forwarding and NAT - using only ip and nft, so the failures become recognisable.→
- User namespaces and UID mapping - the kernel side of rootlessHow a user namespace maps container UID 0 to an unprivileged host UID, where subuid ranges come from, why files appear owned by nobody, and the AppArmor restriction that breaks rootless tooling on modern Ubuntu.→
- What containers do not isolateThe shared kernel, the sysctls that are host-wide, the clock that is not virtualised, and the reason nproc and free report the host inside every container.→
- Attributing host symptoms to containers - triage without the runtime CLIA host is unhealthy and the cause is inside a container. How to get from a PID, a socket, a full filesystem or a runaway cgroup to the container responsible, using only /proc, the cgroup hierarchy and nsenter.→
Part LXXIX
Troubleshooting Methodology
6 checks
- The troubleshooting loop - ten steps and their stop-rulesThe named, repeatable diagnostic loop this course uses: ten steps, the Linux commands for each, and the rule that tells you when a step is finished.→
- Evidence-based diagnosis - capture before you disturbHow to collect Linux evidence that survives a restart, keep a defensible timeline, and tell correlation from causation.→
- Preventing recurrence - closing the failure gap and the detection gapHow to turn a restored service into follow-up actions that actually stop the incident happening again, and that shorten it if it does.→
- Defining the symptom - turning a complaint into a measurementStep 1 of the troubleshooting loop in detail: how to convert "it is slow" into an observed value, an expected value and a first-seen time, and what to do when it will not reproduce.→
- What changed - working the highest-yield questionStep 3 of the troubleshooting loop in detail: the six places a Linux host records what changed, how to read each of them, and why a host that has changed nothing can still break.→
- Narrowing to one subsystem - bisect the path, then test one thingSteps 5 to 7 of the troubleshooting loop: bisecting the request path to localise a fault, writing a hypothesis that can be proved wrong, and designing a test that measures more than it changes.→
Part LXXX
Common Failure Scenarios
6 checks
- Failure: a service will not start, a host will not bootThe distinguishing evidence for the two most common Linux failures: a unit that refuses to start, and a host that stops at an emergency shell.→
- Failure: filesystem full, inodes exhausted, filesystem read-onlyThree failures that all present as "cannot write": how to tell them apart with df -h, df -i and dmesg, and how to recover each.→
- Failure: DNS, routing, packet loss, firewall and certificate expiryHow to localise a network failure to a layer - resolution, route, filtering, or TLS - instead of guessing, and the evidence that identifies each.→
- Failure: OOM kills, CPU saturation and disk latencyTelling a memory failure from a CPU failure from a storage failure when every one of them presents as "the server is slow".→
- Failure: cluster node loss, quorum loss and dependency outagesDiagnosing a lost cluster node, a split partition, a failed fence, and the shared dependencies - DNS, time, identity - that take a whole cluster down at once.→
- Failure: nobody can log in, and sudo stopped workingFour different failures behind one symptom: identity resolution, authentication, authorisation and session setup - and the three commands that tell them apart before you touch anything.→
Part LXXXI
Incident Command
6 checks
- Incident roles and communication - the team coordinationHow to coordinate an incident response team: roles, communication, and the post-incident review.→
- Impact assessment - measuring what is broken for whomHow to establish the impact of an incident before declaring a severity: the four dimensions, measuring from the edge rather than from the component, and handling unknown impact.→
- Stabilisation - restoring service before you understand itThe mitigation catalogue, the one-change-at-a-time rule, and why diagnosing before stabilising costs more than it saves - plus the cases where stabilising first is wrong.→
- Evidence preservation - what recovery destroysWhich evidence each recovery action erases, the sixty-second capture that fits before a restart, log retention arithmetic, and the different rules for a security incident.→
- Escalation and handover - moving an incident between peopleWhen to escalate and what escalation actually transfers, the handover packet for a long incident, and what a vendor needs before they will help.→
- Recovery and stand-down - closing an incident properlyVerifying recovery from the user position, the observation period before stand-down, unwinding the emergency changes, and what carries forward into the review.→
Part LXXXII
Root Cause Analysis
6 checks
- Avoiding human error as cause - the systemic approachWhy human error is not the root cause and how to design systems that prevent humans from making mistakes.→
- Trigger, contributing factors, root cause - taking an incident apartThe three-part decomposition that makes a post-incident review produce work worth doing, and why the trigger is almost never the thing to fix.→
- Reconstructing the timeline - evidence over recollectionHow to build an incident timeline from timestamped evidence rather than memory: the sources, the clock problems, the two clocks you must track separately, and what an unexplained gap means.→
- Causal chains and stopping rules - not settling for the first plausible answerHow to push a causal chain past the first plausible explanation, why five whys produces a single misleading line, and the rules that tell you when the analysis is finished.→
- Systemic causes - the conditions that made the incident possibleHow to find the causes that live above any single host: normalised deviance, missing feedback loops, coupling, and the difference between an incident and a pattern.→
- The blameless postmortem - writing an RCA that produces changeThe structure of a post-incident review document, what blameless actually requires in practice, and the writing habits that decide whether anything gets fixed.→
Part LXXXIII
Operational Documentation
6 checks
- Architecture and dependency maps - the documentation foundationHow to document the architecture and dependencies of a service: diagrams, inventories, and the operational reference.→
- Build and recovery procedures - the operational referenceHow to document the build and recovery procedures: the steps to bring a service up, and the steps to bring it back.→
- Runbooks that work at 3amWriting an operational procedure for a reader who is half awake, does not own the system, and cannot ask anyone: expected output per step, named decision criteria, bounded waits, and how to test a runbook before it is needed.→
- Inventories, maintenance procedures and escalation pathsThe three operational records that are not procedures: what you have and who owns it, the recurring work that has no ticket, and the escalation path that has to work before you need it.→
- The incident record - writing it while it is happeningWhat a contemporaneous incident record contains and why it cannot be reconstructed afterwards: the timeline, negative results, decisions with reasoning, honest detection, and evidence captured before it is destroyed.→
- Documentation that stays true and can be foundThe two ways operational documentation fails without anyone noticing: it drifts out of agreement with the system, and it exists somewhere the reader never looks. Both have mechanical fixes.→
Toolkit
Runbooks
14 checks
- Runbook: Take and verify a backupRun an on-demand or scheduled backup so that it is restorable: quiesce the data, capture it consistently, verify the archive, and record what was taken.→
- Runbook: Cluster node failure - triage, recover, return to serviceA node has left the cluster. Establish whether it failed, was fenced, or was partitioned; confirm resources are running exactly once; and return the node safely.→
- Runbook: Disaster recovery - full cluster lossA production runbook for the worst-case DR scenario: total cluster loss, the procedure, communication, and verification.→
- Runbook: Fencing failure - recover a fenced nodeA production runbook for recovering when fencing fails or a node is incorrectly fenced.→
- Runbook: Investigate a systemd service that will not startDiagnose a failed or flapping systemd unit: read the real error, separate config faults from dependency and permission faults, and restore the service.→
- Runbook: Investigate an OOM killFind out what the OOM killer killed and why: read the kill record, separate a global OOM from a cgroup OOM, and fix the cause rather than adding RAM.→
- Runbook: Keepalived failover - test and recoverA production runbook for testing and recovering from a keepalived failover event.→
- Runbook: Load balancer failure - recover and restoreA production runbook for recovering from a load balancer failure: HAProxy down, backend pool exhausted, or routing issues.→
- Runbook: Pacemaker failover - test and recoverA production runbook for testing and recovering from Pacemaker failover, including STONITH and resource constraints.→
- Runbook: Quorum lossRestore a cluster that has lost quorum: establish which partition is authoritative, recover membership safely, and avoid the forced-quorum actions that cause split brain.→
- Runbook: Reboot a production serverReboot a production host without turning a two-minute restart into an outage: prove it will come back before you take it down, drain it, reboot, and validate.→
- Runbook: Rolling kernel upgrade - the production procedureA production runbook for a rolling kernel upgrade: drain, reboot, validate, return, with rollback procedure.→
- Runbook: Rotate a TLS certificateReplace a TLS certificate without an outage: build the correct chain, validate the key pair before deploying, reload rather than restart, and verify from outside the host.→
- Runbook: Filesystem full - blocks, inodes, and deleted-but-open filesRestore a full filesystem safely: separate block exhaustion from inode exhaustion from deleted-but-open files, reclaim space without deleting evidence, and prevent recurrence.→
Toolkit
Break/fix scenarios
5 checks
- Break/Fix: the resources will not fail over and the node is UNCLEANPacemaker refuses to start a resource anywhere. Separate a failing fence path from a constraint and from shared storage, and understand why the refusal is the guarantee working.→
- Break/Fix: the filesystem is full and deleting the big file did not helpENOSPC on a live host. Find the real consumer, reclaim space from a file that is already unlinked, and choose between rotation, archival and growth without losing data.→
- Break/Fix: a node vanished from the cluster — network, fencing, or split brain?A node drops out of a Corosync ring. Tell a blocked network path apart from a completed fence and from a genuine split brain, using signals that do not lie about UDP.→
- Break/Fix: the server is slow and every resource looks busyA server is slow. Separate the saturated resource from the merely busy ones with the USE method, and reject the three diagnoses that utilisation figures invite.→
- Break/Fix: a service is down and the health check is failingA failure injection drill: stop a critical service, diagnose it from the evidence rather than the assumption, restore it, and write the runbook.→