Linux · Operational reference
Toolkit
Operational artefacts for Linux: checklists for cadences, runbooks for incidents, hands-on labs for skills, and break/fix scenarios for troubleshooting reflexes. None of this replaces reading the corresponding lessons — these are the artefacts you keep open in a second tab during real work.
- Runbooks
- 26
- Checklists
- 13
- Labs
- 68
- Break/Fix
- 21
- Assessments
- 1
Operational checklists
Repetitive tasks performed on a schedule.
Before deployment
Quarterly
Runbooks
Step-by-step procedures. Treat the rollback as part of the procedure — never skip it.
Critical risk
- criticalcluster affectingRunbook: Disaster recovery - full cluster loss~60 min · 12 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: Fencing failure - recover a fenced node~30 min · 8 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: LDAP or AD unavailable - restore central identity~30 min · 7 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: Compromised repository response~240 min · 7 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: Quorum loss~40 min · 8 steps · verified 2026-08-11→
- criticalcluster affectingRunbook: Recover a Linux host that fails to boot~60 min · 7 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: Recover from firewall lockout~30 min · 9 steps · verified 2026-08-09→
- criticalcluster affectingRunbook: SSH access incident - lockout, brute force, key compromise~30 min · 5 steps · verified 2026-08-09→
High risk
- highcluster affectingRunbook: Cluster node failure - triage, recover, return to service~45 min · 9 steps · verified 2026-08-11→
- highservice affectingRunbook: DNS failure - restore name resolution~30 min · 9 steps · verified 2026-08-09→
- highservice affectingRunbook: Keepalived failover - test and recover~30 min · 7 steps · verified 2026-08-09→
- highservice affectingRunbook: Load balancer failure - recover and restore~30 min · 9 steps · verified 2026-08-09→
- highservice affectingRunbook: MAC denial - restore service access~30 min · 5 steps · verified 2026-08-09→
- highservice affectingRunbook: Network incident - service unreachable triage~30 min · 10 steps · verified 2026-08-09→
- highcluster affectingRunbook: Pacemaker failover - test and recover~30 min · 9 steps · verified 2026-08-09→
- highcluster affectingRunbook: Production hardening - apply CIS baseline to a host~60 min · 8 steps · verified 2026-08-09→
- highservice affectingRunbook: Reboot a production server~30 min · 9 steps · verified 2026-08-11→
- highdata loss riskRunbook: Restore procedure - files, ownership and service state~60 min · 9 steps · verified 2026-08-11→
- highservice affectingRunbook: Rolling kernel upgrade - the production procedure~60 min · 9 steps · verified 2026-08-09→
- highservice affectingRunbook: Troubleshoot a firewall that is dropping traffic~45 min · 8 steps · verified 2026-08-11→
Medium risk
- mediumservice affectingRunbook: Take and verify a backup~40 min · 8 steps · verified 2026-08-11→
- mediumservice affectingRunbook: Investigate a systemd service that will not start~25 min · 9 steps · verified 2026-08-11→
- mediumservice affectingRunbook: Investigate an OOM kill~30 min · 8 steps · verified 2026-08-11→
- mediumservice affectingRunbook: Investigate a shell pipeline that produced wrong output~30 min · 8 steps · verified 2026-08-09→
- mediumservice affectingRunbook: Rotate a TLS certificate~35 min · 9 steps · verified 2026-08-11→
- mediumservice affectingRunbook: Filesystem full - blocks, inodes, and deleted-but-open files~30 min · 8 steps · verified 2026-08-11→
Hands-on labs
Time-boxed exercises.
B · Nested virtualisation
- Lab: Ansible basics - apply a configuration to multiple hosts~45 min · 4 objectives→
- Lab: Build an AppArmor profile for a service~45 min · 4 objectives→
- Lab: auditd watch and search~45 min · 4 objectives→
- Lab: Design and validate a backup strategy~60 min · 4 objectives→
- Lab: Backup tools — rsync, Borg, and Restic~90 min · 5 objectives→
- Lab: Build a bond with a VLAN subinterface and verify LACP~60 min · 4 objectives→
- Lab: Recover a Linux host that will not boot~60 min · 4 objectives→
- Lab: Audit how ping is privileged, and replace setuid with capabilities~30 min · 5 objectives→
- Capstone: Build a production Linux cluster from scratch~240 min · 5 objectives→
- Lab: Build a central log pipeline with Vector and Loki~60 min · 4 objectives→
- Lab: Apply cgroup resource limits to a service~45 min · 5 objectives→
- Lab: CIS self-audit on a host~60 min · 4 objectives→
- Lab: Cluster capacity model - design and forecast~60 min · 6 objectives→
- Cluster network topology lab - design and validate~60 min · 4 objectives→
- Lab: CPU saturation investigation~45 min · 6 objectives→
- Lab: Disk identification and partition layout~45 min · 4 objectives→
- Lab: DNS troubleshooting on a live host~45 min · 5 objectives→
- Lab: End-to-end connectivity walk-through on a real host~45 min · 4 objectives→
- Common Linux failure labs - the diagnostic drills~60 min · 6 objectives→
- Lab: Fencing scenario - simulate, verify, and document~60 min · 5 objectives→
- Lab 1: Walk the Filesystem Hierarchy Standard~45 min · 4 objectives→
- Lab: Filesystem check and repair with fsck and xfs_repair~75 min · 5 objectives→
- Lab: Build a fstab entry from scratch and recover from a broken one~45 min · 4 objectives→
- Lab: HA design tradeoffs - choosing the right pattern~60 min · 4 objectives→
- Lab: HAProxy layer 7 load balancer with health checks~60 min · 4 objectives→
- Lab: Investigate inodes, deleted-but-open files, and permission drift~60 min · 6 objectives→
- Lab: journalctl investigation end-to-end~45 min · 4 objectives→
- Lab: Keepalived failover - configure, test, document~60 min · 4 objectives→
- Lab: LVM create, extend, and snapshot~60 min · 7 objectives→
- Lab: LVM snapshot for safe deployment~45 min · 4 objectives→
- Lab: Configure a static address with Netplan and verify it~45 min · 4 objectives→
- Lab: Network baseline with iperf3 and ping~45 min · 3 objectives→
- Lab: Networking foundations - inspect the stack on a real host~45 min · 4 objectives→
- Lab: Configure a static connection with nmcli and verify it~45 min · 4 objectives→
- Lab: Build a basic nftables firewall from scratch~45 min · 5 objectives→
- Lab: Configure NAT for a small internal network~45 min · 4 objectives→
- Lab: OOM investigation and prevention~45 min · 5 objectives→
- Pacemaker 2-node cluster - the quorum challenge~60 min · 6 objectives→
- Pacemaker HA cluster lab - build a 3-node cluster~120 min · 5 objectives→
- Lab: Package operations end-to-end~45 min · 4 objectives→
- Lab: Configure source-based policy routing for two uplinks~60 min · 4 objectives→
- Lab: Process investigation end-to-end~45 min · 4 objectives→
- Lab: Write a production shell script with full error handling~60 min · 4 objectives→
- Lab: Set up Prometheus and node_exporter on a host~60 min · 5 objectives→
- Lab: Quorum math - calculate, simulate, design~45 min · 5 objectives→
- Lab: Build a RAID array, simulate a failure, and rebuild~60 min · 4 objectives→
- Lab: Repository trust and package verification~45 min · 4 objectives→
- Lab: Restore drill - validate the backup strategy end-to-end~90 min · 6 objectives→
- Lab: Rolling kernel upgrade on a cluster~90 min · 5 objectives→
- Lab: Rolling maintenance on a 3-node cluster~60 min · 6 objectives→
- Lab: Diagnose and fix SELinux denials on a service~45 min · 4 objectives→
- Lab 2: Build a production-style log triage pipeline~60 min · 4 objectives→
- Lab: Use ShellCheck to clean up an existing script~35 min · 5 objectives→
- Lab: Hardware sensor monitoring and predictive replacement~60 min · 4 objectives→
- Lab: Harden SSH on a host and validate the result~60 min · 5 objectives→
- Lab: Integrate a Linux host with Active Directory~60 min · 4 objectives→
- Lab: Storage baseline with fio~60 min · 4 objectives→
- Lab: Design and audit a sudoers policy~45 min · 4 objectives→
- Lab: Kernel module and sysctl tuning~45 min · 6 objectives→
- Lab: Apply sysctl hardening and validate the result~45 min · 4 objectives→
- Lab: Replace cron jobs with systemd timers~45 min · 4 objectives→
- Lab: Replace a cron job with a systemd timer~45 min · 4 objectives→
- Lab: Author a systemd service unit end-to-end~60 min · 4 objectives→
- Lab: Capture packets with tcpdump and diagnose a handshake failure~45 min · 4 objectives→
- Lab: Configure NTP on a small cluster and verify synchronisation~45 min · 6 objectives→
- Lab: Troubleshooting drill - apply the methodology~60 min · 6 objectives→
- Lab: USE investigation on a live host~60 min · 4 objectives→
- Lab: User and group management with safe defaults~45 min · 4 objectives→
Break/Fix scenarios
Troubleshooting drills. Each scenario gives symptoms and evidence, then hides the solution behind a reveal.
linux-net
- advancedBreak/Fix: half the peers cannot reach the host and the firewall is not the reason~40 min · 6 symptoms→
- advancedBreak/Fix: 2% of connections time out and the server log has no record of them~40 min · 6 symptoms→
- advancedBreak/Fix: small requests succeed, large ones hang, and ping says the network is fine~40 min · 6 symptoms→
security
- intermediateBreak/Fix: TLS stops working at midnight and the leaf certificate is valid~25 min · 4 symptoms→
- advancedBreak/Fix: 403 on a file with correct ownership, and ausearch finds nothing~40 min · 5 symptoms→
- intermediateBreak/Fix: the key works from the office but not from the VPN~25 min · 5 symptoms→
auth
- advancedBreak/Fix: logins fail, apt refuses the repo, and none of the errors mention time~35 min · 6 symptoms→
- advancedBreak/Fix: Nobody can log in and sudo hangs after the LDAP servers go away~30 min · 5 symptoms→
- intermediateBreak/Fix: the whole team lost sudo overnight and visudo -c says the config is fine~30 min · 6 symptoms→
linux-fs
boot
storage
linux-cluster
performance
service
Assessments
Production-readiness examinations. Each combines auto-scored questions with scenario-based rubrics you can use to grade yourself.