Linux for Production Sysadmins

Part I

Foundations

UNIX and Linux history, kernel vs userland, distributions, FHS, /proc /sys /dev /run /etc /var /usr /home /boot.

6 lessons

Part II

Shell and Command-Line Operations

Bash fluency, quoting, expansion, redirection, pipelines, exit codes, core text tools, command composition.

10 lessons

Part III

Filesystems and Files

Inodes, hard/symbolic links, permissions, umask, ACLs, extended attributes, immutable attributes, timestamps.

6 lessons

Part IV

Users, Groups and Identity

/etc/passwd /etc/shadow /etc/group, UIDs, GIDs, supplementary groups, password policies, service accounts.

6 lessons

Part V

sudo and Privileged Access

Root vs sudo, sudoers, /etc/sudoers.d, least privilege, logging, privileged session management.

6 lessons

Part VI

Processes

PID/PPID, sessions, process groups, threads, states, zombies, signals, priorities, nice values, cgroup views.

6 lessons

Part VII

systemd and Service Management

PID 1, units, services, targets, timers, sockets, mounts, dependencies, drop-in overrides, resource controls.

8 lessons

Part VIII

Logging and journald

Kernel logs, journald, syslog, rsyslog, logrotate, persistent journals, filtering, central logging.

6 lessons

Part IX

Boot Process

Firmware, UEFI, GRUB, initramfs, kernel parameters, emergency and rescue targets, recovery.

6 lessons

Part X

Kernel Management

Kernel versions, modules, dependencies, command line, sysctl, kernel logs, taint, live patching concepts.

6 lessons

Part XI

Package Management

apt, dpkg, dnf, rpm, repositories, signing, dependency resolution, version pinning, rollback considerations.

7 lessons

Part XII

Repository Security and Supply Chain

GPG keys, package provenance, third-party repositories, malicious packages, dependency compromise.

6 lessons

Part XIII

Disks and Block Devices

Block devices, naming, sectors, partitions, GPT, signatures, UUIDs, labels, persistent naming.

6 lessons

Part XIV

Filesystems

ext4, XFS, Btrfs characteristics, journaling, mount options, online growth, inode exhaustion, fsck/xfs_repair.

6 lessons

Part XV

/etc/fstab and Mount Management

UUID, LABEL, mount options, systemd interaction, network filesystems, recovery from broken fstab.

6 lessons

Part XVI

LVM

PV/VG/LV, extension, snapshots, risks of shrinking, recovery, day-2 operations.

7 lessons

Part XVII

Software RAID

mdadm, RAID1/5/6/10, degraded arrays, rebuilds, failure detection, monitoring.

6 lessons

Part XVIII

Enterprise Storage

SAN, NAS, iSCSI, Fibre Channel, NFS, multipath, device-mapper, shared storage failure paths.

6 lessons

Part XIX

Networking Foundations

Ethernet, MAC, ARP, IPv4, IPv6, subnetting, routing, TCP, UDP, ICMP, DNS, DHCP concepts.

8 lessons

Part XX

Linux Network Configuration

Interfaces, addresses, routes, gateways, DNS, Netplan, NetworkManager, systemd-networkd, persistent config.

7 lessons

Part XXI

Advanced Linux Networking

VLANs, bridges, bonding, teaming, MTU, jumbo frames, multiple routing tables, policy routing, IPv6, LACP.

8 lessons

Part XXII

Network Troubleshooting

OSI/TCP-IP layer methodology, ip, ss, mtr, dig, tcpdump, ethtool, arping, nc, openssl s_client, evidence-based diagnosis.

9 lessons

Part XXIII

DNS

Resolver architecture, /etc/resolv.conf, systemd-resolved, recursive resolution, authoritative DNS, TTL, caching, cluster failure modes.

6 lessons

Part XXIV

Time Synchronisation

UTC, monotonic time, NTP, chrony, systemd-timesyncd, operational impact on auth, TLS, logs, distributed systems.

6 lessons

Part XXV

Firewalls

nftables as the modern foundation, plus iptables/firewalld/ufw differences, chains, hooks, stateful filtering, connection tracking, NAT.

9 lessons

Part XXVI

SSH

Client/server, keys, host keys, agent, forwarding, ProxyJump, configuration, certificates, MFA concepts, bastion hosts.

8 lessons

Part XXVII

Authentication and Enterprise Identity

PAM, NSS, LDAP, Active Directory, Kerberos, SSSD, central identity failure modes.

8 lessons

Part XXVIII

SELinux and AppArmor

Mandatory access control, contexts, policies, enforcement, audit logs, denial investigation, profile creation.

6 lessons

Part XXIX

Linux Security Hardening

Least privilege, SSH, sudo, mount options, kernel parameters, firewall, MAC, audit, packages, CIS-style controls.

7 lessons

Part XXX

Linux Capabilities and Privilege

Traditional root model, Linux capabilities, file capabilities, process capabilities, capsh, getcap, setcap.

6 lessons

Part XXXI

Audit and Security Logging

auditd, authentication logs, sudo logs, kernel security events, ausearch, aureport, auditctl, central security logging.

6 lessons

Part XXXII

Vulnerability and Patch Management

Detection, CVEs, severity vs context, exploitability, patch priority, maintenance windows, reboot requirements.

6 lessons

Part XXXIII

Fleet Patch Management

Dev/test/canary/production waves, staged deployments, health validation, rollback, automated patching trade-offs.

6 lessons

Part XXXIV

Configuration Management

Desired state, idempotency, drift, inventory, automation, Ansible-flavoured examples.

6 lessons

Part XXXV

Shell Scripting for Sysadmins

Variables, conditions, loops, functions, traps, exit codes, error handling, set -euo pipefail, ShellCheck.

7 lessons

Part XXXVI

Scheduled Operations

cron, anacron, systemd timers, locking, duplicate prevention, logging, failure detection.

6 lessons

Part XXXVII

Resource Management

CPU, memory, swap, processes, file descriptors, ulimits, cgroups, systemd resource controls, OOM behaviour.

6 lessons

Part XXXVIII

Linux Performance Fundamentals

USE methodology, top, htop, vmstat, mpstat, pidstat, iostat, sar, free, slabtop, evidence-based investigation.

6 lessons

Part XXXIX

CPU Performance

Utilisation, load average, run queue, context switching, interrupts, softirqs, steal time, CPU affinity, NUMA.

6 lessons

Part XL

Memory Performance

Virtual memory, pages, page cache, anonymous memory, buffers, swap, memory pressure, OOM killer, slab.

6 lessons

Part XLI

Storage Performance

Latency, throughput, IOPS, queue depth, utilisation, filesystem cache, fio, iostat, iotop, safe benchmarking.

6 lessons

Part XLII

Network Performance

Bandwidth, latency, packet loss, retransmissions, socket queues, connection states, iperf3, ethtool, sar.

6 lessons

Part XLIII

eBPF and Advanced Observability

eBPF, tracepoints, kprobes, uprobes, BPF maps, bpftrace, BCC, sysadmin use cases.

6 lessons

Part XLIV

Central Monitoring

Prometheus node_exporter, Grafana, alerting on CPU, memory, FS, inodes, disk latency, network, services, time sync, hardware.

6 lessons

Part XLV

Central Logging

journald, rsyslog, syslog, Fluent Bit, Vector, Loki, Elasticsearch, retention, filtering, cardinality, capacity.

6 lessons

Part XLVI

OpenTelemetry

Metrics, logs, traces, collectors, agents/gateways, Linux infrastructure telemetry.

6 lessons

Part XLVII

Backup Strategy

Filesystem, application, database, configuration, snapshot, consistency, encryption, retention, immutable copies, offsite, 3-2-1.

6 lessons

Part XLVIII

Backup Tools

rsync, tar, Borg, Restic, filesystem snapshots, enterprise backup integration, selection criteria.

6 lessons

Part XLIX

Restore

Restoration over backup success: files, ownership, ACLs, services, configuration, full restore exercises.

6 lessons

Part L

Disaster Recovery

RPO, RTO, disaster scenarios, rebuild vs restore, bare-metal recovery, DNS, certificates, identity, full cluster loss.

6 lessons

Part LI

Linux Fleet Architecture

Management plane, configuration management, monitoring, logging, identity, automation, secrets, patching across 50-5000 nodes.

6 lessons

Part LII

High Availability Fundamentals

Availability, redundancy, fault tolerance, failure domains, active/active, active/passive, N+1, N+2, host vs service availability.

6 lessons

Part LIII

Quorum and Split Brain

Quorum, majority, membership, partitions, split brain, witness/quorum device, failure detection.

6 lessons

Part LIV

Fencing and STONITH

Why fencing exists, split-brain data corruption, fencing devices, STONITH, hardware-independent principles.

6 lessons

Part LV

Pacemaker and Corosync

Corosync membership, Pacemaker, resources, resource agents, constraints, failover, fencing integration.

6 lessons

Part LVI

Keepalived and VRRP

Virtual IPs, VRRP, health checks, master/backup, failover, limitations.

6 lessons

Part LVII

Linux Load Balancing

HAProxy, nginx, IPVS, Layer 4 vs Layer 7, health checking, algorithms, session persistence, connection draining.

6 lessons

Part LVIII

Clustered Service Architecture

Stateless, shared state, replicated state, external state, application architecture constraints, identifying the right pattern.

6 lessons

Part LIX

Shared Storage and Clusters

NFS, SAN, clustered filesystems, distributed storage, two-nodes-mounting-the-same-block-filesystem risk.

6 lessons

Part LX

Distributed Storage Concepts

Ceph, Gluster, object storage, replicated block, replication, quorum, failure domains, consistency, recovery.

6 lessons

Part LXI

DRBD Concepts

Primary/secondary, replication, split-brain, clustering integration, when DRBD materially helps HA.

6 lessons

Part LXII

Cluster Networking

Management, application, storage, heartbeat, OOB, redundant interfaces, switches, VLANs, failure domains.

6 lessons

Part LXIII

Cluster Time, DNS and Identity Dependencies

DNS unavailable, NTP skew, LDAP unavailable, certificate expiry, designing around dependency failures.

6 lessons

Part LXIV

Rolling Maintenance

Validate, drain, patch, reboot, validate, return, observe, batch sizing, maintenance mode, health checks.

6 lessons

Part LXV

Rolling Kernel Upgrades

Kernel package install, reboot requirement, boot validation, rollback, cluster capacity during maintenance, live patching concepts.

6 lessons

Part LXVI

Capacity Planning for Clusters

Normal/peak utilisation, failure capacity, maintenance capacity, growth, N+1 headroom, 90%-on-3-nodes trap.

6 lessons

Part LXVII

Cluster Monitoring

Node health, service health, quorum, membership, failovers, resource status, storage, network, replication, time sync.

6 lessons

Part LXVIII

Cluster Incident Response

Ten canonical scenarios: node unreachable, partition, quorum loss, fencing failure, shared storage loss, LB failure, clock skew, DNS outage, deployment regression, memory exhaustion.

6 lessons

Part LXIX

Hardware Health

SMART, NVMe health, RAID controllers, IPMI, BMC, Redfish concepts, ECC memory, thermal sensors, smartctl, nvme, sensors, ipmitool.

6 lessons

Part LXX

Out-of-Band Management

IPMI, iDRAC, iLO, Redfish, serial console, remote console, power control, relation to fencing and DR.

6 lessons

Part LXXI

TLS and PKI

Private keys, certificates, CSRs, CAs, chains, SAN, expiry, revocation, openssl, certificate troubleshooting, expiry incidents.

6 lessons

Part LXXII

Secrets

Passwords, private keys, API tokens, service credentials, secrets in scripts/Git/world-readable/shell history, external managers.

6 lessons

Part LXXIII

Change Management

Change plans, peer review, pre-checks, rollback, maintenance windows, testing, validation, observation period.

6 lessons

Part LXXIV

Configuration Drift

How environments become inconsistent, detection, remediation, drift across nodes, packages, firewall rules, configuration.

6 lessons

Part LXXV

Immutable vs Mutable Infrastructure

Traditional servers, configuration management, golden images, cloud images, immutable replacement, trade-offs.

6 lessons

Part LXXVI

Virtualisation and Linux

Linux as a VM: virtual hardware, virtio, VMware tools, cloud guest agents, CPU topology, memory ballooning, snapshots, time sync.

6 lessons

Part LXXVII

Linux in the Cloud

cloud-init, metadata services, ephemeral disks, persistent block storage, security groups, IAM concepts, instance lifecycle.

6 lessons

Part LXXVIII

Containers from the Linux Perspective

Namespaces, cgroups, OverlayFS, capabilities — the Linux primitives containers build on; cross-link to the Docker course.

6 lessons

Part LXXIX

Troubleshooting Methodology

Define symptom, determine impact, check recent changes, collect evidence, identify subsystem, form hypothesis, test safely, restore service, find root cause, prevent recurrence.

6 lessons

Part LXXX

Common Failure Scenarios

Service won’t start, system won’t boot, filesystem full, inode exhaustion, read-only FS, failed disk, degraded RAID, LVM full, DNS, route, packet loss, firewall, cert expiry, auth, sudo, OOM, CPU saturation, disk latency, broken fstab, kernel upgrade, cluster node loss, quorum loss.

6 lessons

Part LXXXI

Incident Command

Impact assessment, severity, communication, roles, stabilisation, evidence preservation, recovery, escalation, post-incident review.

6 lessons

Part LXXXII

Root Cause Analysis

Trigger vs contributing factor vs root cause, systemic causes, avoiding "human error" as the explanation.

6 lessons

Part LXXXIII

Operational Documentation

Architecture diagrams, inventories, build procedures, recovery procedures, dependency maps, maintenance procedures, escalation.

6 lessons

Part LXXXVI

Capstone: Production Linux Cluster

A 3-node production Linux cluster with HA load balancing, shared storage, monitoring, central logging, configuration management, central identity, DNS, NTP, backup, secrets, certificates — end-to-end operations.

0 lessons

Part LXXXVII

Break It and Fix It

Major failure-injection section: symptoms first, evidence provided, root cause and remediation hidden behind reveal.

0 lessons