Lab: Kernel module and sysctl tuning

This lab trains the muscle memory of kernel module management and sysctl tuning. Every production host has both, and every host has different settings — knowing how to inspect and change them is the foundation.

Objective

By the end of this lab, you can:

• Inspect loaded modules and their dependency tree.
• Load and unload a module at runtime.
• Blacklist a module and verify it does not load at next boot.
• Show that blacklist alone does not stop an explicit modprobe, and close the remaining paths with install.
• Make a module load at every boot with /etc/modules-load.d, paired with the sysctl key that module owns.
• Apply sysctl values persistently and verify.
• Audit the host’s kernel taint state.

Architecture

flowchart LR
  M[Module management]
  S[sysctl tuning]
  T[Taint audit]
  M --> S --> T

Requirements

• A Linux host (Ubuntu 24.04 LTS or Debian 12 preferred).
• Root or sudo access.

Scenario

You inherit a Linux host. Audit its current state, tune three sysctl parameters for production, blacklist an unused module, and document the taint state.

Tasks

Task 1: Inspect loaded modules

lsmod | head -20
lsmod | wc -l
lsmod | grep -E '^(nvme|ahci|ext4|xfs)'

Note:

• Total modules loaded.
• Storage-related modules (your fleet will depend on these).
• Any proprietary or out-of-tree modules (rare on stock distributions).

Task 2: Investigate a module’s metadata

modinfo ext4
modinfo $(lsmod | awk 'NR > 1 {print $1}' | head -1)

For the ext4 module:

• vermagic must match the running kernel.
• depends lists the modules ext4 requires.
• license is GPL for in-tree modules.

For a different module:

• Pick a module that interests you (a network driver, a filesystem).
• Note its size, dependencies, and description.

Task 3: Load and unload a module

Check the preconditions first. This task unloads a module, and the module you pick decides whether that is a no-op or an outage:

# Any bridges on this host?
ip -br link show type bridge

# Any container runtime?
command -v docker >/dev/null && docker ps -q | head
command -v podman >/dev/null && podman ps -q | head

# Any bridge or overlay networking modules already in use?
lsmod | grep -E '^(br_netfilter|bridge|overlay)'

If any of those return output, do not run this task on this host. Use a scratch VM.

dummy is the right demonstration module: it creates a virtual interface, nothing else in the kernel depends on its hooks, and unloading it cannot change how packets are filtered.

# Find a module that is not currently loaded
lsmod | grep dummy || echo "not loaded"

# Load it, verbosely, so you see what modprobe actually did
sudo modprobe -v dummy

# Confirm it is loaded
lsmod | grep dummy

# Unload it
sudo modprobe -r dummy

# Confirm it is unloaded
lsmod | grep dummy || echo "unloaded"

Task 4: Blacklist a module

# Blacklist a module you do not need
sudo tee /etc/modprobe.d/blacklist-lab.conf >/dev/null <<EOF
# Lab: blacklist the firewire stack (we do not use it)
blacklist firewire-core
blacklist firewire-ohci
blacklist firewire-sbp2
EOF

# Regenerate the initramfs (so the blacklist takes effect from boot)
sudo update-initramfs -u   # Debian-family
# sudo dracut -f             # RHEL-family

# Verify the blacklist file is in place
cat /etc/modprobe.d/blacklist-lab.conf

# Reboot to apply
sudo systemctl reboot

After reboot:

lsmod | grep firewire

The firewire modules should not be loaded.

Now prove the limit of blacklist, because it is narrower than its name suggests:

# An explicit modprobe still succeeds against a blacklisted module
sudo modprobe firewire-core
lsmod | grep firewire_core

The module loads. man 5 modprobe.d restricts the blacklist keyword to alias-based auto-loading — the path the kernel uses when hardware appears. An explicit modprobe, a load pulled in as another module’s dependency, and a load from the initramfs all bypass it. A control written as “the module is blacklisted” therefore passes config review and still fails the audit.

To close every path, pair the blacklist with an install line:

sudo modprobe -r firewire-core

sudo tee -a /etc/modprobe.d/blacklist-lab.conf >/dev/null <<'EOF'

# blacklist stops the alias path; install stops every path
install firewire-core /bin/false
EOF

sudo update-initramfs -u   # Debian-family
# sudo dracut -f            # RHEL-family

# modprobe now reports the install rule and loads nothing
sudo modprobe -v firewire-core
lsmod | grep firewire_core || echo "not loaded - install rule held"

Task 5: Apply sysctl values

sudo tee /etc/sysctl.d/99-production-tuning.conf >/dev/null <<EOF
# Production sysctl tuning — applied by 99- prefix to win over
# distribution defaults.

# TCP: enable SYN cookies, raise backlog
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.somaxconn = 4096

# Reverse-path filtering
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Ignore broadcast pings (smurf protection)
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Kernel: allow more PIDs
kernel.pid_max = 4194304
EOF

sudo sysctl --system

# Verify
sysctl net.ipv4.tcp_syncookies net.core.somaxconn kernel.pid_max

Task 6: Pair a module-owned sysctl with modules-load.d

Some sysctl keys do not exist until a module is loaded. The net.bridge.* keys are created by br_netfilter. Persisting one of them without persisting the module is only half the change: systemd-sysctl.service runs at sysinit, before anything has asked for br_netfilter, so the key is absent and the write fails with one log line while the host otherwise comes up healthy.

Reproduce the failure first:

sudo modprobe -r br_netfilter 2>/dev/null || true
sudo sysctl -w net.bridge.bridge-nf-call-iptables=1

Expect cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory.

Now persist both halves, and make each file name the other:

# The module, loaded at every boot by systemd-modules-load.service
printf 'br_netfilter\n' \
  | sudo tee /etc/modules-load.d/50-lab-bridge.conf

# The sysctl that only exists once that module is loaded
sudo tee /etc/sysctl.d/99-lab-bridge.conf >/dev/null <<'EOF'
# Requires br_netfilter — see /etc/modules-load.d/50-lab-bridge.conf
net.bridge.bridge-nf-call-iptables = 1
EOF

Test the pair now rather than at the next reboot:

sudo systemctl restart systemd-modules-load.service
systemctl status systemd-modules-load.service --no-pager
lsmod | grep br_netfilter

sudo systemctl restart systemd-sysctl.service
sysctl net.bridge.bridge-nf-call-iptables

Restarting the two units is the check that catches a typo today. A module name misspelled in /etc/modules-load.d/ is reported by systemd-modules-load.service and logged; nothing else tells you.

Reboot and confirm the value is still 1. That is the proof the load is persistent, not just applied by hand.

Task 7: Audit taint

cat /proc/sys/kernel/tainted
# Decode the bitmask
python3 -c "
n = $(cat /proc/sys/kernel/tainted)
flags = 'PFSRBMUDWCAIOLKEX'
print(','.join(f for i, f in enumerate(flags) if n & (1 << i)) or '(not tainted)'
"

Document what each flag means in your context. The K (live-patch) flag is normal on hosts with kpatch or canonical-livepatch enabled. The G flag (reserved/vendor) may indicate a vendor- specific condition.

Task 8: Audit your changes

ls /etc/modprobe.d/
ls /etc/modules-load.d/
ls /etc/sysctl.d/
cat /etc/modprobe.d/blacklist-lab.conf
cat /etc/modules-load.d/50-lab-bridge.conf
cat /etc/sysctl.d/99-production-tuning.conf
cat /etc/sysctl.d/99-lab-bridge.conf

All four files should be in place. The modprobe file blacklists the firewire stack and blocks the remaining load paths with install; modules-load.d makes br_netfilter load at boot; the two sysctl files apply the production tuning and the module-owned bridge key.

Note which directory does what. /etc/modprobe.d/ only shapes what happens when something else asks for a module — it never causes a load. /etc/modules-load.d/ is the directory that causes one.

Validation

The lab is complete when:

• You can list and inspect modules.
• You have loaded and unloaded a module at runtime.
• The firewire modules do not load at boot.
• You have observed modprobe firewire-core succeeding against the blacklist, and failing once install firewire-core /bin/false is in place.
• br_netfilter is loaded after a reboot with no manual step, from /etc/modules-load.d/50-lab-bridge.conf.
• The sysctl values from /etc/sysctl.d/99-production-tuning.conf are in effect after reboot, and so is net.bridge.bridge-nf-call-iptables.
• You have decoded and documented the host’s taint state.

Expected outcome

A host with a documented module blacklist and a documented sysctl tuning. Both files are in place and effective. The taint state is understood and explained in the runbook.

Troubleshooting

• modprobe -r fails with “Module is in use” — find what is using it with lsmod | grep <module> and stop the dependent service.
• Blacklist does not take effect after reboot — confirm the initramfs was regenerated. On Debian-family: update-initramfs -u; on RHEL-family: dracut -f.
• Module still loads despite the blacklist — expected if something calls modprobe explicitly or pulls it in as a dependency. blacklist covers only the alias path. Add install NAME /bin/false and rebuild the initramfs.
• sysctl --system reports “cannot stat” for a key — the module that creates that key is not loaded. Add it to /etc/modules-load.d/ and restart systemd-modules-load.service, then systemd-sysctl.service.
• modules-load.d entry ignored — check systemctl status systemd-modules-load.service for the failed name, and confirm the file ends in .conf with one module per line.
• sysctl —system does not show the new values — check for typos; confirm the file is named *.conf and lives in /etc/sysctl.d/.
• Taint bit decoding fails — install Python or decode the bitmask manually with a calculator.

Cleanup

# Remove the lab-specific files
sudo rm /etc/modprobe.d/blacklist-lab.conf
sudo rm /etc/modules-load.d/50-lab-bridge.conf
sudo rm /etc/sysctl.d/99-production-tuning.conf
sudo rm /etc/sysctl.d/99-lab-bridge.conf
sudo update-initramfs -u
sudo sysctl --system
sudo systemctl reboot

After reboot, the host returns to its baseline state.

What you learned

You can now manage kernel modules and sysctl parameters on a production host. The discipline is to use drop-in files, document the rationale, and test changes before deploying. Both modprobe and sysctl are runtime tools — the discipline is to use them for diagnostics and to persist the changes via the configuration files.