Kernel command line parameters - the settings you can only make at boot · Linux

linux-bootloader-grub covers how to get a parameter onto the kernel command line: edit /etc/default/grub, regenerate, or use grubby on a BLS host. This lesson is about the other half — what the kernel does with the line once it has it, which settings genuinely require it, and how to prove one worked.

The reason this matters is that operators reach for sysctl first, because sysctl is discoverable and takes effect immediately. A significant set of kernel behaviour is simply not reachable that way. Memory is partitioned, CPUs are isolated and IOMMU groups are formed long before userspace exists, so those decisions can only be expressed on the command line.

Three places a kernel knob can live

Mechanism	Applied when	Changeable at runtime	Example
Kernel command line	At boot, before init	No — needs a reboot	`intel_iommu=on`
Module parameter (`modprobe.d`)	When the module is loaded	Only if the module is unloadable	`options kvm_intel nested=1`
`sysctl`	Any time	Yes	`vm.swappiness=10`

Picking the wrong one is not an error that announces itself. sysctl -w on a key that does not exist fails loudly, which is fine — but writing a modprobe.d option for a built-in module succeeds silently and does nothing at all.

What belongs on the command line

The common cases, all of which decide something before userspace exists:

intel_iommu=on / amd_iommu=on / iommu=pt — device isolation groups, needed for passthrough.
hugepagesz=1G hugepages=16 — 1 GB pages must be reserved from contiguous memory at boot; there is no runtime path.
isolcpus=, nohz_full=, rcu_nocbs= — CPU isolation for latency-sensitive workloads; the scheduler domains are built at boot.
crashkernel= — memory reserved for the kdump kernel.
mitigations= — speculative execution defences.
console= — where kernel messages go, covered in linux-serial-console-and-sol.
net.ifnames=0 — the interface naming scheme, decided as devices are enumerated.

Note that hugepagesz/hugepages is a partial case worth knowing: 2 MB pages can be allocated at runtime through vm.nr_hugepages, and often fail to be, because after some uptime there is no contiguous 2 MB region left. 1 GB pages have no runtime path at all. “Reserve at boot” is the reliable answer for both.

How the kernel splits the line

The kernel does not treat the command line as one namespace. It sorts each token into one of four buckets:

BOOT_IMAGE=/boot/vmlinuz-6.8.0-51-generic  root=UUID=b043... ro
  crashkernel=512M  nvme_core.io_timeout=255  systemd.unit=rescue.target  single

  crashkernel=512M          -> a parameter the kernel itself knows
  nvme_core.io_timeout=255  -> module.parameter form, routed to the nvme_core module
  systemd.unit=...          -> unrecognised: exported as environment for init
  single                    -> unrecognised bare word: passed as an argument to init

The module.parameter=value form is the important one. It works whether the module is built into the kernel or loaded later, which makes it the only way to configure a built-in module — and it is the reason a parameter can look like a kernel parameter when it is really a module one.

Everything the kernel does not recognise is handed to userspace rather than rejected. That is what makes systemd.unit= work without the kernel knowing anything about systemd. It is also what makes typos invisible.

Read-only / Safecmdline verification— /proc/cmdline is what the kernel received. The dmesg line is what it made of it, and the second message is the one that earns its place: since kernel 5.3 the kernel names every token it did not recognise. Here a transposed io_timout instead of io_timeout produced a parameter that was silently exported to userspace as an environment variable and had no effect on the driver at all - and /proc/cmdline shows it present and correct-looking.

$ cat /proc/cmdline
sudo dmesg | grep -i 'command line'

BOOT_IMAGE=/boot/vmlinuz-6.8.0-51-generic root=UUID=b0434124-e13c-4dff-9006-5abf403266fe ro crashkernel=512M nvme_core.io_timeout=255 mitigations=auto

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-51-generic root=UUID=b0434124... ro crashkernel=512M nvme_core.io_timeout=255 mitigations=auto
[    0.036142] Unknown kernel command line parameters "nvme_core.io_timout=255", will be passed to user space.

Illustrative output

A mistyped parameter is accepted, echoed back to you, and does nothing

This is the defining failure mode of kernel command line work, and it is why the verification step is not optional.

An operator adds nvme_core.io_timout=255 — one transposed letter — to work around drives timing out under load. GRUB regenerates without complaint. The host reboots cleanly. cat /proc/cmdline shows the parameter, exactly as typed, which looks like confirmation. Six weeks later the drives are still timing out and nobody suspects the setting, because it is visibly present.

The kernel did tell you, once, in a message nobody read:

sudo dmesg | grep -i 'Unknown kernel command line'

Add that grep to the post-reboot validation for any change that touches the command line. It costs nothing and it converts a silent no-op into a one-line answer.

For a module parameter there is a second, stronger check — /sys/module publishes the value the module is actually using:

cat /sys/module/nvme_core/parameters/io_timeout

If that file reads the default rather than your value, the parameter did not take, whatever /proc/cmdline says.

Module parameters: three ways in, one way to check

modinfo -p lists what a module accepts, with types and descriptions. Run it before inventing a parameter name:

Read-only / Safemodinfo -p— modinfo reads the module file metadata and changes nothing. The parenthesised type matters: passing a string to a uint parameter is rejected at load time with a message in dmesg, not at boot with anything visible. If a parameter you want is not in this list, it does not exist in this kernel version - the set changes between releases.

$ modinfo -p nvme_core

multipath:turn on native support for multiple controllers per subsystem
iopolicy:Default multipath I/O policy; 'numa' (default), 'round-robin' or 'queue-depth'
admin_timeout:timeout in seconds for admin commands (uint)
io_timeout:timeout in seconds for I/O (uint)
shutdown_timeout:timeout in seconds for controller shutdown (byte)

Illustrative output

The three ways to set one:

# 1. At load time, for a module loaded after boot
sudo modprobe nvme_core io_timeout=255

# 2. Persistently, for a loadable module
echo 'options nvme_core io_timeout=255' | sudo tee /etc/modprobe.d/nvme.conf

# 3. On the kernel command line, which works for built-in modules too
#    nvme_core.io_timeout=255

And the one way to check, regardless of which you used:

cat /sys/module/nvme_core/parameters/io_timeout

A handful of parameters are writable at runtime through that same path — the file is mode 0644 rather than 0444 when the module author allowed it. Most are read-only, and a read-only one requires unloading and reloading the module, which for a storage or network driver means it is effectively a reboot.

`modprobe.d` does nothing for a module that is built into the kernel

options ext4 ... in /etc/modprobe.d/ is accepted by the file parser, written to disk, and ignored forever. modprobe handles loadable modules; a built-in one was never loaded and never will be, so nothing consults the configuration.

Find out which you have before you write the file:

modinfo -F filename ext4     # (builtin)
modinfo -F filename xfs      # /lib/modules/.../xfs.ko.zst

grep -E '^CONFIG_EXT4_FS=' "/boot/config-$(uname -r)"   # =y built in, =m loadable

For anything (builtin), the kernel command line is the only mechanism: ext4.some_param=value.

There is a second, sneakier version of the same problem. A module that is loadable but is loaded from the initramfs — which is true of the root filesystem driver, the storage controller driver and often the network driver — reads the modprobe.d files that were baked into the initramfs image, not the ones on the root filesystem, because the root filesystem is not mounted yet.

So a change to /etc/modprobe.d/ for such a module needs the initramfs rebuilding:

sudo update-initramfs -u          # Debian family
sudo dracut --force               # RHEL family

Miss that step and the setting applies on the next module reload and not on the next boot, which produces a host that behaves differently depending on how it got there.

Rolling out a command line change

The command line is fleet-wide state that only takes effect at the next boot, which makes it unusually easy to get into an inconsistent condition: half the fleet carries the parameter because it rebooted, half does not.

1. Decide the mechanism. Is it a sysctl? Then it is not this.
   Is the module built-in? Then modprobe.d is not an option.
2. Change it on ONE host. Reboot it.
3. cat /proc/cmdline                        parameter present
4. dmesg | grep 'Unknown kernel command line'   nothing you added
5. cat /sys/module/<mod>/parameters/<param>     the value you set
6. Verify the behaviour you wanted actually changed.
7. Only then roll it out - and track it as a campaign, because
   hosts do not have it until they reboot.

Step 6 is separate from step 5 for a reason. A parameter can be present, correctly spelled and correctly applied and still not do what you expected, because the documentation was for a different kernel version or the effect is conditional on hardware you do not have.

Record the intent next to the parameter

GRUB_CMDLINE_LINUX accumulates. A production host typically carries half a dozen parameters added over several years by people who have moved on, and there is no field for why.

Keep a comment block in /etc/default/grub — it is a shell-sourced file, so # comments are safe — naming each parameter, the reason, the ticket and who owns removing it:

# crashkernel=512M   kdump, required by the platform standard
# nvme_core.io_timeout=255  INC-4471, drives timing out under rebuild load
# mitigations=off    NOT SET, and must not be - see the campaign lesson
GRUB_CMDLINE_LINUX="crashkernel=512M nvme_core.io_timeout=255"

Without it, the next operator faces a choice between leaving a parameter they do not understand and removing one that is load bearing, and will pick the first one every time.

Three places a kernel knob can live

What belongs on the command line

How the kernel splits the line

Module parameters: three ways in, one way to check

Rolling out a command line change

Knowledge check