GRUB — the bootloader

GRUB is the bootloader on virtually every modern Linux distribution. It receives control from the firmware, presents a menu (or not), and loads the kernel and initramfs.

GRUB’s role in the boot sequence

sequenceDiagram
  participant FW as Firmware
  participant GRUB as GRUB
  participant K as Kernel
  participant I as Initramfs

  FW->>GRUB: load EFI binary from ESP
  GRUB->>GRUB: read /boot/grub/grub.cfg
  GRUB->>GRUB: present menu or auto-boot
  GRUB->>K: load vmlinuz-<version>
  GRUB->>I: load initramfs-<version>.img
  GRUB->>K: pass kernel command line
  K->>I: mount / as tmpfs
  I->>I: load drivers, mount rootfs
  I->>I: hand off to systemd (PID 1)

GRUB itself is a small EFI binary that knows how to read filesystems and load files. The configuration (grub.cfg) lives in /boot/grub/ (legacy GRUB) or grub-mkconfig generates it from /etc/default/grub and /etc/grub.d/.

The configuration files

$ ls /etc/grub.d/

00_header
05_debian_theme
10_linux
20_linux_xen
30_os-prober
30_uefi-firmware
40_custom
41_custom

$ cat /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

$ cat /boot/grub/grub.cfg | head -30

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
...

menuentry Ubuntu --class ubuntu --class gnu-linux ... {
linux   /boot/vmlinuz-6.6.31-linuxkit
initrd  /boot/initrd.img-6.6.31-linuxkit
}
...

The kernel command line

The kernel command line is the bridge between GRUB and the running kernel. Critical parameters:

Regenerating GRUB configuration

$ sudo grub-mkconfig -o /boot/grub/grub.cfg

Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.6.31-linuxkit
Found initrd image: /boot/initrd.img-6.6.31-linuxkit
Found linux image: /boot/vmlinuz-5.15.0-1053-aws
done

$ sudo update-grub; sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.6.31-linuxkit
...

Which grub.cfg does the firmware actually read?

update-grub always writes /boot/grub/grub.cfg, on BIOS and UEFI alike. grub2-mkconfig writes wherever -o points, and on the RHEL family that path depends on the firmware mode:

# Debian/Ubuntu, BIOS or UEFI
sudo update-grub                                        # -> /boot/grub/grub.cfg

# RHEL/CentOS/Rocky 7-8, BIOS
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# RHEL/CentOS/Rocky 7-8, UEFI - this is the file the firmware loads
sudo grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

# Never guess. Ask the system which paths exist:
ls -l /boot/grub2/grub.cfg /boot/efi/EFI/*/grub.cfg

RHEL 9 converged the two: the file on the ESP became a small stub that chains to /boot/grub2/grub.cfg, so a single regeneration covers both. On RHEL 7 and 8 UEFI hosts it is two separate files, and writing to /boot/grub2/grub.cfg on a UEFI host produces a config that is never read.

RHEL 8+ uses BootLoaderSpec, so grub.cfg is not where kernel arguments live

Since RHEL 8 (and Fedora 30) the menu entries are BootLoaderSpec files, one per kernel, under /boot/loader/entries/*.conf. Regenerating grub.cfg does not rewrite them, so editing GRUB_CMDLINE_LINUX and running grub2-mkconfig changes nothing for the kernels already installed — the command succeeds, the reboot happens, and the argument is simply absent. Use grubby, which edits the BLS entries directly:

# What arguments do the installed kernels actually carry?
sudo grubby --info=ALL

# Add and remove arguments across every entry
sudo grubby --update-kernel=ALL --args="crashkernel=auto"
sudo grubby --update-kernel=ALL --remove-args="quiet"

Installing GRUB

$ sudo grub-install /dev/sda

Installing for x86_64-efi platform.
Installation finished. No error reported.

$ sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu /dev/sda

Installing for x86_64-efi platform.
Installation finished. No error reported.

GRUB recovery

When GRUB is broken, the host does not boot. Three recovery paths:

1. GRUB rescue shell — GRUB itself shows a minimal shell. From it, manually load the kernel and initramfs.
2. Live CD/USB — boot a different system, chroot into the broken one, fix GRUB.
3. OOB management — IPMI/iDRAC/iLO console, mount the broken disk as a virtual media, fix from there.

$ grub rescue> ls

(hd0) (hd0,msdos1) (hd0,gpt1)

Production discipline

1. **Edit /etc/default/grub**, never /boot/grub/grub.cfg directly. The generated file is overwritten by every grub-mkconfig
2. Test every kernel command-line change on a single host before rolling out to the fleet
3. **Confirm the change landed with cat /proc/cmdline after the reboot.** A successful grub2-mkconfig proves nothing on a UEFI or BLS host - it may have written a file the firmware never reads, or one the BLS entries override
4. Document the recovery procedure in the runbook — including OOB console access, EFI boot entry recreation, and a working live-USB image
5. Verify the boot entry after every firmware update. Vendors sometimes wipe NVRAM; the host reboots into the firmware setup
6. Back up the working grub.cfg and EFI binary before major changes

Knowledge check