Growing a guest online - disk, filesystem, memory and CPU · Linux

Growing a VM is presented as a hypervisor operation: change a number in the VM configuration and the guest is bigger. The hypervisor half is genuinely that easy. The guest half is a chain of four or five separate steps, each of which reports success independently, and the reason resize tickets get reopened is that the chain stops partway and nothing says so.

The principle behind all of it: the hypervisor changes the virtual hardware, and the guest keeps using the numbers it already has until something makes it look again.

Disk: the chain, end to end

Enlarge the virtual disk on the hypervisor.
Make the guest kernel see the new device size.
Grow the partition, and make the kernel re-read the partition table.
Grow the LVM physical volume and logical volume, if LVM is in use.
Grow the filesystem.

Step 1 is where the change is made and steps 2 to 5 are where it is delivered. Each has its own failure.

Step 2: rescan the device

Whether you need to do anything depends on the bus:

# What am I working with?
lsblk -o NAME,TYPE,TRAN,SIZE,MOUNTPOINTS

# SCSI, SAS, and virtio-scsi: ask the device to rescan
echo 1 | sudo tee /sys/block/sda/device/rescan

# Confirm the kernel now agrees with the hypervisor
lsblk -o NAME,SIZE /dev/sda

A virtio-blk device (/dev/vdX) signals its own size change through a configuration-change interrupt, so the kernel usually picks it up with no action. A SCSI device - which includes virtio-scsi, VMware paravirtual SCSI, and anything emulated - does not, and needs the rescan write above.

If lsblk still shows the old size, nothing later in the chain can work, and the error you get from growpart will be about there being no free space rather than about the rescan.

Step 3: grow the partition

sudo growpart --dry-run /dev/sda 2
sudo growpart /dev/sda 2
sudo partx -u /dev/sda
lsblk -o NAME,SIZE /dev/sda

Note the space: growpart /dev/sda 2, disk and partition number as separate arguments. --dry-run reports what it would do without doing it, and it is worth running first because a partition number typo is how you resize the wrong partition.

growpart attempts the kernel partition-table update itself via partx --update, and partx -u afterwards is a cheap confirmation. Older advice says partprobe; on a disk with a mounted partition partprobe can return EBUSY while partx -u succeeds, so prefer partx.

Growing a GPT disk leaves the backup header in the wrong place

GPT keeps a primary header at the start of the disk and a backup copy at the very end. Enlarging the disk moves “the end”, so the backup header is now stranded in the middle, and the disk is technically inconsistent.

Tools notice at different times. parted prompts to fix it interactively, which is why parted in a script is a bad idea here. sgdisk -v reports it, and some firmware and some tooling complain on the next boot - which is a startling message to meet during an unrelated reboot three months later.

sudo sgdisk -v /dev/sda           # verify; reports a misplaced backup header
sudo sgdisk -e /dev/sda           # move the backup header to the end of the disk

-e is --move-second-header. Run it after the disk grows and before you grow the partition, and the rest of the chain proceeds normally. growpart handles this itself in current versions, so the check is often a no-op - which is exactly why it is worth running rather than assuming.

Steps 4 and 5: LVM and the filesystem

# LVM: grow the PV to fill the enlarged partition, then the LV
sudo pvresize /dev/sda2
sudo vgs; sudo pvs
sudo lvextend -r -l +100%FREE /dev/mapper/vg0-root

# No LVM: grow the filesystem directly
sudo resize2fs /dev/sda2          # ext4, online grow supported
sudo xfs_growfs /                 # XFS takes the MOUNT POINT, not the device

Three details that account for most of the confusion here:

lvextend -r (--resizefs) grows the filesystem in the same command. Without -r, the LV is bigger and df is unchanged, which is the single most common “the resize did not work” report.
xfs_growfs takes a mount point. Passing it a device path fails with an error that reads like the device is wrong. resize2fs takes a device. They are opposites and it catches everyone once.
Both grow online, on a mounted filesystem, including the root filesystem. That is normal and supported.

Every command succeeded and df is unchanged

The chain has five links and each one reports its own success. A partial run therefore looks like a complete run:

lsblk -o NAME,SIZE /dev/sda      # link 2: does the kernel see the bigger disk?
lsblk -o NAME,SIZE /dev/sda2     # link 3: is the partition bigger?
sudo pvs                         # link 4a: is the PV bigger?
sudo lvs                         # link 4b: is the LV bigger?
df -h /                          # link 5: is the filesystem bigger?

Run all five, in order, and the first one showing the old size is where the chain stopped. That is a ten-second diagnosis of a problem that otherwise gets escalated as “the storage team resized it and it did not work”.

The two most common stopping points are link 2 on a SCSI device where nobody wrote to rescan, and link 5 after an lvextend without -r.

Also check inodes while you are there. A filesystem that is full of small files can be out of inodes with plenty of free space, and growing the device does not create inodes on ext4

df -i tells you, and the fix is a different one entirely.

Memory hotplug

Two different mechanisms are often confused:

Ballooning changes how much of the guest’s already configured memory it may use. Covered in the overview lesson; the total never exceeds what the VM was given.

Hotplug adds new memory blocks to the guest, up to the maximum memory the VM was configured to allow. The blocks arrive as entries under /sys/devices/system/memory/, and whether they are usable depends on one file:

Read-only / Safeauto_online_blocks decides whether hot-added memory is usable

$ lsmem --summary; cat /sys/devices/system/memory/auto_online_blocks

Memory block size:                128M
Total online memory:               48G
Total offline memory:               0B
online

Illustrative output

If auto_online_blocks reads offline, hot-added blocks appear in lsmem as offline and contribute nothing. free and /proc/meminfo do not count them, and the guest keeps running under memory pressure with the memory sitting there unused - a resize that appears to have been applied on the hypervisor and had no effect at all.

lsmem                        # STATE column: online or offline
sudo chmem -e 4G             # bring 4G of offline memory online
lsmem --summary

The durable fix is the kernel parameter memhp_default_state=online on the kernel command line, or the distribution’s udev rule that onlines new blocks. Set it in the image rather than fixing it per host.

Removal is not symmetrical

Memory can only be taken away if the blocks are movable and nothing unmovable is allocated in them. lsmem shows this in its REMOVABLE column, and the honest summary is that growing a guest is routine and shrinking it is unreliable and should be planned as a reboot.

CPU hotplug

lscpu -e                     # per-CPU list, with an ONLINE column
sudo chcpu -e 8-11           # bring vCPUs 8 to 11 online
sudo chcpu -d 11             # take vCPU 11 offline

Newly added vCPUs may come up offline depending on the distribution’s udev rules, which is what chcpu -e is for. chcpu -r triggers a rescan for newly added CPUs where the platform needs it. CPU 0 generally cannot be offlined - there is no online file for it on most builds, which is why chcpu -d 0 fails in a way that looks like a permission problem.

Hot-added CPUs are frequently ignored by the application

The vCPUs come online, nproc reports the new count, and throughput does not change.

Most long-running processes read the processor count once, at start-up, and size themselves from it permanently:

A JVM fixes its garbage collector thread count and default pool sizes at launch.
nginx with worker_processes auto resolves auto at start and does not re-evaluate.
Thread pools in most application frameworks are sized in an initialiser.

So the hot-add gives the kernel more CPUs to schedule on and gives the application no more work to schedule. Realising the benefit needs a restart or a reload of the service - which is the thing hotplug was meant to avoid.

That does not make CPU hotplug useless. It is genuinely valuable for kernel-side work, for anything spawning processes per request, and for buying headroom before a planned restart. It just means the change ticket should say “hot-add the vCPUs, then reload the service in the next window”, rather than treating the hot-add as the whole job.

What you cannot do online

Shrinking is a different operation and it is not online

Growing is safe because every step adds space that nothing is using. Shrinking removes space that something might be using, and the guarantees invert:

XFS cannot shrink at all. There is no tool and no supported procedure. The only route is a new smaller filesystem and a data copy.
ext4 can shrink only when unmounted. resize2fs refuses to shrink a mounted filesystem, which means an offline window - and for a root filesystem, a rescue environment.
The order reverses, and getting it wrong destroys data. Growing goes device, partition, PV, LV, filesystem. Shrinking goes filesystem first, then LV, then PV, then partition. An lvreduce before the filesystem has been shrunk truncates a filesystem that still believes it owns those extents, and the damage is not recoverable by resizing back.
Take a backup first, and verify it restores. Not a snapshot on the same storage - a backup. This is the one operation in this lesson where a mistake is unrecoverable.

linux-lvm-shrinking-and-risk covers the procedure in detail. The operational advice is simpler: in a virtualised estate, do not shrink. Build a new guest at the right size and migrate the data. It is more work and it has a rollback.

Validation after any resize

lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
df -h; df -i
sudo pvs; sudo vgs; sudo lvs
lsmem --summary
lscpu -e | tail -5
findmnt --verify

findmnt --verify checks /etc/fstab against reality, which catches the other way a resize goes wrong: a device path that changed during the work and an fstab entry that still names the old one, discovered at the next reboot rather than now.