Linux as a VM - virtual hardware, guest tools and the clock

The overwhelming majority of production Linux runs as a guest, on KVM, VMware, Hyper-V or a cloud hypervisor. Most of what you know still applies. A handful of things behave differently enough to cause incidents, and they are the subject of this lesson.

Virtio: paravirtualised devices

An emulated device makes the guest driver talk to software pretending to be an Intel e1000 or an IDE controller. Every register access traps to the hypervisor. It works with any guest and it is slow.

Virtio replaces the pretence with a shared-memory ring buffer that both sides understand. Fewer traps, far higher throughput, much lower CPU cost.

Confirm what the guest is really using - the VM’s configuration and the guest’s view can disagree after a migration or a template clone:

# Block devices: TRAN should read "virtio" (or "nvme" in cloud)
lsblk -o NAME,TRAN,ROTA,SIZE

# PCI devices and the driver bound to each
lspci -k | grep -A3 -i virtio

# What the guest bus actually holds
ls /sys/bus/virtio/devices/

# Network driver in use
ethtool -i eth0 | head -3

Guest agents

A guest agent is a small daemon that lets the hypervisor ask the guest to do things it cannot do from outside.

# KVM / Proxmox / oVirt
sudo systemctl enable --now qemu-guest-agent
systemctl status qemu-guest-agent

# VMware
sudo systemctl enable --now vmtoolsd
vmware-toolbox-cmd -v

Without an agent the hypervisor can still run the VM. What it loses:

• Filesystem freeze before a snapshot. With an agent, the hypervisor calls fsfreeze so the snapshot is taken on a quiesced filesystem. Without it, the snapshot is crash-consistent - equivalent to pulling the power.
• Graceful shutdown. “Shut down guest” becomes a virtual power button press, or a hard power-off if the guest ignores ACPI.
• IP address reporting. The inventory shows no addresses, which breaks automation that discovers hosts from the hypervisor.
• Online resize signalling and time re-synchronisation after resume.

The freeze is the one that costs data. A crash-consistent snapshot of a running database restores like a server that lost power mid-transaction: often recoverable, sometimes not, and never something you want to discover during a restore.

Memory ballooning

The balloon driver lets the host reclaim memory from a guest. The host inflates the balloon; the driver allocates pages inside the guest and hands them back to the host. The guest’s usable memory shrinks without its configured size changing.

lsmod | grep virtio_balloon
free -h
cat /proc/meminfo | grep -E 'MemTotal|MemAvailable'

The clock in a VM

A virtual CPU is descheduled whenever the host wants the core. Timekeeping that assumes a continuously running CPU therefore drifts, and a paused or migrated guest can resume with its clock far behind.

# Which clock source is the kernel using?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource

# Is chrony disciplining it, and from what?
chronyc tracking
chronyc sources -v

On KVM the guest should be using kvm-clock, a paravirtualised source that reads host time directly. VMware guests use the equivalent host-time source. Run chrony as well: the paravirtualised clock keeps the guest close, and NTP keeps the fleet consistent.

Two rules that avoid most VM time incidents:

• Do not run both hypervisor time sync and NTP with both authoritative. Two things stepping the same clock produce jumps that neither can explain.
• Expect a step after a resume or migration. A guest resumed from a pause comes back behind real time. Alert on the absolute offset in both directions and on sync status - see linux-time-skew-operational-impact for why a one-sided alert misses exactly this case.

Snapshots

VM build validation

lsblk -o NAME,TRAN                          # virtio or nvme, not sata/ide
ethtool -i eth0 | grep driver               # virtio_net
systemctl is-active qemu-guest-agent        # active
cat /sys/devices/system/clocksource/clocksource0/current_clocksource   # kvm-clock
chronyc tracking | grep 'Leap status'       # Normal
lsmod | grep virtio_balloon                 # present only if ballooning is intended

Six commands. Run them on every new template and after every migration.

Knowledge check