Guest agents - the out-of-band channel into a running VM

The overview lesson listed what a guest agent provides. This one is about how it works, because the mechanism explains both its most useful property and its most surprising failure mode.

The channel is not the network

A guest agent does not listen on a TCP port. On KVM it talks over a virtio-serial device - a paravirtual character device that the hypervisor attaches to the VM, appearing in the guest as a node under /dev/virtio-ports/:

ls -l /dev/virtio-ports/
systemctl status qemu-guest-agent --no-pager

On a KVM guest with the agent configured you will see org.qemu.guest_agent.0. VMware uses a different mechanism

• a hypervisor backdoor interface that vmtoolsd speaks - with the same architectural consequence.

That consequence is the reason agents matter operationally:

• The channel works when the guest has no network configuration at all, so it works on a host you have just locked out with a firewall rule, and during boot before networking starts.
• It works when the guest’s routing is broken, its DNS is down, or its interface was renamed by a kernel upgrade.
• It requires no credentials in the guest. There is no password, no SSH key, no user account involved.

The first two are why the agent is your recovery path. The third is the security consideration, and we come back to it.

First, confirm you are actually in a VM and which kind:

$ systemd-detect-virt; lscpu | grep -E 'Hypervisor|Virtualization type'

kvm
Hypervisor vendor:                       KVM
Virtualization type:                     full

systemd-detect-virt prints the detected technology and exits 0 when virtualised, or prints none and exits 1 when not - which makes it usable directly in a build check:

systemd-detect-virt --quiet --vm && echo "this is a VM"

The agent that is running and invisible

This is the failure worth memorising, because both halves of the evidence look correct.

The hypervisor reports “guest agent is not responding”. You log into the guest and check:

$ systemctl is-active qemu-guest-agent; ls /dev/virtio-ports/

active
ls: cannot access '/dev/virtio-ports/': No such file or directory

The package is installed, the unit is enabled and active, and the hypervisor still cannot reach it - because the virtio-serial device was never attached to this VM. The agent daemon started, found no channel, and sat there. It is running exactly as designed and it is useless.

Three things produce this:

• A VM created from a template or a definition that predates the agent, where nobody added the channel device.
• A VM restored from an old backup, or migrated between hypervisors with different defaults.
• A guest where the agent was installed later, by someone who correctly installed and enabled the package and had no way to add hardware to a running VM.

Freeze hooks: filesystem-consistent versus application-consistent

The agent’s most valuable function is guest-fsfreeze-freeze. The hypervisor asks the agent to freeze, the agent calls FIFREEZE on each mounted filesystem, the kernel flushes dirty pages and blocks new writes, the snapshot is taken, and the agent thaws.

That gives a filesystem-consistent snapshot: the filesystem structures are coherent and it will mount without a journal replay.

It does not give an application-consistent one. A database with data in its own buffers, a transaction half written, or a write-ahead log ahead of the data files, is snapshotted mid-flight. It is better than a crash-consistent image and it is not the same as a clean shutdown.

The bridge is the freeze hook. Before the freeze, and after the thaw, qemu-ga runs a hook script with an argument telling it which phase it is in:

ls -l /etc/qemu/fsfreeze-hook /etc/qemu/fsfreeze-hook.d/

The distribution ships a fsfreeze-hook dispatcher that runs every executable in fsfreeze-hook.d/, passing freeze or thaw:

#!/bin/bash
# /etc/qemu/fsfreeze-hook.d/postgresql.sh
set -euo pipefail

case "$1" in
  freeze)
    su - postgres -c "psql -qAt -c \"SELECT pg_backup_start('snapshot');\""
    ;;
  thaw)
    su - postgres -c "psql -qAt -c \"SELECT pg_backup_stop();\""
    ;;
  *)
    exit 1
    ;;
esac

Two rules for anything in that directory, both learned the hard way:

The freeze path must be fast and cannot touch the filesystems being frozen after the freeze begins. The guest is blocking writes while the hook runs. A hook that takes thirty seconds is thirty seconds of stalled I/O for every process on the guest, which application timeouts will notice.

The thaw path must be idempotent and must not fail. If the freeze succeeded and the thaw hook errors, you have a guest with frozen filesystems and no automatic recovery - every write blocks indefinitely and the VM appears hung. Handle the “was not actually frozen” case without exiting non-zero, and test the thaw independently.

What an agent grants to the hypervisor

guest-exec runs an arbitrary command inside the guest, as root, with no guest credential and no guest-side authentication step. guest-set-user-password sets a password for any account. guest-file-read and guest-file-write reach any file.

That is not a defect - it is the whole purpose of an out-of-band management channel, and it is what lets you recover a guest whose network you have broken. But it means one thing precisely:

Anyone who can administer the hypervisor has root on every guest running an agent, without touching the guest’s authentication, and without an entry in the guest’s auth log.

Follow that through:

• Hypervisor administration is a higher privilege tier than guest root, and its access controls, MFA and audit logging should reflect that. It frequently gets treated as an infrastructure convenience.
• Guest-side auditing cannot see it. Anything relying on the guest’s own logs to establish who did what has a blind spot exactly the size of the agent, and the record you need is on the hypervisor.
• A guest holding data with a compliance boundary inherits that boundary onto the virtualisation platform. This surprises people during audits.

Where the risk outweighs the convenience, qemu-ga can be started with specific RPCs disabled, so the agent still performs freeze and shutdown while refusing command execution. The option is set in the agent’s environment file

• /etc/sysconfig/qemu-ga on RHEL-family systems, or the unit’s environment on others - and the option name changed across QEMU versions, so check what your build offers before writing it into a role:

qemu-ga --help

Disabling guest-exec is a common and reasonable hardening step on guests holding regulated data. Disabling the freeze RPCs is not - that turns every snapshot back into a crash-consistent one.

VMware and cloud agents

The pattern repeats with different names:

Two notes that save time:

On VMware, install open-vm-tools from the distribution repository rather than the vendor bundle. It is the same code, it is what VMware recommends on modern Linux, and it upgrades with the rest of the system instead of breaking on the next kernel update.

The cloud agents carry the same privilege observation as guest-exec, and more visibly: an SSM or waagent channel is a remote root command path from the provider’s control plane into the instance, governed by cloud IAM rather than by anything inside the guest. That is usually what you want

• it is how you get a session on an instance with no inbound access - and it belongs on the list of ways someone can reach root on your hosts.

Build validation

systemd-detect-virt --quiet --vm || echo "not a VM - skipping guest checks"
systemctl is-active qemu-guest-agent
test -e /dev/virtio-ports/org.qemu.guest_agent.0 || echo "FAIL: no agent channel"
ls /etc/qemu/fsfreeze-hook.d/ 2>/dev/null

Four lines. The second one is the check everybody runs and the third is the one that catches the failure in this lesson.

Knowledge check