Consoles and recovery: noVNC, SPICE, serial and qm terminal · Proxmox VE

Every guest you administer over SSH has a failure mode where SSH is exactly what stopped working: a firewall rule applied without a rollback timer, a network configuration that did not survive a reboot, a full root filesystem, a service that fails and takes sshd with it, a kernel that panics before userspace starts.

The console is the answer, and Proxmox offers four of them with genuinely different reach. Choosing between them under pressure requires knowing which ones survive which failures — and one of them has to be configured in advance or it is not there when you need it.

Four consoles, and what each survives

Console	Reaches	Survives a broken network	Shows the boot process	Needs guest configuration
noVNC	The emulated graphics adapter	Yes	Yes, if the guest writes to VGA	No
SPICE	The same, with a richer protocol	Yes	Yes	A client (`virt-viewer`) on your workstation
Serial (`qm terminal`)	A virtual serial port	Yes	Only if configured on both sides	Yes
QEMU monitor (`qm monitor`)	The hypervisor, not the guest	Yes	Not applicable — it is below the guest	No

The distinction that matters is the last column against the third. noVNC works with no preparation and is therefore what everyone reaches for. A serial console requires setting up in advance and gives you things noVNC cannot: a scrollback buffer, copy and paste that works, text you can search, a kernel log that keeps flowing when the graphics stack is broken, and a session you can drive from a script.

noVNC and SPICE

noVNC is the default and needs nothing. It renders the emulated display in a browser and is the correct tool for a graphical installer, a Windows guest, or any situation where you want to see what a screen would show.

Its limitations are real and are the reason this lesson continues:

It is a picture. There is no text to copy. A stack trace has to be transcribed by hand or photographed.
There is no scrollback. A kernel panic that scrolls past is gone.
Keyboard layout is a recurring problem. A guest expecting one layout and a browser sending another produces mistyped passwords that look like authentication failures.
It shows nothing if the guest has stopped writing to the display, which is exactly what a guest configured for serial-only output does.

SPICE improves the interactive experience — better performance, clipboard integration, USB redirection, multiple monitors — at the cost of needing a client installed locally. It is worth configuring for desktop-style guests and is not a recovery tool in any way noVNC is not.

Configuration changedisplay type— Changing the display type takes effect at the next cold start. Setting it to a serial port disables VGA output entirely.

set -euo pipefail
VMID=118

qm set "$VMID" --vga qxl        # SPICE
qm set "$VMID" --vga std        # plain VGA for noVNC
qm set "$VMID" --vga serial0    # redirect the web console to serial port 0

The serial console, done properly

Two halves, and the whole value depends on both being present.

Configuration changehost side: add a serial port— socket creates a unix socket on the host, which is what qm terminal connects to. The device appears at the next cold start.

set -euo pipefail
VMID=118

qm set "$VMID" --serial0 socket

qm shutdown "$VMID" --timeout 300
qm start "$VMID"

The serial[n] option accepts either a host device path or the literal socket. Passing through a real /dev/ttyS0 is for hardware scenarios; socket is what you want for a guest console, and the documentation is explicit that qm terminal is how you open a connection to it.

Configuration changeguest side, Linux: kernel output plus a login prompt— Run inside the guest. The kernel parameter makes boot messages appear on the serial line; the getty gives you somewhere to log in once boot completes.

set -euo pipefail

# 1. Kernel and bootloader output on the serial line.
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"/GRUB_CMDLINE_LINUX_DEFAULT="\1 console=tty0 console=ttyS0,115200n8"/' \
/etc/default/grub

# 2. GRUB itself on the serial line, so you can pick a rescue kernel.
cat >> /etc/default/grub <<'EOF'
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
EOF

update-grub

# 3. A login prompt on ttyS0 once userspace is up.
systemctl enable --now serial-getty@ttyS0.service

The order of the two console= parameters matters: the last one named becomes /dev/console, which is where the init system and early userspace send output. Listing tty0 first and ttyS0 second gives you kernel messages on both and the primary console on serial, which is the arrangement you want for a headless guest.

Read-only / Safeconnecting— Text, in your own terminal, with your own scrollback and your own copy and paste.

# qm terminal 118

starting serial terminal on interface serial0 (press Ctrl+O to exit)

app-frontend-01 login:

Illustrative output

Recovering a guest that will not boot

The sequence below is ordered by cost. Each step preserves more evidence than the one after it.

Read-only / Safe1. Establish what state the guest is actually in— Read-only. Distinguish a guest that is running and unreachable from one that is stopped, locked, or frozen.

set -euo pipefail
VMID=118

qm status "$VMID" --verbose
qm config "$VMID"

# A lock left by an interrupted operation blocks everything else.
qm config "$VMID" | grep -E '^lock:' && echo 'VM is locked; find out why before clearing'

# If the agent is configured, is the guest frozen rather than hung?
qm guest cmd "$VMID" fsfreeze-status 2>/dev/null || true

Service impact possible2. Boot into a rescue target from the guest bootloader— Requires a serial or graphical console. This is the fix for a full root filesystem, a bad fstab entry, or a service that fails at boot - none of which need a restore.

set -euo pipefail
VMID=118

# Attach to the console BEFORE starting, so you catch the bootloader.
# In one terminal:
#   qm terminal $VMID
# In another:
qm start "$VMID"

# At the GRUB menu: press 'e', append to the linux line, then Ctrl-X.
#   systemd.unit=rescue.target       single-user with the root filesystem
#   systemd.unit=emergency.target    minimal, root mounted read-only
#   init=/bin/bash                   no init at all, last resort
#
# In emergency mode the root filesystem is read-only. Remount it before
# editing anything:
#   mount -o remount,rw /

The Linux course covers what to do once you are there in emergency and rescue modes and boot failure diagnosis. The Proxmox-specific part is only how to get a console that shows you the bootloader, which is what the first half of this lesson was about.

Data-loss risk3. Mount the guest disk from the host, when the guest cannot boot at all— The VM must be STOPPED. Mounting a disk that a running guest is also writing to will corrupt the filesystem.

set -euo pipefail
VMID=118

# Absolutely first: confirm it is stopped.
qm status "$VMID" | grep -q 'status: stopped' || { echo 'VM is not stopped'; exit 1; }

# For a ZFS-backed disk, the zvol has partition device nodes already.
ls -l /dev/zvol/rpool/data/vm-"$VMID"-disk-0*
mkdir -p /mnt/rescue
mount /dev/zvol/rpool/data/vm-"$VMID"-disk-0-part1 /mnt/rescue

# For a qcow2 or raw file on directory storage, use libguestfs rather than
# a loop device: it handles LVM and multiple partitions and does not require
# the host kernel to trust the guest filesystem.
#   apt install -y libguestfs-tools
#   guestmount -a /var/lib/vz/images/118/vm-118-disk-0.qcow2 -i --ro /mnt/rescue

# Fix, then unmount before starting the VM again.
umount /mnt/rescue

Mounting a guest filesystem on the host is not a neutral act

Two hazards, and both have bitten people badly enough to be worth stating.

Concurrent mount. If the VM is running, or is started while the host has the filesystem mounted, two independent kernels are writing to one filesystem with no coordination. The result is corruption, and it is not always immediate — which means the damage can be discovered days later, after the backup rotation has covered the good copies.

Trusting a guest filesystem. Mounting an untrusted or damaged filesystem with the host kernel exposes the host kernel’s filesystem parser to that image. For a guest you administer this is an acceptable risk; for a customer guest, a compromised machine, or an image of unknown provenance it is not, and guestmount — which parses the filesystem in an isolated helper rather than in the host kernel — is the right tool.

Prefer, in order: fix it from the guest console; mount read-only with guestmount; mount read-write only when nothing else will do.

What qm monitor reaches that nothing inside the guest can

qm monitor opens the QEMU monitor: an interface to the hypervisor process, not to the guest operating system. It works when the guest is panicked, wedged, or has never booted, because it is not asking the guest anything.

What it is genuinely useful for:

info block — the block devices QEMU believes it has, with their backing files and cache modes. This is the authoritative answer when a guest and the VM configuration disagree about what disks exist.
info network — the same for network devices.
info status — running, paused, or stopped at the QEMU level, which can differ from what the guest thinks.
info registers and info cpus — the CPU state of a guest that has stopped executing, which is the only evidence available for some hangs.
dump-guest-memory — write the guest’s memory to a file for offline analysis, which is how you get a crash dump out of a guest that cannot produce one itself.

The monitor is also a loaded weapon. It can eject media, hot-unplug devices, and stop the CPU. stop at the monitor pauses the guest instantly with no notice to it, and people have typed it expecting it to stop the monitor.

Treat it as a diagnostic instrument for questions about the virtual hardware, and reach for qm subcommands for anything that has one.

What happens when the whole cluster GUI is unreachable and a guest needs recovering?

This is the scenario that decides whether the preparation was worth it, because noVNC and SPICE both arrive through pveproxy — the same web service that is currently not answering.

What still works, given SSH to any node:

qm terminal for any guest with a serial port configured. It runs locally on the node and talks to a unix socket. No web service involved.
qm monitor, for the same reason.
qm start, qm stop, qm config, qm listsnapshot and the rest of the CLI, provided the node has quorum. Without quorum /etc/pve is read-only, so you can read configuration and start guests but not change anything.

What does not work: anything that renders a screen in a browser.

The conclusion is uncomfortable and worth sitting with. A cluster where no guest has a serial port has exactly one console mechanism, and it depends on a service that fails for its own reasons. Adding --serial0 socket to a template is a two-minute change that removes that single point of failure for every guest created afterwards.

Knowledge check

Why this matters

Consoles are infrastructure you install before you need them, and the failure mode of not having them is not “slightly less convenient”. It is that the only remaining option is a restore — of a machine whose disk is probably fine and whose problem is one line in a configuration file.

The specific, cheap action this lesson argues for is putting --serial0 socket and a serial getty into every template. It costs nothing at runtime, it works when pveproxy does not, it gives you text you can copy into an incident channel, and it is the only console that shows a bootloader menu on a guest configured for headless operation.

Everything else here is ordinary Linux recovery. The Proxmox-specific contribution is getting you a terminal that still works when the usual ones do not.