CPU topology in a guest, and what the guest cannot see

Everything a guest reports about its CPUs comes from the hypervisor. Not “is derived from” - is chosen by. The socket count, the cores per socket, the threads per core, the model name, the NUMA layout and the cache sizes are all values someone configured, and a guest has no way to check any of them against reality.

That is fine until something makes a decision from those numbers. Several things do.

Reading what you were given

$ lscpu | grep -E '^CPU|On-line|Thread|Core|Socket|NUMA|Model name|Hypervisor|Virtualization type'

CPU(s):                                  12
On-line CPU(s) list:                     0-11
Model name:                              AMD Ryzen 9 9955HX 16-Core Processor
Thread(s) per core:                      1
Core(s) per socket:                      12
Socket(s):                               1
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Hypervisor vendor:                       KVM
Virtualization type:                     full

Twelve vCPUs, one socket, one thread per core. The underlying host has sixteen physical cores with SMT, so none of the guest numbers describe the hardware. They describe a decision.

The same twelve vCPUs could equally have been presented as twelve sockets of one core each, and on many hypervisors that is the default when nobody sets a topology. The guest boots and works identically. What differs is everything downstream.

Consequence one: licensing

Per-socket licensing is still common - several database engines, some middleware, and a number of enterprise support subscriptions count sockets rather than cores.

A twelve-vCPU guest presented as 12 sockets x 1 core is a twelve-socket machine as far as that licence is concerned. The same guest as 1 socket x 12 cores is one socket. Identical performance, an order of magnitude difference in licence cost, and the only visible artefact is two lines of lscpu output that nobody reads during provisioning.

lscpu | grep -E '^Socket|^Core|^Thread'

Put that in the build validation for any guest running licensed software, and put the expected topology in the VM template rather than leaving it to the hypervisor default.

Consequence two: what applications size themselves from

Software reads the topology and makes decisions:

• The JVM sizes its default thread pools, garbage collector threads and heap ergonomics from the available processor count.
• PostgreSQL tuning guidance, and most autotuners, work from core counts.
• NGINX with worker_processes auto starts one worker per reported CPU.
• Applications that pin work to physical cores read /sys/devices/system/cpu/cpu*/topology/thread_siblings_list to avoid scheduling two threads on one physical core:

cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
cat /sys/devices/system/cpu/cpu0/topology/physical_package_id

In a guest presented as one thread per core, that file says each vCPU is alone on its core. It may well be sharing a physical core with a vCPU from another guest entirely. The application’s careful avoidance of hyperthread contention is operating on a fiction.

None of this is a reason to distrust the numbers - it is a reason to size latency-sensitive workloads with a topology that matches something real, and to reserve or pin the vCPUs at the hypervisor when the answer actually matters.

Consequence three: NUMA

A guest with NUMA node(s): 1 believes all its memory is equidistant from all its CPUs. If the VM is larger than one host NUMA node and the hypervisor has not exposed a virtual NUMA topology, that belief is wrong: some memory accesses cross a socket boundary and cost substantially more.

numactl --hardware
lscpu | grep -i numa

The symptom is a workload whose throughput varies between runs on identical guests, with no visible cause in the guest. The guest cannot see the problem, because from inside there is one node and the latency variation just looks like noise.

Two rules that avoid it:

• Keep a VM inside one host NUMA node where you can. A guest sized just under a node is often faster than one sized just over it, despite having fewer vCPUs.
• Where a VM must span nodes, expose a virtual NUMA topology that mirrors the host, so the guest scheduler and any NUMA-aware application can make correct placements. See linux-cpu-affinity-and-numa for the guest-side tooling.

The CPU model constrains migration

The hypervisor decides which CPU features to expose. Two broad choices:

Pass through the host CPU. The guest sees every instruction set extension the physical CPU has - AVX-512, newer AES instructions, whatever the generation offers. Fastest, and it welds the guest to that CPU generation.

Present a baseline model. The guest sees a defined, older feature set common to every host in the cluster. Slightly slower for code that could have used the newer instructions, and portable across the whole cluster.

grep -o 'hypervisor' /proc/cpuinfo | head -1
lscpu | grep -oE 'avx512[a-z0-9_]*' | sort -u | head

The hypervisor flag in /proc/cpuinfo is the CPU’s own statement that it is virtualised. The AVX-512 line is the one that matters for migration.

Steal time: the one window onto the host

A guest cannot see host load. What it can see is how much time its vCPUs spent runnable but not running because the host gave the physical core to someone else. That is steal time.

$ vmstat 1 3

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r  b   swpd    free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
2  0      0 4042657 591488 532398    0    0    22   160 7047  812 21  4 61  0 14  0
3  0      0 4041953 591508 532499    0    0     0   732 6939 3155 23  5 48  0 24  0
4  0      0 4041902 591508 532501    0    0     0   118 7210 3402 19  6 51  0 24  0

The st column is steal. Sustained double-digit steal means the host is oversubscribed and your guest is queueing for physical CPU. The application sees this as latency, and every in-guest diagnostic will point at the application, because from inside the guest the CPU genuinely is slow.

The raw counter is field 8 of the cpu line in /proc/stat, which is what monitoring agents read:

awk '/^cpu /{print "steal_ticks=" $9}' /proc/stat

(The cpu label is field 1, so the eighth value is field 9 in awk terms.)

Steal time is the single most useful guest-side metric in a virtualised estate and it is missing from most default dashboards, which show us, sy, wa and stop.

What remains invisible

Steal time is the exception. This is the list of things a guest genuinely cannot determine, and each one has produced an investigation that went nowhere:

ROTA deserves a mention because it is quietly misleading:

lsblk -o NAME,TRAN,ROTA,SIZE

ROTA=1 in a guest means the hypervisor told the guest the device is rotational. It may be NVMe underneath. Anything that tunes from that flag - an I/O scheduler choice, a database’s random-read cost estimate - is tuning from a declaration rather than a measurement.

The operational conclusion is the same in every row: an in-guest metric alone cannot close a virtualisation performance question. You need the host-side view as well, and the monitoring design should pair them. A dashboard with guest CPU, guest I/O latency and steal time next to host CPU run-queue and datastore latency answers questions that either half alone cannot.

Identifying the platform

Two commands worth knowing for any host you have just inherited:

systemd-detect-virt
sudo dmidecode -s system-manufacturer
sudo dmidecode -s system-product-name

dmidecode -s takes a keyword and prints one value; dmidecode -s with no argument lists the valid keywords. On a KVM guest the manufacturer typically reads QEMU, on VMware VMware, Inc., and on cloud instances the provider’s name - which is often the quickest way to establish what a poorly documented host actually is.

Knowledge check