Timekeeping in a guest - clock sources, steps and the resumed VM · Linux

The overview lesson gave the two rules that avoid most VM time incidents. This one is about why they are the rules, because when a time problem does happen the diagnosis depends on knowing which layer produced it.

The root cause of all of it is one sentence: a vCPU is not always running. The host deschedules it whenever it wants the physical core, and any timekeeping that assumes a continuously executing CPU loses time across that gap without knowing it did.

Clock sources

The kernel reads time from a clock source, and the guest has several to choose from:

Read-only / Safea KVM guest should be on kvm-clock

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource /sys/devices/system/clocksource/clocksource0/available_clocksource

kvm-clock
kvm-clock tsc acpi_pm

Illustrative output

Source	How it reads time	Cost	Survives migration
`kvm-clock`	Paravirtual: a shared page the host updates	Very low	Yes - the host corrects it
`tsc`	Reads the CPU timestamp counter directly	Lowest	Only with hardware TSC scaling
`acpi_pm`	Traps to the emulated ACPI power-management timer	High: a VM exit per read	Yes, slowly
`hpet`	Traps to the emulated HPET	High	Yes, slowly

kvm-clock is the right answer on KVM and it is the default on any modern guest. The equivalent on VMware and Hyper-V is their own paravirtual source. Set nothing and check occasionally; the situation worth catching is a guest that fell back to acpi_pm, where every clock_gettime() becomes a VM exit and a busy application can spend a measurable fraction of its CPU asking what time it is.

Why not just use the TSC

The TSC is the fastest thing available: a single instruction, no trap. On modern hardware it is invariant, meaning it ticks at a constant rate regardless of CPU frequency scaling and C-states, so a guest pinned to one host can use it safely.

The problem is movement. TSC values and frequencies differ between physical hosts. Live-migrating a guest that reads the TSC directly means the counter it is reading changes rate and offset underneath it. Modern CPUs have TSC scaling support in hardware - the tsc_scale flag on AMD, an equivalent VMX control on Intel - which lets the hypervisor present a consistent TSC across hosts, and where it is missing the hypervisor has to intervene.

kvm-clock sidesteps all of it by asking the host, which is the entity that knows the answer.

Watch for the kernel deciding the TSC is not trustworthy:

sudo dmesg -T | grep -iE 'clocksource|tsc'

Marking TSC unstable due to clocksource watchdog or ... due to check_tsc_sync_source failed is the kernel demoting itself to a slower source. It is a symptom, not a fault to fix in the guest: it usually means host CPU oversubscription or a hypervisor configuration issue, and the guest is now paying a much higher cost per time read.

What chrony does with a large offset

This is the behaviour that surprises people, and it is a deliberate design decision rather than a bug.

By default chrony slews: it speeds the clock up or slows it down until the offset is gone, so time never goes backwards and never jumps. Applications that assume wall time advances monotonically are not broken by it.

It steps - jumps the clock directly - only under the makestep directive, and the usual configuration limits stepping to the first few updates after start:

# /etc/chrony/chrony.conf
makestep 1.0 3

That reads: step if the offset exceeds 1 second, but only if there have been no more than 3 clock updates since chronyd started. In other words, step at boot and slew from then on.

The consequence for a VM is direct. A guest paused for two hours and resumed comes back two hours behind. chronyd has been running for weeks, so its step allowance is long exhausted, and it begins to slew. Slewing is bounded by maxslewrate, and the Linux kernel’s own ceiling on the frequency offset a driver can set is 100000 ppm - ten per cent. Correcting a two-hour offset at ten per cent takes twenty hours, and the default rate is lower still.

The guest is now hours out of sync, converging slowly, and chronyc tracking reports it working correctly the entire time:

chronyc tracking
chronyc sources -v
timedatectl

Two ways to handle it, and the choice is a real trade:

# Allow stepping at any time. Correct for guests that are
# paused, snapshotted or migrated. Time can go backwards.
makestep 1.0 -1

A negative limit disables the “only the first N updates” restriction. This is the right setting for guests that get suspended or restored, and it means an application can observe wall-clock time moving backwards - which breaks anything using CLOCK_REALTIME to measure an interval, and is genuinely dangerous for a database that derives ordering from timestamps.

The alternative is to force a correction explicitly, at a moment you choose, as part of the resume procedure:

sudo chronyc makestep

That steps once, on demand. For a database guest, stepping during a controlled window after resume - before the application is allowed back in - is safer than leaving makestep permanently unrestricted.

The restored VM whose clock is three weeks old

A guest is restored from a snapshot taken three weeks ago, or resumed from a long-suspended state. Everything looks healthy: it boots, services start, the network works.

Then:

TLS handshakes fail with “certificate is not yet valid” against certificates issued in the last three weeks, and with an expiry error against ones that have since been renewed.
Kerberos and Active Directory authentication fails completely. The default maximum clock skew is five minutes, so three weeks is not a degraded state, it is a total authentication failure - reported as a credentials problem.
Log timestamps interleave incorrectly with every other host, so the incident timeline you build from central logging is wrong in a way that is very hard to spot.
Every systemd timer with a Persistent=true setting fires at once when the clock is corrected forwards, because each one concludes it missed its window.
Monitoring shows a gap and then a spike, and rate calculations over the discontinuity produce nonsense.

None of those symptoms says “the clock is wrong”. Every one of them points somewhere else - at the CA, at the directory service, at the monitoring system.

Make the clock check the first step of any restore or resume procedure, before the application is allowed to serve:

timedatectl
chronyc tracking | grep -E 'System time|Leap status'

The two-masters problem

Both the hypervisor and NTP can set the guest clock. When both are authoritative they fight, and the resulting jumps belong to neither.

On VMware, open-vm-tools can synchronise guest time to the host. If you also run chrony, disable the tools synchronisation and let chrony be the single authority.
On KVM, there is no equivalent continuous sync - the paravirtual clock gives the guest a good starting point and chrony disciplines from there, which is the desired arrangement by default.
On Hyper-V, the time synchronisation integration service does the same job as the VMware one and needs the same decision.

The rule is one authority. The paravirtual clock source is not an authority in this sense - it is how the guest reads time - so kvm-clock plus chrony is one authority, not two.

ptp_kvm: host time without the network

A KVM guest can read the host clock directly through a PTP device, avoiding the network round trip entirely. The ptp_kvm module creates a PTP hardware clock node, which chrony can then use as a reference clock:

sudo modprobe ptp_kvm
ls -l /dev/ptp0

# /etc/chrony/chrony.conf
refclock PHC /dev/ptp0 poll 2

The accuracy is far better than network NTP because there is no network. The trade-off is that the guest is now synchronised to its host, so the fleet is only consistent if the hosts are - which they should be, and which now becomes a hard requirement rather than a nice-to-have.

If /dev/ptp0 does not appear, the module is not loaded or the platform does not support it; the module is KVM-specific.

Validation

cat /sys/devices/system/clocksource/clocksource0/current_clocksource
timedatectl | grep -E 'synchronized|NTP service|RTC in local TZ'
chronyc tracking | grep -E 'Stratum|System time|Leap status'
chronyc sources -v | head -5
grep -E '^(makestep|maxslewrate|refclock)' /etc/chrony/chrony.conf 2>/dev/null

What each line should say: the clock source is the paravirtual one; System clock synchronized: yes and RTC in local TZ: no; the stratum is sensible and Leap status is Normal; there is at least one selected source; and the makestep policy is one you chose deliberately rather than inherited.

The RTC line matters more than it looks. A guest with its hardware clock in local time will be an hour wrong twice a year, on a schedule, in a way that only shows up in the first boot after the transition.

Alert on offset in both directions and on sync status

The common mistake is an alert on “clock is more than N seconds behind”. A guest resumed from a snapshot is behind; a guest whose clock was stepped forward past the real time is ahead; and a guest whose chronyd died is neither, but is drifting freely and will be both in turn.

Alert on three things:

The absolute offset, in both directions.
Whether the clock is synchronised at all - the Leap status from chronyc tracking, or System clock synchronized from timedatectl.
Whether chronyd is running, which the first two do not tell you if the metric is scraped from a stale cache.

A guest with no time source at all reports a small offset forever, right up until it does not. See linux-time-skew-operational-impact for the general form of this argument.