TopicGems: Why a VM Clock Can Drift From the Host Clock

I ran sudo apt update in one of my lab VMs and got an error I'd never seen before:

E: Release file for http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease is not valid yet (invalid for another 41d 2h 15m 8s). Updates for this repository will not be applied.

Not valid yet. As in, the update was from the future?

It wasn't. The VM was stuck in the past. I had reverted it to a snapshot from about six weeks earlier, booted it, and the guest was completely convinced it was still that day. The host sitting right underneath it knew better.

Two clocks, one physical machine, six weeks apart.

That was the day I stopped assuming VM time "just works."

First, the VM doesn't really own a clock

A physical computer has a small battery-backed clock on the motherboard and timer hardware that keeps counting whether the OS is paying attention or not.

A virtual machine has none of that. The hypervisor pretends to be that hardware. When the guest asks "what time is it?", it reads a virtual clock, or it counts timer ticks that the hypervisor hands over.

Think of someone counting seconds out loud while a friend taps their shoulder every second. If the friend stays reliable, the count is right. If the friend gets distracted, falls asleep, or gets pulled away for a few minutes, the counter has no idea. They just carry on from wherever they were.

That's drift. The guest's idea of time and real time slowly (or suddenly) stop agreeing.

Here's the thing that took me a while to learn: drift comes in two very different shapes, and the fix depends on which one you have.

Creep: the clock loses or gains a few seconds over hours or days.
Jumps: the clock is suddenly off by minutes, hours, or weeks after something happened to the VM.

So the first question to ask when a VM's time looks wrong is a simple one.

Question 1: Did it creep, or did it jump?

Look at the timestamps in the guest's logs, or just compare the guest clock to your phone right now. Then think about what happened to the VM recently.

If it jumped, the VM was almost certainly frozen at some point. The guest's clock is part of its saved state, so when a VM is paused, suspended, or restored, the guest simply doesn't know that time kept moving outside. The usual suspects:

Reverting to a snapshot (my apt error, exactly this)
Pausing or suspending the VM
Closing a laptop lid while VirtualBox or VMware Workstation is running
Live migration between hosts
Restoring from a saved state or backup

Snapshots are handy, but they have side effects nobody mentions. I wrote about the storage side of that in How Cloud Snapshots Quietly Consume Storage, and time is another one on the list.

A small detail I found interesting: VirtualBox's Guest Additions don't always correct the clock the same way. Small gaps get nudged back gradually, so the clock seems to crawl toward the right time. Once the gap passes a threshold (20 minutes by default), it snaps the clock straight to the host time. That's why one lid-close left my VM slowly catching up, while another fixed itself in a blink.

If it crept, the VM was awake the whole time, and something was making its timer ticks arrive late or get lost. That's the next section.

Creep usually means the host was too busy

A vCPU isn't a real core. It's a thread the host has to schedule. If the host is overloaded, your VM's vCPU can be waiting in line, and the timer interrupts waiting with it.

You can actually see this from inside the guest. Run:

top

and look at the st value in the CPU line. That's steal time: the share of time your VM wanted to run but the host gave the CPU to someone else. If it's constantly above a few percent, the host is squeezed, and clocks can suffer for it.

I caused this myself. I gave a Kali VM 4 vCPUs on a 4-core laptop while a Windows VM was also running, thinking "more cores, faster lab." It was the opposite: everything stuttered, and the guest clock wandered. Dropping it to 2 vCPUs fixed both problems.

The other creep culprit is the clock source the guest is using. On Linux you can check:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

On KVM guests you'll normally see kvm-clock, which is built for virtual machines. On Hyper-V you'll see something with hyperv in the name. If you see something older like hpet or acpi_pm on a modern setup, that's worth a look, because those are slower and less precise.

Question 2: Is the host itself even correct?

This one embarrassed me.

"Drift from the host" quietly assumes the host is right. Most hypervisor tools copy the host's time into the guest, so if the host is wrong, the guest is faithfully wrong too.

I once "fixed" a guest clock three times before I checked the machine underneath. The host's NTP had been disabled after a reinstall, and it had been wandering the whole time.

Check the host first. On Windows:

w32tm /query /status

On Linux:

timedatectl status

You want to see the clock as synchronized and a real time source. Two minutes here can save you an afternoon.

One more thing that looks like drift but isn't: if the guest is off by an exact number of hours, that's not drift at all. That's a timezone or RTC mismatch. Windows stores local time in the hardware clock while Linux expects UTC, and a VM can end up confused about which one it's reading. On VirtualBox you can tell the VM to use UTC:

VBoxManage modifyvm "YourVMName" --rtcuseutc on

On Proxmox, there's a Use local time for RTC option in the VM's Options tab, and it's normally set for you when you pick Windows as the OS type, but it's worth confirming.

Question 3: Who's the boss of time inside the guest?

This was my actual mistake, and I suspect it's the most common one.

A guest can get its time from two places at once: the hypervisor tools (VMware Tools, VirtualBox Guest Additions, Hyper-V Integration Services, QEMU guest agent) and its own NTP service (chrony, systemd-timesyncd, Windows Time).

If both are active, they can fight. One nudges the clock, the other nudges it back, and you get jitter or odd little jumps that are annoying to trace.

I turned on chrony in a VM that also had host time sync enabled, and then spent a while wondering why the offsets looked so noisy.

The rule I follow now: pick one boss. For most of my VMs I let the guest's own NTP be the main source. Some hypervisors still do a sync at certain events (like resume or snapshot revert) even when periodic sync is off, so check your platform's timekeeping guidance for the exact settings instead of assuming a checkbox does everything.

A quick platform cheat sheet:

VirtualBox: Guest Additions handle sync. To disable host time sync for a VM: VBoxManage setextradata "YourVMName" "VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" 1
VMware: the time sync option lives in VMware Tools settings. Read VMware's own timekeeping guide for your version before changing .vmx options.
Proxmox / KVM: install qemu-guest-agent inside the guest and enable it in the VM's Options, so the host and guest can communicate properly.
Hyper-V: in the VM's settings, look under Integration Services for the Time synchronization checkbox.

And one caution for Windows: a VM that's part of a domain should normally take its time from the domain controller. Don't let the hypervisor push a different time in on top of that.

The five-minute health check

When a VM's clock looks wrong, this is what I run now, before touching anything else.

On Linux:

timedatectl status
chronyc tracking
chronyc sources -v

Look for "System clock synchronized: yes," a sensible time source, and a small offset in the tracking output.

On Windows (elevated prompt):

w32tm /query /status
w32tm /query /source

To force a correction:

sudo chronyc makestep

on Linux with chrony, or:

w32tm /resync

on Windows.

That makestep command is the one that fixed my apt error. It tells chrony to jump the clock right now instead of slowly nudging it. In chrony.conf there's also a line like makestep 1.0 3, which allows a big correction during the first few updates after startup. That's very handy for VMs that wake up far behind.

Why a wrong clock is more than cosmetic

A wrong time doesn't announce itself as a time problem. It shows up looking like something else:

TLS errors like "certificate is not yet valid" or "expired"
Kerberos login failures, since it usually tolerates only about 5 minutes of difference
Two-factor codes that never work, because they're based on 30-second windows
Cloud API errors such as AWS's RequestTimeTooSkewed when the gap passes about 15 minutes
Logs that don't line up, which is a real headache if you're ever investigating an incident and trying to build a timeline across machines

That last one matters if you do any security work. If the clocks disagree, your evidence disagrees with itself.

Things that looked like fixes but weren't

Rebooting the VM. It resets nothing if the time source is wrong or the host is off. You just reboot into the same problem.

Setting the time by hand with date -s. It works for about an hour, then drift returns. It can also confuse services that depend on steady timestamps, so I only use it for a quick lab reset.

Stepping the clock around on a busy server without thinking. Jumping time (especially backwards) on a live database or app server can cause strange side effects. On anything that matters, let it correct gradually or plan a maintenance window.

Restoring a snapshot and moving on. After a revert, the first thing I do now is force a time sync, before I test anything.

I did eventually go back to that broken VM. I'd spent a good ten minutes poking at mirror settings and repository files, convinced the problem was somewhere in apt.

One sudo chronyc makestep, and six weeks of lag vanished in a second. apt update ran clean.

If a VM starts behaving in ways that make no sense, run date before you run anything else. It's the cheapest check there is, and it has embarrassed me more than once.