Tuesday, July 19, 2011

Fixing Time Errors on VMware vSphere and ESX Hosts

Time synchronization across a Windows domain is very important.  If a member server's clock varies more than 5 minutes from other domain servers, Kerberos tickets will fail.  This causes random authentication errors for users and/or applications which are sometimes difficult to troubleshoot.

Normally, time is synchronized in a Windows domain using the domain hierarchy.  The domain controller holding the PDC Emulator FSMO role is normally configured to get time from an authoritative NTP time source, and syncs time with all the other DCs in the domain.  The domain clients in each site sync time from the DCs in their local site, maintaining a relatively close synchronization of time across the domain.

Virtual machines are no different than physical computers and normally sync time using the same domain hierarchy.  Lately, however, I've seen VMs running on VMware vSphere boot up with random time differences from the domain.  I've seen this problem with three different clients lately, so I figured this might be a pervasive enough issue to blog about.

The trouble happens when the VMware vSphere, ESX or ESXi host does not have an accurate source of time, or time "drifts" due to an inaccurate system clock module.  vSphere and ESX hosts run a proprietary operating system and are not domain member servers, therefore they do not participate in domain hierarchy time synchronization. 

Most companies that use VMware hosts use vCenter to manage these hosts and their VMs.  Often, the servers that run vCenter are domain member computers and administrators think that since the vCenter syncs time with the domain, the hosts and VMs do, too.  Not true.  You need to configure the vSphere or ESX hosts to sync time from an accurate time source, otherwise the VM guests may start up with the wrong time - this can happen even if time synchronization between the virtual machine and the ESX server in VMware Tools is not enabled.

Here's how to configure your vSphere or ESX hosts to get time from an authoritative source.
  • Logon to vCenter and select your vSphere or ESX host.
  • Click the Configuration tab and then Time Configuration under the Software heading.  Notice that the time on the vSphere host does not match the domain time shown on the Windows client running vCenter .

  • Click Properties in the top left of the Configuration tab.  This opens the Time Configuration window.

  • Click the Options button and add a new NTP server that is the accurate source of time.  I recommend using the PDC emulator, since it should already be configured as an authoritative time source. 

  •  Select the checkbox to Restart NTP service to apply changes and click OK twice to close the Time Configuration window.  You will see that the vSphere/ESX host now has the correct time and is configured to use dc01.companyabc.com as its time server.

You may need to restart the VM guests running on that VMware host to have them sync time with the domain.  The Windows Time service will not correct the time on the VMs if it varies too much from domain time.  All domain computers sync time when they start up on the domain, regardless of how far out of sync they were.

I have not seen this type of behavior with Hyper-V, only vSphere, ESX and ESXi hosts. 9:34 PM