Here's a fun one...starting sometime last night something happened to one of the hosts in one of our ESXi 5.0 U2 clusters. DRS tried 4 migrations overnight (standard vmotion, not storage) to this particular host and all 4 failed at 9% with this error:
A general system error occurred: The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the virtual machine to continue running on the source. To avoid this failure, either increase the maximum allowable switchover time or wait until the virtual machine is performing a less intensive workload.
The kicker? The VMs were all found powered off. Even better? Even stranger, during the time between the start of the failed vmotion and the server powering off, all entries in the Windows event logs showed up as 6 hours ahead of the current time.
All existing VMs on the affected host remained online and working fine including time sync on the host and VMs. Once I restarted the management agents on the host it was able to accept vmotions again.
This of course puts a damper on our stance that it's "never a VMware issue", so I'm wondering if anyone has run into this issue before and if it seems to be a recurring issue or if I just ran into a one off fluke? I'm pretty amazed that this would actually power the VMs off, instead of doing what the error message states, and just leaving them as is on the source host.
Any ideas?
Adam