Schedule VM Hardware version upgrade – Lessons Learned!

Hello,

Recently I was having a discussion with one of my customers over how to schedule or automate  VM Hardware version upgrade along with their quarterly Guest OS patching activity.

Unlike VMware tools upgrade VM hardware version upgrade was difficult thing to achieve for the customers as it required a lot of co-ordination between the infra and the app team.

We came across below PowerCLI script which can be used to schedule VM hardware version upgrade. Once a VM is configured for hardware version upgrade the hardware is automatically upgraded to the version specified during the next power cycle.

We were excited to run it as it was meeting all our requirements.

But there is a catch and we will discuss about it in this blog so that you don’t run into similar issues.

I will suggest not to run the script across the entire virtual infrastructure at once. Plan it and perform it on a set of VM’s first, observe the VMs for a week or two and when you are confident run it against all the VMs. This will help you reduce the overall impact.

Below is the script to schedule the hardware version upgrade :-

foreach($vmlist in (Get-Content -Path D:\temp\123.txt))

{

$vm = Get-VM -Name $vmlist

$do = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec

$do.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo

$do.ScheduledHardwareUpgradeInfo.UpgradePolicy = “always”

$do.ScheduledHardwareUpgradeInfo.VersionKey = “vmx-11”

$vm.ExtensionData.ReconfigVM_Task($do)

}

Now if you see the script we are trying to schedule hardware upgrade to version 11 which is compatible with ESXi 6.0.

So if you have VM’s running lower hardware version on ESXi 6.0 you can run this script for all those VM’s.

The important thing to keep in mind is VM hardware version 11 is not compatible with ESXi 5.5 so ensure all the hosts in the cluster are running 6.0 version else the VM’s will not be able to migrate to 5.5 hosts.

In our case the customer by mistake ran this script for all the VM’s which were part of the cluster with ESXi 6.0 hosts.  Hence the script also ran on VM’s which were already running version 11.

Later we realized the VM’s running version 11 could not be migrated to other ESXi hosts and threw an error “The destination host is not compatible with the hardware version to which the virtual machine is scheduled to be upgraded”

We are still not sure what is the reason as the version provided in the script was 11 and the destination hosts does support version 11. But since the VMs are already running version 11 may have caused this issue.

Now the worst part is the resolution mentioned in the KB  https://kb.vmware.com/s/article/2080107 is to Power off the VM and make changes in the vmx file to revert the settings.

The above script will add below lines related to hardware version upgrade in vmx and we have to manually remove it.
virtualHW.scheduledUpgrade.when = “always”
virtualHW.scheduledUpgrade.state = “done”
tools.upgrade.policy = “upgradeAtPowerCycle”

 

Powering off the virtual machine was not an option in our case as the changes were made on ~700 production VMs running latest hardware version.

Then we came across a script in another blog http://www.running-system.com/error-destination-host-not-compatible/ which can help make these changes even without powering off the VM. We tried it out on one VM and had a sigh of relief. It worked!!

The below script was a saviour.

Before running the script on all the affected VM’s we first ran it on one VM named “test” and validated that everything was working as expected.

Note :- If you want to run it on all the VM’s part of vCenter server remove VM name test after Get-VM in the second line.

Connect-VIServer vCenter_FQDN

$vms = Get-VM test
foreach ($vm in $vms)
{
write-host “$($vm.name)”
$vmguest = $vm | Get-VMGuest

write-host $vmguest.OSFullName

$vmview = $vm | Get-View
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo
$vmConfigSpec.Tools.ToolsUpgradePolicy = “manual”
$vmview.ReconfigVM($vmConfigSpec)

$spec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
$spec.ScheduledHardwareUpgradeInfo = New-Object -TypeName VMware.Vim.ScheduledHardwareUpgradeInfo
$spec.ScheduledHardwareUpgradeInfo.UpgradePolicy = “never”
$spec.ScheduledHardwareUpgradeInfo.ScheduledHardwareUpgradeStatus = “none”
$vm.ExtensionData.ReconfigVM_Task($spec)
$vm.ExtensionData.Config.ScheduledHardwareUpgradeInfo

# end foreach ($vm in $vms)

}

 

So the learning here is to first filter the VM’s which are running lower version than the version compatible with ESXi hosts in the cluster.

What does that mean is, if you have ESXi 6.5 in cluster you should filter out all the VMs running hardware version 11 or lower. Do not run it on VM’s running hardware version 13.

I hope you don’t run into similar issues in your environment but even if you do there is a script available to get you out of it without a downtime.

 

Hope this helps!