Troubleshooting vCLS VMs Failing to Power On in Nested ESXi Environments
As a Solutions Architect in the VMware Cloud Foundation (VCF) Division at Broadcom, I recently encountered an issue when deploying a new VCF Workload Domain using the VCF Holodeck Toolkit, which leverages Nested ESXi. Specifically, I noticed that the vSphere Cluster Services (vCLS) VMs kept failing to power on and threw the following error message: “No host is compatible with the virtual machine.” This was quite strange, especially since the vCLS VMs ran fine when the VCF Management Domain was setup.
After investigating the issue, I found that the vCLS VMs expected to have the MWAIT CPU instruction exposed, as indicated by the following message in the vmware.log file: “Feature ‘cpuid.mwait’ was 0, but must be 0x1.” This is because newer vSphere releases expect to configure Per-VM EVC, but the vCLS VM may not function properly within a Nested ESXi environment.
To resolve this issue, I worked with Ben Sier, who works on Holodeck, and he provided a workaround. Here are the steps to resolve the issue:
Step 1 – Upgrade ESXi VM Compatibility (vHW)
We first need to upgrade the ESXi VM Compatibility (vHW) of the vCLS VMs to the latest version (should be v14 if you are using VCF 5.1 or 8.0 Update 2). To do this, right click on the vCLS VM and select Upgrade VM Compatibility.
Step 2 – Enable Per-VM EVC Configuration
Log back into your vCenter Server and then enable the Per-VM EVC configuration but set it to disabled. In a few seconds, you should notice that the vCLS VM can now be successfully powered on and will enable Per-VM EVC automatically.
If you wanted to automate these steps, here are quick PowerCLI snippets that can be used:
“`powershell
# Step 1 – Upgrade ESXi VM Compatibility (vHW)
Get-VM -Name
# Step 2 – Enable Per-VM EVC Configuration
Get-VM -Name
“`
Conclusion
In this blog post, we discussed an issue that may arise when deploying vCLS VMs in a Nested ESXi environment, where the VMs fail to power on with the error message “No host is compatible with the virtual machine.” We also provided a workaround to resolve the issue by upgrading the ESXi VM Compatibility (vHW) and enabling the Per-VM EVC configuration. These steps can be automated using PowerCLI snippets.
I hope this blog post helps you troubleshoot any issues you may encounter when deploying vCLS VMs in Nested ESXi environments. If you have any further questions or concerns, please feel free to reach out to me.