ESXi Loses All Network Configuration… but Still Runs? 😱 Quick Fix and Explanation! This title maintains the urgency and surprise conveyed by the original title, while also adding a bit of personality with the use of emojis and a playful tone. The Quick Fix part of the title emphasizes that the post will provide a solution to the problem, while the Explanation! part indicates that the post will also cover the underlying reasons for the issue.

A strange situation occurred in one of my lab environments over the weekend. vSAN Health reported that one of the hosts had lost networking connectivity to the rest of the cluster. This is something I’ve seen intermittently at times, so I waited for the condition to clear up. When it didn’t clear up, I went to look at the host to put it into maintenance mode, but found that I wasn’t getting the expected vSAN options. I have seen situations recently where the enable vSAN option on the VMkernel interface had been cleared and vCenter thinks there are networking issues. I thought maybe it was this again.

Not that situation in itself was normal, but what I found when I went to view the state of the VMkernel Adapters from the vSphere Web Client was even stranger. No adapters listed! The host wasn’t reported as being disconnected and there was still connectivity to it via the Web UI and SSH. To make sure this wasn’t a visual error from the Web Client, I SSHed into the host and ran esxcli to get a list of the VMkernel interfaces. Unfortunately, I couldn’t find vmknic for dvsID: xxxxx. From the CLI, I also couldn’t get a list of interfaces.

I tried restarting the core services without luck and still had a host that was up with VMs running on it without issue, yet reporting networking issues and having no network interfaces configured per the running state. Going to the console…the situation was not much better. Nothing…no network or host information at all (smile). Not being able to reset the management network, my only option from here was to reboot the server.

Upon reboot using last known good configuration as shown below:

All previous network settings were restored, and I had a list of VMkernel interfaces again present from the Web Client and from the CLI. Because of the “dirty” vSAN reboot, as is usual with anything that disrupts vSAN, the cluster needed some time to get itself back into working order, and while some VMs were in an orphaned or unavailable state after reboot, once the vSAN re-sync had completed, all VMs were back up and operational.

The workaround to bring back the host networking seemed to do the trick, but I don’t know what the root cause was for the host to lose all of its network config. I have an active case going with VMware Support at the moment with the logs being analyzed. I’ll update this post with the results when they come through.

ESXi Version: 6.7.0.13006603

vSphere Version: 6.7.0.30000

NSX-v: 6.4.4.11197766

VMware Troubleshooter!

A VMware Troubleshooting Internet Resource on everything VMware related