Homelab Update: Nested ESXi Lab Environment and Storage Performance Issues Resolved
As an avid tech enthusiast and virtualization fanatic, I’ve recently had to say goodbye to my beloved Zettagrid Labs, and boy, have I missed it! The absence of a lab environment has made it challenging for me to generate content related to technology. However, all that changed on Monday when my new homelab arrived, and I couldn’t wait to dive in and start setting things up. Little did I know that I would encounter some issues with storage performance, but more on that later.
For those who might be interested, I purchased a SuperMicro SYS-5028D-TN4T based on this TinkerTry Bundle, which has gained popularity among vExpert homelabers. This powerhouse of a system features an Intel Xeon D-1541 CPU and 128GB of RAM, making it the perfect choice for my Nested ESXi lab environment. The system comes equipped with an embedded Lynx Point AHCI Controller that supports up to six SATA devices and is listed on VMware’s Compatibility Guide for ESXi 6.5.
I created a new VM and kicked off a fresh install of Windows, but what should have taken around two hours to complete took a whopping three days! That’s when I knew something was amiss. The issue began with uploading the Windows 2016 ISO to the datastore, which took about 30 minutes. I suspected that the storage performance might be the culprit, but I couldn’t have imagined just how poor it would be.
Upon further investigation, I discovered that the native driver included with ESXi 6.5 was causing the slow storage performance. The driver, listed as ahci version 3.0.22vmw in the VMware Compatibility Guide, was causing write latencies of over 2000ms and read latencies of around 700-1000ms. These numbers are way beyond what I would expect from a decent class SSD. To make matters worse, ESXTOP showed that my new VM had even worse write latency.
After some research and guidance from the vExpert Slack Homelab Channel, I found that disabling the native VMware driver for the storage controller resolved the issue. The driver was listed as a device driver in the installed module list, and there was also an additional native VMware driver present. By disabling the native driver as per this VMware KB article, I was able to get the storage controller to use the correct device driver listed in the Compatibility Guide.
After a host reboot, I checked to see if the storage controller was using the corrected driver, and lo and behold, it was! Not only that, but the six HBA ports were now visible, as opposed to just one before. I once again created a new VM and installed Windows, and this time around, the install completed in a little under five minutes, a far cry from the three days it took earlier.
Upon running CrystalDiskMark, I was relieved to see that my SSDs were now performing as expected, with read and write speeds within the expected range. It’s safe to say that this post should save anyone else who might be experiencing similar issues with storage performance and the native VMware driver in ESXi 6.5.
References:
http://www.supermicro.com/products/system/midtower/5028/SYS-5028D-TN4T.cfm
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2044993
Copyright © 2024 VIRTUALIZATION IS LIFE!. Powered by WordPress & Infinite Theme