My Journey from Infrastructure Admin to Cloud Architect: Lessons Learned from Nested vSAN Homelab Installations
As an infrastructure admin turned cloud architect, my journey has been filled with challenges and lessons learned. One of the most valuable experiences has been working with nested vSAN homelab installations that constantly suffer power losses and network issues. These environments have taught me a plethora of useful troubleshooting tricks, but more importantly, they have given me a deeper understanding of the importance of thorough testing and consulting technical support before applying any changes in production environments.
Recently, I encountered an issue in my lab where I wanted to see if it was vSAN related. I discovered an option in ESXi to boot hosts with selected modules disabled. This feature allows you to press Shift+O to disable modules during host boot-up. I decided to experiment by disabling the vSAN module, and here’s what I learned:
To disable the vSAN module, I used the following command:
jumpstart.disable=vsan,lsom,plog,virsto,cmmds
After disabling the vSAN module, I verified if it was loaded by running the following command:
esxcli system module list
vCenter recognized that the host in the cluster did not have its vSAN service enabled. This was not a surprise, as I had intentionally disabled the vSAN module during boot-up. However, what caught my attention was that the host was still part of my vSAN cluster even though it did not have the vSAN module loaded.
To make the host load back the vSAN module, I simply restarted it. After the host rebooted, the vSAN module was loaded again, and the host was back in the vSAN cluster. Interestingly, I received an additional notification from vCenter that I had a partition in my cluster before the restart. This was a nice surprise, as it confirmed that my vSAN cluster was still functional even after disabling the vSAN module.
This experience taught me a few valuable lessons:
1. Thorough testing is crucial: Before applying any changes in production environments, it’s essential to test them thoroughly in a controlled environment like a homelab. This helps you identify potential issues before they impact your users.
2. Consult technical support when unsure: As an infrastructure admin turned cloud architect, I’ve learned that consulting technical support is crucial when you’re not sure about the results of a command or configuration change. Technical support engineers can provide valuable insights and help you avoid potential pitfalls.
3. vSAN is resilient: My experience with nested vSAN homelab installations has shown me that vSAN is incredibly resilient. Even when hosts experience power losses or network issues, vSAN data seems to survive these unexpected failures. It’s just the cluster services that sometimes need a little help.
4. Homelabs are essential for learning: Homelabs provide a safe environment to experiment with new technologies and configurations. They allow you to learn from your mistakes without impacting your users or production environments.
5. Always document your findings: Keeping track of your experiments, observations, and lessons learned is crucial. Documenting your experiences helps you reflect on what worked well and what didn’t, and it allows you to share your knowledge with others.
In conclusion, my journey from infrastructure admin to cloud architect has been filled with challenges and opportunities to learn. Working with nested vSAN homelab installations has taught me valuable lessons about the importance of thorough testing, consulting technical support, and the resilience of vSAN. These experiences have helped me become a better cloud architect, and I’m excited to continue learning and growing in this field.