Expanding and Shrinking Kubernetes Clusters with Nutanix Kubernetes Engine (NKE)

In our previous article, we explored deploying a Kubernetes cluster using the Nutanix Kubernetes Engine (NKE). Today, we will delve into day-two operations for managing Kubernetes clusters, focusing on expanding and shrinking clusters. We will begin with a brand new Kubernetes cluster deployed in our previous article, consisting of 2 control plane nodes, 3 etcd, and 3 worker nodes.

Scaling Up and Shrinking the Cluster

Let’s start by scaling up our cluster. To do this, we need to add more worker nodes to the existing pool. We can increase the total number of nodes in the pool, and NKE will create a new VM and join it to the Kubernetes cluster. Here’s how to resize the worker pool:

1. Click on Node Pools in the NKE interface.

2. Select the existing pool, and click the Actions button.

3. Click Resize from the dropdown menu.

4. Increase the total number of nodes to four, and click Update.

NKE will create a new VM and join it to the Kubernetes cluster. We can see the new worker in the pool:

Now, let’s scale an existing deployment to run a few containers on the new worker:

1. Click on Deployments in the NKE interface.

2. Select the existing deployment, and click the Actions button.

3. Click Scale from the dropdown menu.

4. Adjust the replicas to 4, and click Update.

We can see four containers running on the new worker:

Deleting a Worker Node

Let’s delete one of the existing worker nodes. To do this, we need to click the Delete button in the NKE interface. However, before deleting, NKE will check the Kubernetes cluster’s health and ensure it’s OK before moving on to the next step. We can see that pods from worker 2 were restarted on workers 0 and 1, while worker 2 is being deleted:

And.., just to demonstrate, an error occurs after deleting one worker:

Pro tip: Check the /home/nutanix/data/logs/karbon_core.out log file. This log contains all the information you need to troubleshoot NKE issues.

Shrinking the Cluster

Now let’s shrink our pool to one node using the resize button as before:

1. Click on Node Pools in the NKE interface.

2. Select the existing pool, and click the Actions button.

3. Click Resize from the dropdown menu.

4. Set the total number of nodes to 1, and click Update.

NKE will delete two worker nodes. This is a step-by-step procedure, and workers will be deleted one by one. Before deleting, NKE drains the node (I believe it uses something like kubectl drain), and pods restart on other nodes in the cluster. At each step, NKE will check the Kubernetes cluster’s health and ensure it’s OK before moving on to the next step:

We can see that the cluster has been successfully shrunk to one node:

Non-Default Pools

As a reminder, the minimum number of workers for a production cluster is three. However, we can still shrink our original pool to one node even though it’s not the default pool:

I have one worker node in the default pool and two nodes in the new one. You may remember that before, we couldn’t shrink our original pool to one node, but now we can. In this situation, if you try to shrink small-pool, you will face an error, as we’ve faced before, because the total number of nodes will be less than three:

Notice, that we can’t delete our original pool. It should contain at least one node. That’s all for now. I hope you know better how to expand worker pools, add another one, or shrink them. Don’t forget about the minimum worker’s count.