Load testing Kubernetes clients without breaking the bank

What if I told you can do performance tests on Kubernetes Clients without spending too much money, and much faster than conventional methods?

Up until recently, I used to spend a few hundred dollars on throw away clusters that I created for performance tests.

These clusters would usually have 100+ nodes, and up to 10.000 pods on each cluster. They were created and destroyed on the same day for obvious reasons.

Not only this was an expensive process, but it was also extremely slow to provision all these resources and containers.

I use a MacBook Pro M1 Pro for development. It's an incredibly performant machine, so my first thought was to run everything locally on minikube.

I knew I would not be able to run as many nodes and pods as on the cloud, but I really thought I could run a very large number of them.

We got close to 500 pods spread across 5 minikube nodes, but unfortunately it did not work it. But even with idle pods, it was just too much for a single host.

My next move was to do what any sensible developer would do: ask the Internet — aka Reddit.

Redditor u/omatskiv mentioned KWOK, which stands for Kubernetes WithOut Kubelet, pretty clever name, right?

I case you don't know, Kubelet is a component that runs on each node and is responsible for managing the lifecycle of containers, like creating and deleting them.

Running Kubernetes without Kubelet basically means no containers will ever be created, which might sound pretty useless, but from a Kubernetes Client perspective, all we need is the API Server backed by an etcd instance with a lot of objects.

That's essentially what KWOK does, which is truly a fantastic tool for testing clients.

The following command creates a new cluster on Docker. While a new minikube cluster might take up to a minute to complete, creating a KWOK cluster is almost instant.

$ kwokctl create cluster --name=local

Although there's no kubelet on nodes, most of kubernetes restrictions and rules still apply.

As an example, we can't create pods without first having nodes on the cluster. It's also not possible to create pods that have resource requests higher than what's available in the cluster.

Adding nodes to a KWOK is just like creating any other resource: a YAML file and Kubectl apply it.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Node
metadata:
  name: node-0
status:
  phase: Running
  allocatable:
    cpu: 2
    memory: 4Gi
    pods: 110
  capacity:
    cpu: 2
    memory: 4Gi
    pods: 110
EOF

Within the YAML definition we have full control over things like: how many pods it can host, how much cpu/memory it has, what % of it is allocatable and so on.

Creating a Node or a Pod in KWOK is just like making a payment transaction on a sandbox environment. There's no real money involved, it's all just a record on the database, which for some use cases, that's all we need!

I then decided to create a single server on Hetzner and host four KWOK clusters in it.

Why four clusters?

Aptakube can connect to multiple clusters simultaneously, and while some customers have been using it with 30 clusters simultaneously, the majority of users are likely to connect to up to six clusters at any time.

Each cluster then hosts about 500 Nodes and 5000 Pods, which gives me a total of 20.000 Pods. This is way higher than I used to before on Digital Ocean.

And you know the best part? I can keep this server running 24/7, and it only costs me about $15/mo.

20.000 Pods for just $15/mo!

It's way cheaper and faster than spinning up real kubernetes clusters, which also means I can do these tests more often.

I'm sharing some code examples below, but it's basically just a script that loops from 0 to 500 and calls Kubectl apply on each iteration.

I do this to create all the Nodes and Deployments. The controllers will then do its job and spawn Pods based on each Deployment spec.

Creating Nodes using Node.js

import { execSync } from "child_process";

async function createNode(name, cpu, memory, pods) {
  const yaml = `
apiVersion: v1
kind: Node
metadata:
  name: ${name}
status:
  phase: Running
  allocatable:
    cpu: ${cpu}
    memory: ${memory}
    pods: ${pods}
  capacity:
    cpu: ${cpu}
    memory: ${memory}
    pods: ${pods}`;

  try {
    execSync("kubectl apply -f -", {
      input: yaml,
      stdio: ["pipe", "pipe", "pipe"],
    });
    console.log(`Node ${name} created.`);
  } catch (error) {
    console.error("Error:", error.stderr.toString());
  }
}

for (let i = 10; i < 500; i++) {
  await createNode(`node-${i}`, "4", "16Gi", "110");
}

Creating Deployments using Node.js

import { execSync } from "child_process";

async function createDeployment(name, replicas) {
  const yaml = `
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${name}
  labels:
    app: ${name}
spec:
  replicas: ${replicas}
  selector:
    matchLabels:
      app: ${name}
  template:
    metadata:
      labels:
        app: ${name}
    spec:
      containers:
      - name: ${name}
        image: nginx
        ports:
        - containerPort: 80`;

  try {
    execSync("kubectl apply -f -", {
      input: yaml,
      stdio: ["pipe", "pipe", "pipe"],
    });
    console.log(`Node ${name} created.`);
  } catch (error) {
    console.error("Error:", error.stderr.toString());
  }
}

for (let i = 0; i < 500; i++) {
  await createDeployment(`nginx-${i}`, 10);
}

While most values are hardcoded in the examples above, you can easily change them to be dynamic and randomly generated to create even more realistic scenarios.

That's all folks, I hope this post was useful for you!