📈Improving performance of resource discovery

As soon as a cluster is registered in gopaddle, gopaddle installs a few add-ons under gopaddle-servers namespace. The kubeagent add-on is responsible for discovering Kubernetes resources, tracking resource updates, and annotating the resources with runbook information. For large multi-node clusters, it is recommended to increase the capacity of the kubeagent to improve the performance of the Cluster Resource View and Runbook discovery.

Increasing capacity of multipass VM in case of MacOS

1. Before increasing the capacity of the kubeagent, check the current memory capacity and consumption of the multipass microk8s vm on MacOS.

multipass info microk8s-vm

2. To increase the capacity of the microk8s multipass VM on MacOS, please follow the prerequisites step 5 stated here.

Increasing capacity of kubeagent

1. Enable metrics-server to check the current consumption of the kubeagent pod.

microk8s enable metrics-server

2. Get the current usage of the kubeagent pod.

kubectl top pod -n gopaddle-servers

3. If the resource usage of the kubeagent exceeds the limits set for the deployment, increase the limits of the kubeagent deployment. The default requests and limits of the kubeagent deployment is as below:

        resources:
          limits:
            cpu: 1200m
            memory: 4G
          requests:
            cpu: 500m
            memory: 1G

4. Update the requests and limits of the kubeagent, if necessary.

kubectl edit deploy/kubeagent -n gopaddle-servers

5. Once the kubeagent deployment is updated, wait for the kubeagent to move to ready state

kubectl wait --for=condition=ready pod -l app=kubeagent -n gopaddle-servers  --timeout=15m

6. In the gopaddle UI, Refresh the cluster View -> Resources to check the performance

Increase QPS and Burst limits of kubeagent

1. Check the kubeagent logs and look for any throttling issues:

kubectl logs -l app=kubeagent -n gopaddle-servers --all-containers=true --tail=100

A sample throttling message would look like this:

Waited for 1.01761187s due to client-side throttling, not priority and fairness, request: GET:https://10.152.183.1:443/api/v1/nodes

2. Increase the QPS and Burst parameters by editing the command line arguments of the container in the kubeagent deployment. The default QPS and Burst values are 100 and 200 respectively.

3. Edit the kubeagent deployment.

kubectl edit deploy/kubeagent -n gopaddle-servers

4. Update the command line arguments in the kubeagent specification.

containers:
      - args:
        - |-
          #!/bin/bash
          # echo "----------- start conatainer ------------"
          ./kubeagent --qps=200 --burst=400
        command:
        - /bin/sh
        - -c

5. Once the kubeagent deployment is updated, wait for the kubeagent to move to ready state.

kubectl wait --for=condition=ready pod -l app=kubeagent -n gopaddle-servers  --timeout=15m

6. In the gopaddle UI, Refresh the cluster View -> Resources to check the performance