Kubernetes platform management software runs on Kubernetes and is used to manage the resource objects running on Kubernetes.

1. Testing Approach

Test the management capability of the platform software under a given load and a given cluster scale, not the management capability of Kubernetes. The platform software’s management capability is mainly reflected in the ability to create, read, update, and delete workloads and PVs through the UI, and to view the monitoring and logs of workloads directly in the UI.

Being clear about what is being tested and why matters a great deal. The test target is not Kubernetes. For Kubernetes-related tests, the community will provide better answers. What needs to be tested here is the platform software’s ability to manage Kubernetes resource objects.

2. Test Combinations

Based on the recommendations of the Kubernetes community:

• No more than 5,000 nodes
• No more than 150,000 pods in total
• No more than 300,000 containers in total
• No more than 100 pods per node

All tests must be carried out under the recommended configuration of the relevant components. Since upstream provides recommendations, downstream just needs to follow them, and any problems can be submitted back upstream.

2.1 Load Ratio

A commonly used sequence is 50%, 90%, 99%. That is, hold the Kubernetes cluster’s load level at 50%, 90%, and 99%, and then test the relevant metrics.

The type of workload is also something to think about. Workloads that are widely used and recognized by the community are the best choice, but when it comes to stress testing, there is not yet a consensus on test workloads or on accepting the test results of any particular workload.

The main recommendation here is to use the examples provided by the Kubernetes community:

• WordPress with MySQL - https://github.com/kubernetes/examples
• Go app with Redis - https://github.com/kubernetes/examples
• Java shopping - https://github.com/danielbryantuk/oreilly-docker-java-shopping/tree/master/kubernetes
• Nginx - https://github.com/kubernetes/examples/blob/master/staging/pod

These workloads can basically cover the common types, and they also make comparisons with similar products easier.

2.2 Number of Nodes

Following the large-cluster best practices provided by Kubernetes officially, the cluster scale is divided into six tiers:

The specific machine configurations can be found in Google Cloud as follows:

2.3 Etcd Configuration

For large-scale Kubernetes clusters, Etcd configuration becomes very important. Because the Kubelet on every node needs to connect to Etcd, adding nodes puts the most direct pressure on Etcd.

According to the recommended configuration from the Etcd community, just configure Etcd according to the number of nodes.

The number of Etcd nodes can be kept at 3, 5, or 7; more is not better, since more nodes make writes slower. For details, see: Etcd and Etcdctl in Practice.

2.4 Test Equipment and Cost

Running stress tests on large-scale Kubernetes clusters will certainly cost a lot of money.

If there are enough physical machine resources, you can install IaaS software on the physical machines and test based on VMs. Typically an IaaS cloud provider maps 2 vCPUs to one physical CPU, while memory is sold 1:1. During virtualization, you need to ensure that physical resources have some redundancy; otherwise the test results may not be meaningful as a reference.

If there are not enough physical machines, then you can only fall back on cloud servers. Using orchestration tools such as Terraform is a good choice, since it can create resources quickly. Once testing is done, they can be destroyed quickly too, which reduces part of the cost. For reference: How to use Terraform Provider to deliver IaC-level applications.

3. Test Output

• The maximum number of Nodes that can be managed
• The maximum number of Workloads that can be managed
• The maximum number of PVs that can be managed
• The maximum log volume that can be sustained (frequency, size, related to the number of Nodes)
• The maximum monitoring volume that can be sustained (related to the number of Nodes)
• Recommended configuration at each tier
• Management capability at each tier
• Fault tolerance and recovery capability (HA) of the platform management software under each load tier

4. Test Dimensions

4.1 Feature Points

• UI pages
• Query workloads
• Manage workloads/PVs
• Query/search logs
• View monitoring
• …(core features)

4.2 Observed Metrics

• Rendering (whether the relevant features can be opened)
• Speed (within 2 seconds; over 10 seconds counts as a failure)
• Accuracy (whether operations behave correctly and returned data is correct)
• High availability (HA)

5. References

• https://kubernetes.io/docs/setup/best-practices/cluster-large/
• https://cloud.google.com/compute/docs/machine-types
• https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/hardware.md