The monitoring metrics described in this article cover only the basic Kubernetes metrics, not business-related metrics. The components involved are prometheus-server, kube-state-metrics, and node-exporter, and the data retention period is 3 days.

1. Monitoring-related components in the cluster

1
2
3
4

helm -n monitor list

NAME    	NAMESPACE      	REVISION	UPDATED                                	STATUS  	CHART            	APP VERSION
prom-k8s	monitor	        1       	2022-05-12 16:47:53.789549796 +0800 CST	deployed	prometheus-15.0.1	2.32.0

1
2
3
4
5
6
7
8

kubectl -n monitor get deploy,daemonset

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prom-k8s-kube-state-metrics   1/1     1            1           102d
deployment.apps/prom-k8s-prometheus-server    1/1     1            1           102d

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prom-k8s-prometheus-node-exporter   20        20        20      20           20          <none>          102d

2. How the metrics are calculated

• Total number of nodes in the cluster

count(kube_node_created{app_kubernetes_io_instance="prom-k8s"}) by (cluster)

• Cluster version

sum by (cluster,kubelet_version)(kube_node_info{app_kubernetes_io_instance="prom-k8s"})

• Number of running PODs

sum(kube_pod_info) by (cluster)

• Memory used by the monitoring components

sum(container_memory_working_set_bytes{image!="", namespace="monitor"}) by (cluster)

• CPU used by the monitoring components

sum (rate (container_cpu_usage_seconds_total{namespace="monitor"}[5m])) by (cluster)

3. Analysis of the statistical data

The average POD count per cluster is 306, the average memory usage is 1246 MB, and a single POD consumes roughly 40 MB of memory. CPU consumption can basically be ignored.