1. Local Storage Capacity Required disk size (GB) = data retention period _ samples fetched per second _ metric data size / 1024 / 1024 / 1024 Where Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d]) Average size of a sample within one hour: rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d])/rate(prometheus_tsdb_compaction_chunk_samples_sum[1d]) The disk consumption for one day (86400 seconds) can be queried directly in Prometheus:

Hosts Host memory usage exceeds threshold 1 - node_memory_MemAvailable_bytes{mode!="idle"} / node_memory_MemTotal_bytes Threshold: 0.9 Host CPU usage exceeds threshold 1 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (host_name) Threshold: 0.85 Host disk usage exceeds threshold 1 - avg without (fstype)(node_filesystem_free_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'} / node_filesystem_size_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'}) Threshold: 0.8 Windows Windows host memory usage exceeds threshold 1 - 1 * windows_os_physical_memory_free_bytes{job="windows_exporter",mode!

If you need to monitor the network status between two hosts, or between a host and an external service, then the Blackbox Exporter mentioned in this article is worth a try. 1. Installing Blackbox 1.1 Deploying on a Host Download the binary package 1 2 3 4 5 wget https://github.

1. Why kube-status-metrics is needed Kubernetes monitoring mainly focuses on two kinds of metrics: Basic performance metrics CPU, memory, disk, network, and so on. These can be collected by deploying node-exporter as a DaemonSet and having Prometheus scrape the relevant metrics. Resource object metrics The replica count of a Deployment, the running state of a Pod, and so on.

1. Download the Binaries 1 2 3 4 wget https://github.com/thanos-io/thanos/releases/download/v0.26.0/thanos-0.26.0.linux-amd64.tar.gz tar xvf thanos-0.26.0.linux-amd64.tar.gz mv thanos-0.26.0.linux-amd64/thanos /usr/bin/ rm -rf thanos-0.26.0.linux-amd64* 2. Install Thanos Query 1 vim /etc/systemd/system/thanos-query.service 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 [Unit] Description=Thanos Query After=network-online.target [Service] Restart=on-failure ExecStart=/usr/bin/thanos query \ --log.