1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.

1. Install node exporter 1 2 3 wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/ If you cannot reach GitHub, refer to: Several ways to access GitHub from mainland China . 2. Configure and start the exporter Add a Systemd configuration 1 vim /etc/systemd/system/node_exporter.service Add the following content: 1 2 3 4 5 6 7 8 9 10 11 12 [Unit] Description=Node Exporter After=network.

The Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge. 1. Terminology Grafana A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.

Internet services often depend on tens of thousands of hosts, and those hosts are deployed all over the world, on top of a tangled mess of user environments. To keep a service stable and reliable, you need to monitor it. The goals of monitoring can be considered from several angles: from the system dimension, understand CPU, memory, and disk usage; from the network monitoring dimension, understand upstream and downstream rates; from the service monitoring dimension, understand the health and availability of the application.