1. Layering of Monitoring As shown above, when building a monitoring system, two strategies are used: Layered monitoring. The benefit of separating IaaS, MySQL middleware, and App layer monitoring is that the systems have high availability and fault tolerance between them. When App layer monitoring stops working, IaaS layer monitoring will immediately make that visible.

The hard part of a monitoring system is storing large volumes of time-series data and providing high-performance query capability; the hard part of an alerting system is designing an efficient alert engine and implementing a flexible alert escalation mechanism. I have been tracking monitoring and alerting systems for a while now, and this post mainly organizes some of the concepts, components, and research options related to monitoring and alerting.

1. Install node exporter 1 2 3 wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/ If you cannot reach GitHub, refer to: Several ways to access GitHub from mainland China . 2. Configure and start the exporter Add a Systemd configuration 1 vim /etc/systemd/system/node_exporter.service Add the following content: 1 2 3 4 5 6 7 8 9 10 11 12 [Unit] Description=Node Exporter After=network.

nload is a terminal tool for traffic monitoring. It reads the /proc/net/dev file and draws traffic graphs. Below is a brief introduction to using it. 1. Installation CentOS 1 yum install -y nload If your system’s repositories cannot find the nload package, run the following command to add a new

The Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge. 1. Terminology Grafana A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.