1. Aggregating Data with Query As shown above, the components that Thanos Query can connect to are: Thanos Store Gateway Thanos Query Thanos Receive Prometheus, by way of Sidecar By cascading Thanos Queries, we can perform correlated queries across components and build an extremely large monitoring system. This also means that every connected component should provide a sufficiently fast Prometheus API.

1. Layering of Monitoring As shown above, when building a monitoring system, two strategies are used: Layered monitoring. The benefit of separating IaaS, MySQL middleware, and App layer monitoring is that the systems have high availability and fault tolerance between them. When App layer monitoring stops working, IaaS layer monitoring will immediately make that visible.

The hard part of a monitoring system is storing large volumes of time-series data and providing high-performance query capability; the hard part of an alerting system is designing an efficient alert engine and implementing a flexible alert escalation mechanism. I have been tracking monitoring and alerting systems for a while now, and this post mainly organizes some of the concepts, components, and research options related to monitoring and alerting.

1. Install node exporter 1 2 3 wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/ If you cannot reach GitHub, refer to: Several ways to access GitHub from mainland China . 2. Configure and start the exporter Add a Systemd configuration 1 vim /etc/systemd/system/node_exporter.service Add the following content: 1 2 3 4 5 6 7 8 9 10 11 12 [Unit] Description=Node Exporter After=network.

The Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge. 1. Terminology Grafana A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.