Hosts Host memory usage exceeds threshold 1 - node_memory_MemAvailable_bytes{mode!="idle"} / node_memory_MemTotal_bytes Threshold: 0.9 Host CPU usage exceeds threshold 1 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (host_name) Threshold: 0.85 Host disk usage exceeds threshold 1 - avg without (fstype)(node_filesystem_free_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'} / node_filesystem_size_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'}) Threshold: 0.8 Windows Windows host memory usage exceeds threshold 1 - 1 * windows_os_physical_memory_free_bytes{job="windows_exporter",mode!

The hard part of a monitoring system is storing large volumes of time-series data and providing high-performance query capability; the hard part of an alerting system is designing an efficient alert engine and implementing a flexible alert escalation mechanism. I have been tracking monitoring and alerting systems for a while now, and this post mainly organizes some of the concepts, components, and research options related to monitoring and alerting.