Prometheus

Internet services often depend on tens of thousands of hosts, and those hosts are deployed all over the world, on top of a tangled mess of user environments. To keep a service stable and reliable, you need to monitor it. The goals of monitoring can be considered from several angles: from the system dimension, understand CPU, memory, and disk usage; from the network monitoring dimension, understand upstream and downstream rates; from the service monitoring dimension, understand the health and availability of the application. Of course, once you have monitoring data, you can also configure alerts to notify maintenance staff and assist in troubleshooting.

1. Open-Source Monitoring Tools

1.1 Nagios

Pros
- Monitors all protocols (HTTP, FTP, SSH, POP3, SMTP, SNMP, MySQL…)
- Completely standalone, with no dependencies
- Supports flap detection
- Plugin-based
Cons
- Unfriendly web pages
- No chart display for data

1.2 Zabbix

Pros
- Monitors all protocols (HTTP, FTP, SSH, POP3, SMTP, SNMP, MySQL…)
- Friendly web interface
- Can monitor Log files
- Supports Client agent (Pull and Push model)
Cons
- Configuration is fairly complex
- Requires a relational database (mysql, PostgreSQL..)

1.3 Prometheus

Pros
- Written in Go, good performance
- Multi-language client support (sdk)
- Supports Pull and Push modes
- Supports multiple data visualization modes (Grafana)
Cons
- Lacks plugin support

2. Setting Up Prometheus + Grafana

Prometheus is an open-source monitoring system that originated from Google’s Borgmon and is developed in Golang. Many people call it the next-generation monitoring system.

The basic principle of Prometheus is to periodically scrape the state of monitored components over the HTTP protocol. As long as a component provides an HTTP interface, it can join the monitoring system, with no need for any SDK or other integration process. This makes it very well suited to virtualized environments such as VMs, Docker, Mesos, and Kubernetes.

The HTTP interface that exposes information about the monitored component is called an exporter.

Grafana is an open-source chart visualization system — put simply, chart configuration is fairly convenient and the charts it produces look good.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# docker-compose.yml
version: '2'
services:
    prometheus:
        image: prom/prometheus:latest
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
        command:
            - '-config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'
    node-exporter:
        image: prom/node-exporter:latest
        ports:
            - '9100:9100'
    grafana:
        image: grafana/grafana:latest
        environment:
            - GF_SECURITY_ADMIN_PASSWORD=password
        depends_on:
            - prometheus
        ports:
            - "3000:3000"

Prometheus is configured with YAML. The global section configures some global information, and scrape_configs configures the specific targets you want to scrape.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# prometheus.yml
global:
    scrape_interval: 5s
    external_labels:
        monitor: 'my-monitor'
scrape_configs:
    - job_name: 'prometheus'
      target_groups:
          - targets: ['localhost:9090']
    - job_name: 'node-exporter'
      target_groups:
          - targets: ['node-exporter:9100']

Taking CentOS as an example, install Docker and Docker compose

1
2
3
4
5
yum install docker
yum install epel-release
yum install python-pip
pip install --upgrade pip
pip install docker-compose

Start Docker

1
2
service docker start
docker-compose up