The Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge.

1. Terminology

• Grafana

A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.

• Prometheus

A time series database, mainly used to collect, store, and serve query data.

• Exporter

A program that exposes service monitoring metrics, providing an API endpoint for Prometheus to scrape monitoring data from.

• PromQL

Prometheus’s built-in data query language, which provides rich support for querying, aggregating, and performing logical operations on time series data.

2. Installing Prometheus

• Add the Helm repository

1
2

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

• Install Prometheus

1

helm install prometheus -n monitor prometheus-community/prometheus --create-namespace --version 19.0.0

If prometheus-node-exporter never comes up, the default port 9100 may be occupied. You can use the following command to edit the DaemonSet and change the default port:

1

kubectl -n monitor edit ds prometheus-node-exporter

• Uninstall

1

helm uninstall prometheus -n monitor

3. Installing Grafana

• Add the Helm repository

1
2

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

1

helm -n monitor install grafana grafana/grafana

Note that with this installation method, Grafana’s data is stored under the /var/lib/grafana path inside the Pod, so restarting Grafana loses the related configuration. In a production environment you need to mount a storage volume. If a StorageClass is available, you can use the following parameter for persistent storage:

1

helm -n monitor install grafana grafana/grafana --set persistence.enabled="true" --version 6.29.9

• Get the login password for the admin account

1
2
3

kubectl -n monitor get secret grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

JIsmMsWaN8rF5ryS7rVohHyFWKzyahR7u0OJsiJL

• Change the Grafana service’s access type to NodePort

1

kubectl -n monitor patch svc grafana -p '{"spec": {"type": "NodePort"}}'

1
2
3
4

kubectl -n monitor get svc grafana

NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
grafana   NodePort   10.233.6.118   <none>        80:31892/TCP   117s

This way, you can access Grafana via the host IP + 31892.

• Uninstall

1

helm -n monitor uninstall grafana

4. Configuration and Usage

4.1 Adding a Data Source

In the left navigation bar, find [Data Sources]

Just enter Prometheus’s access address http://prometheus-server.monitor.svc.

4.2 Adding a Template

In the left navigation bar, find the Import button under [+]. I chose to import the dashboard with id 10856. You can also go to the Grafana website and pick a suitable dashboard to import, https://grafana.com/grafana/dashboards.

As shown below, here you need to select the data source you added above.

4.3 Viewing the Dashboard

4.4 grafana.ini Configuration

Some Grafana settings can only be made by editing grafana.ini; below I list only the few I used:

1. Edit the configuration file

1

kubectl -n monitor edit cm grafana

2. Add configuration

• Allow anonymous access

1
2
3

  grafana.ini: |
    [auth.anonymous]
    enabled = true

• Allow Iframe embedding

1
2
3

  grafana.ini: |
    [security]
    allow_embedding = true

3. Restart Grafana for the changes to take effect

1

kubectl -n monitor rollout restart deployment grafana

5. References

• https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack