https://www.chenshaowen.com/en/tags/monitoring/Recent content in Monitoring on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sat, 11 Jul 2026 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/container-deploy-victoriametrics.htmlSat, 11 Jul 2026 00:00:00 +0000Sat, 11 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-victoriametrics.htmlVictoriaMetrics (VM for short) is a time-series database compatible with the Prometheus ecosystem. It uses fewer resources than Prometheus and comes in both single-node and cluster editions. The single-node edition can directly replace Prometheus storage; the cluster edition splits reads, writes, and storage across vmstorage / vminsert / vmselect, giving it stronger horizontal scaling.微信公众号MiddlewareContainersVictoriaMetricsPrometheusMonitoringOperationshttps://www.chenshaowen.com/en/blog/container-deploy-clickhouse.htmlWed, 11 Feb 2026 00:00:00 +0000Wed, 11 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-clickhouse.html1. ClickHouse Single Node 1.1 Configure Environment Variables 1 2 3 4 5 6 7 8 export CONTAINER_CLI=nerdctl export IMAGE=clickhouse/clickhouse-server:24 export CLICKHOUSE_INSTANCE_NAME=clickhouse export CH_DATA=/data/ops/clickhouse/$CLICKHOUSE_INSTANCE_NAME export CLICKHOUSE_PORT=9000 export CLICKHOUSE_PROMETHEUS_PORT=9363 export CLICKHOUSE_USER=default export CLICKHOUSE_PASSWORD=xxxxxx 1.2 Generate the Configuration Files 1 2 3 4 5 6 7 8 9 10 11 12 13微信公众号MiddlewareContainersClickHousePrometheusMonitoringOperationshttps://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.htmlWed, 13 Aug 2025 00:00:00 +0000Wed, 13 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.html1. What DCGM Is DCGM (Data Center GPU Manager) is a toolset NVIDIA provides for managing and monitoring GPUs in the data center. It offers the following capabilities: GPU behavior monitoring GPU configuration management GPU policy supervision GPU health and diagnostics GPU accounting and process statistics NVSwitch configuration and monitoring微信公众号AINVIDIADCGMGPUMonitoringOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.htmlFri, 26 Jul 2024 01:00:00 +0000Fri, 26 Jul 2024 01:00:00 +0000https://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.html1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.微信公众号MonitoringMemoryKubernetesGrafanaOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.htmlSat, 12 Aug 2023 00:00:00 +0000Sat, 12 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.html1. What Needs Dial Testing GPT was all the rage this year, and I deployed and developed several applications and mini programs to learn from. Of course, following the principle of helping vendors test their features, right now only the GPT 3.5 API costs a little every day — the微信公众号GitHubActionsMonitoringDial TestingOperationsLearningAvailabilityFreeCICDBest Practiceshttps://www.chenshaowen.com/en/blog/insight-kubernetes-network-by-kindling.htmlSat, 10 Sep 2022 00:00:00 +0000Sat, 10 Sep 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/insight-kubernetes-network-by-kindling.htmlI recently had a requirement to collect Kubernetes’ external network access. So I set up and tried out some related projects. This post is mainly about how to install Kindling and configure Grafana to view Kubernetes network connection data. 1. What Is Kindling Kindling solves the problem of how to observe the network without intruding on the application; its functionality is mainly achieved by exposing kernel events for observation.微信公众号KubernetesKindlingNetworkObservabilityOperationseBPFMonitoringhttps://www.chenshaowen.com/en/blog/some-common-promql-for-alarming.htmlSun, 21 Aug 2022 11:22:55 +0000Sun, 21 Aug 2022 11:22:55 +0000https://www.chenshaowen.com/en/blog/some-common-promql-for-alarming.htmlHosts Host memory usage exceeds threshold 1 - node_memory_MemAvailable_bytes{mode!="idle"} / node_memory_MemTotal_bytes Threshold: 0.9 Host CPU usage exceeds threshold 1 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (host_name) Threshold: 0.85 Host disk usage exceeds threshold 1 - avg without (fstype)(node_filesystem_free_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'} / node_filesystem_size_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'}) Threshold: 0.8 Windows Windows host memory usage exceeds threshold 1 - 1 * windows_os_physical_memory_free_bytes{job="windows_exporter",mode!微信公众号PrometheusAlertingMonitoringOperationsPromQLhttps://www.chenshaowen.com/en/blog/how-to-deploy-the-elasticsearch-stack-on-kubernetes.htmlWed, 06 Jul 2022 00:00:00 +0000Wed, 06 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-deploy-the-elasticsearch-stack-on-kubernetes.htmlIf you use Logstash to centrally receive the log input from Filebeat, it easily creates a single point of bottleneck; if you use Kafka to receive the log input from Filebeat, the timeliness of the logs is not guaranteed. Here we output the logs collected by Filebeat directly to Elasticsearch.微信公众号KubernetesElasticsearchLoggingOperationsELKMonitoringhttps://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlSat, 25 Jun 2022 00:00:00 +0000Sat, 25 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlIf you need to monitor the network status between two hosts, or between a host and an external service, then the Blackbox Exporter mentioned in this article is worth a try. 1. Installing Blackbox 1.1 Deploying on a Host Download the binary package 1 2 3 4 5 wget https://github.微信公众号PrometheusMonitoringProbingOperationsLearningNetworkBlackbox ExporterAvailabilityhttps://www.chenshaowen.com/en/blog/how-to-insight-the-pipeline-of-tekton.htmlTue, 07 Jun 2022 00:00:00 +0000Tue, 07 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-insight-the-pipeline-of-tekton.html1. Scraping Tekton Metrics Add a ConfigMap configuration file 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: config-observability namespace: tekton-pipelines labels: app.kubernetes.io/instance: default app.kubernetes.io/part-of: tekton-pipelines data: metrics.backend-destination: prometheus metrics.taskrun.level: "task" metrics.微信公众号KubernetesTektonMonitoringMetricsCICDOperationshttps://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.htmlThu, 02 Jun 2022 01:00:00 +0000Thu, 02 Jun 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.html1. Why kube-status-metrics is needed Kubernetes monitoring mainly focuses on two kinds of metrics: Basic performance metrics CPU, memory, disk, network, and so on. These can be collected by deploying node-exporter as a DaemonSet and having Prometheus scrape the relevant metrics. Resource object metrics The replica count of a Deployment, the running state of a Pod, and so on.微信公众号KubernetesMonitoringMetricsCollectionPrometheusOperationsLearningObservabilitykube-status-metricshttps://www.chenshaowen.com/en/blog/manage-multiple-prometheus-using-thanos.htmlTue, 22 Mar 2022 00:00:00 +0000Tue, 22 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/manage-multiple-prometheus-using-thanos.html1. Layering of Monitoring As shown above, when building a monitoring system, two strategies are used: Layered monitoring. The benefit of separating IaaS, MySQL middleware, and App layer monitoring is that the systems have high availability and fault tolerance between them. When App layer monitoring stops working, IaaS layer monitoring微信公众号ThanosPrometheusMonitoringStorageKubernetesOperationshttps://www.chenshaowen.com/en/blog/the-monitor-of-kubernetes-using-prometheus-grafana.htmlFri, 03 Sep 2021 00:00:00 +0000Fri, 03 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-monitor-of-kubernetes-using-prometheus-grafana.htmlThe Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge. 1. Terminology Grafana A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.微信公众号PrometheusGrafanaKubernetesMonitoringOperationsObservabilityhttps://www.chenshaowen.com/en/blog/how-to-dial-up-testing-using-jenkins.htmlWed, 04 Dec 2019 00:00:00 +0000Wed, 04 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-dial-up-testing-using-jenkins.html1. Why Dial Testing For some of the critical services in a system, we usually set up monitoring. When a failure occurs, it can be detected as quickly as possible and a notification sent to the people who care. Once a failure has occurred, the failure process can be traced微信公众号JenkinsDial TestingMonitoringDevOpsOperationshttps://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..htmlMon, 24 Sep 2018 00:00:00 +0000Mon, 24 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..html1. Kubectl Basic Commands 1.1 Creating Objects 1 2 3 4 5 6 7 8 # Create resources; a remote URL also works kubectl create -f ./my.yaml # Create resources from multiple files kubectl create -f ./my1.yaml -f ./my2.yaml # Create resources using all manifest files in a directory kubectl create -f .微信公众号KubernetesTestingMonitoringOperationsLearningkubectlStress TestingObservability