AI-Related Exporters in a Kubernetes Cluster

1. dcgm-exporter

dcgm-exporter is the GPU monitoring tool provided by the official NVIDIA community.

Project address https://github.com/NVIDIA/dcgm-exporter

1.1 Installation

Add the Helm chart repository

1
helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts

1
helm repo update

Install

1
2
3
4
5
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter --namespace monitor --create-namespace \
      --set serviceMonitor.enabled=false \
      --set image.tag=3.3.3-3.3.0-ubuntu22.04 \
      --set nodeSelector."accelerator\/provider"=nvidia-gpu \
      --version 3.3.1

You need to label the NVIDIA GPU nodes

1
kubectl label node <node-name> accelerator/provider=nvidia-gpu

1.2 Metrics

GPU utilization

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_GPU_UTIL	Gauge	%	GPU utilization
DCGM_FI_DEV_MEM_COPY_UTIL	Gauge	%	GPU memory copy utilization
DCGM_FI_DEV_ENC_UTIL	Gauge	%	GPU encoder utilization
DCGM_FI_DEV_DEC_UTIL	Gauge	%	GPU decoder utilization

Memory

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_FB_FREE	Gauge	MiB	Remaining GPU frame buffer
DCGM_FI_DEV_FB_USED	Gauge	MiB	GPU frame buffer usage

Below is the Markdown table of the metric information you provided:

Frequency

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_SM_CLOCK	Gauge	MHz	GPU SM clock frequency
DCGM_FI_DEV_MEM_CLOCK	Gauge	MHz	GPU memory clock frequency

Profiling

Metric name	Metric type	Unit	Description
DCGM_FI_PROF_GR_ENGINE_ACTIVE	Gauge	%	Fraction of time within an interval that the Graphics or Compute engine is Active
DCGM_FI_PROF_SM_ACTIVE	Gauge	%	Fraction of time within an interval that at least one warp is Active on an SM (average)
DCGM_FI_PROF_SM_OCCUPANCY	Gauge	%	Ratio of warps resident on an SM to the maximum warps that SM can hold (average)
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE	Gauge	%	Fraction of cycles in unit time that Tensor Pipes are Active on average
DCGM_FI_PROF_DRAM_ACTIVE	Gauge	%	Fraction of cycles that memory copies are active (a cycle is 100% if it has one DRAM instruction)
DCGM_FI_PROF_PIPE_FP64_ACTIVE	Gauge	%	Fraction of cycles in unit time that F64 Pipes are Active on average
DCGM_FI_PROF_PIPE_FP32_ACTIVE	Gauge	%	Fraction of cycles in unit time that F32 Pipes are Active on average
DCGM_FI_PROF_PIPE_FP16_ACTIVE	Gauge	%	Fraction of cycles in unit time that F16 Pipes are Active on average
DCGM_FI_PROF_NVLINK_RX_BYTES	Counter	B/s	Data traffic received over NVLink
DCGM_FI_PROF_NVLINK_TX_BYTES	Counter	B/s	Data traffic transmitted over NVLink
DCGM_FI_PROF_PCIE_RX_BYTES	Counter	B/s	Bytes received over the PCIe bus
DCGM_FI_PROF_PCIE_TX_BYTES	Counter	B/s	Bytes transmitted over the PCIe bus
DCGM_FI_DEV_PCIE_REPLAY_COUNTER	Counter	times	Retry count on the GPU PCIe bus
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL	Counter	-	Total NVLink bandwidth counter for all GPU lanes

Temperature and power

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_GPU_TEMP	Gauge	℃	Current GPU temperature
DCGM_FI_DEV_MEMORY_TEMP	Gauge	℃	Current GPU memory temperature
DCGM_FI_DEV_POWER_USAGE	Gauge	W	Current GPU power usage
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION	Counter	mJ	Total energy since GPU boot

XID errors and violations

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_XID_ERRORS	Gauge	-	Most recent error code
DCGM_CUSTOM_XID_ERRORS_TOTAL_COUNTER	Counter	-	Total number of error codes
DCGM_FI_DEV_POWER_VIOLATION	Counter	μs	Cumulative duration of violations caused by the power limit
DCGM_FI_DEV_THERMAL_VIOLATION	Counter	μs	Cumulative duration of violations caused by thermal limits
DCGM_FI_DEV_SYNC_BOOST_VIOLATION	Counter	μs	Cumulative duration of violations caused by the sync boost limit
DCGM_FI_DEV_BOARD_LIMIT_VIOLATION	Counter	μs	Cumulative duration of violations caused by the board limit
DCGM_FI_DEV_LOW_UTIL_VIOLATION	Counter	μs	Cumulative duration of violations caused by the low utilization limit
DCGM_FI_DEV_RELIABILITY_VIOLATION	Counter	μs	Cumulative duration of violations caused by the board reliability limit

Retired memory pages

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_RETIRED_SBE	Counter	count	Memory pages retired due to single-bit errors
DCGM_FI_DEV_RETIRED_DBE	Counter	count	Memory pages retired due to double-bit errors

Other

Metric name	Metric type	Unit	Description
DCGM_FI_DEV_VGPU_LICENSE_STATUS	Gauge	-	vGPU license status
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS	Counter	-	Rows remapped due to uncorrectable errors
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS	Counter	-	Rows remapped due to correctable errors
DCGM_FI_DEV_ROW_REMAP_FAILURE	Gauge

2. npu-exporter

NPU-Exporter is a service component developed by Huawei specifically to collect the various monitoring information and metrics of Huawei NPUs and package them into a Prometheus-specific data format. Project address https://github.com/Ascend/ascend-npu-exporter

2.1 Installation

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/observation/v5.0.RC3-npu-exporter.yaml

You need to label the Huawei NPU nodes

1
kubectl label node <node-name> accelerator/provider=huawei-npu

The Container Runtime here is Docker; if it is Containerd you need to change it in the startup parameters.

2.2 Metrics

Processor information

Metric name	Unit	Description
machine_npu_nums	count	Number of Ascend AI processors.

Network information

Metric name	Value	Description
npu_chip_info_bandwidth_rx	MB/s	Real-time receive rate of the Ascend AI processor network port (only supported on Atlas training series products). The labels contain the following fields:
npu_chip_info_bandwidth_tx	MB/s	Real-time send rate of the Ascend AI processor network port (only supported on Atlas training series products). The labels contain the following fields:
npu_chip_info_link_status	1: UP, 0: DOWN	Link status of the Ascend AI processor network port (only supported on Atlas training series products). The labels contain the following fields:
npu_chip_info_network_status	1: healthy, 0: unhealthy	Network health status of the Ascend AI processor (only supported on Atlas training series products). The labels contain the following fields:

Error and health information

Metric name	Value	Description
npu_chip_info_error_code	See the notes	Ascend AI processor error code. The labels contain the following fields: A -1 shown in the last column indicates a DCMI interface call error, possibly caused by a driver fault; 0 means no error code. For details of the error codes, see the “Atlas A2 Center Inference and Training Hardware Black Box Error Code Information List”
npu_chip_info_name	-	Ascend AI processor name and ID. The labels contain the following fields:
npu_chip_info_health_status	1: healthy, 0: unhealthy	Ascend AI processor health status. The labels contain the following fields:

Performance information

Metric name	Unit	Description
npu_chip_info_power	Watts (W)	Ascend AI processor power consumption (for 910 and 310 it is processor power; for 310P it is board power). The labels contain the following fields:
npu_chip_info_temperature	Celsius (℃)	Ascend AI processor temperature. The labels contain the following fields:
npu_chip_info_utilization	%	Ascend AI processor AI Core utilization. The labels contain the following fields:
npu_chip_info_aicore_current_freq	MHz	Current AI Core frequency of the Ascend AI processor. The labels contain the following fields:

Memory information

Metric name	Unit	Description
npu_chip_info_used_memory	MB	Used DDR memory of the Ascend AI processor. The labels contain the following fields:
npu_chip_info_total_memory	MB	Total DDR memory of the Ascend AI processor. The labels contain the following fields:
npu_chip_info_hbm_used_memory	MB	Used HBM memory of the Ascend AI processor (exclusive to Atlas training series products). The labels contain the following fields:
npu_chip_info_hbm_total_memory	MB	Total HBM memory of the Ascend AI processor (exclusive to Atlas training series products). The labels contain the following fields:
container_npu_total_memory	MB	Total NPU memory with container information, whole cards only. The container information contains the following fields:
container_npu_used_memory	MB	Used NPU memory with container information, whole cards only. The container information contains the following fields:
vnpu_pod_total_memory	KB	Total memory owned by the vNPU: (supported only on Atlas inference series products)
vnpu_pod_used_memory	KB	Memory in use by the vNPU: (supported only on Atlas inference series products)

Container and virtual NPU information

Metric name	Unit	Description
npu_exporter_version_info	-	NPU-Exporter version information.
npu_container_info	-	NPU container information; the output contains the following fields:
container_npu_total_memory	MB	Total NPU memory with container information, whole cards only. The container information contains the following fields:
container_npu_used_memory	MB	Used NPU memory with container information, whole cards only. The container information contains the following fields:
container_npu_utilization	%	NPU utilization with container information, whole cards only. The container information contains the following fields:
vnpu_pod_aicore_utilization	%	AI Core utilization of the vNPU: (supported only on Atlas inference series products)
vnpu_pod_total_memory	KB	Total memory owned by the vNPU: (supported only on Atlas inference series products)
vnpu_pod_used_memory	KB	Memory in use by the vNPU: (supported only on Atlas inference series products)

3. node-exporter

node-exporter is a tool open-sourced by Prometheus for collecting various host metrics.

Project address https://github.com/prometheus/node_exporter

3.1 Installation

When Prometheus Server is installed, node-exporter is usually already installed by default, so there is no need to install it again.

Add the Helm chart repository

1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

1
helm repo update

Install

1
helm install prometheus-node-exporter prometheus-community/prometheus-node-exporter --namespace monitor --create-namespace

3.2 Metrics

Below is the Markdown table after removing the category column:

Metric name	Type	Description
node_cpu_seconds_total	Counter	Node CPU usage time (unit: seconds)

Memory

Metric name	Type	Description
node_memory_MemTotal_bytes	Gauge	Total node memory size (unit: bytes)
node_memory_MemFree_bytes	Gauge	Free node memory size (unit: bytes)
node_memory_Buffers_bytes	Gauge	Node buffer size (unit: bytes)
node_memory_Cached_bytes	Gauge	Node page cache size (unit: bytes)

Disk

Metric name	Type	Description
node_filesystem_avail_bytes	Gauge	Free space available to users on the partition (unit: bytes)
node_filesystem_size_bytes	Gauge	Total capacity of the partition space (unit: bytes)
node_filesystem_free_bytes	Gauge	Free physical space on the partition (unit: bytes)
node_disk_read_bytes_total	Counter	Total bytes read from the partition (unit: bytes)
node_disk_written_bytes_total	Counter	Total bytes written to the partition (unit: bytes)
node_disk_reads_completed_total	Counter	Total number of reads on the partition
node_disk_writes_completed_total	Counter	Total number of writes on the partition

Network

Metric name	Type	Description
node_network_receive_bytes_total	Counter	Total bytes of received traffic (unit: bytes)
node_network_transmit_bytes_total	Counter	Total bytes of transmitted traffic (unit: bytes)
node_network_receive_packets_total	Counter	Total packets of received traffic (unit: packets)
node_network_transmit_packets_total	Counter	Total packets of transmitted traffic (unit: packets)
node_network_receive_drop_total	Counter	Total dropped packets of received traffic (unit: packets)
node_network_transmit_drop_total	Counter	Total dropped packets of transmitted traffic (unit: packets)

4 node-problem-detector

Node Problem Detector, NPD for short, is a Kubernetes cluster node monitoring plugin used for node fault detection.

Project address https://github.com/kubernetes/node-problem-detector

4.1 Installation

Add the Helm chart repository

1
helm repo add deliveryhero https://charts.deliveryhero.io/

1
helm repo update

Install

1
helm install node-problem-detector deliveryhero/node-problem-detector --namespace monitor --create-namespace

4.2 Metrics

Disk

Metric name	Type	Description
node_filesystem_avail_bytes	Gauge	Free space available to users on the partition (unit: bytes)
node_filesystem_size_bytes	Gauge	Total capacity of the partition space (unit: bytes)
node_filesystem_free_bytes	Gauge	Free physical space on the partition (unit: bytes)
node_disk_read_bytes_total	Counter	Total bytes read from the partition (unit: bytes)
node_disk_written_bytes_total	Counter	Total bytes written to the partition (unit: bytes)
node_disk_reads_completed_total	Counter	Total number of reads on the partition
node_disk_writes_completed_total	Counter	Total number of writes on the partition

Network

Metric name	Type	Description
node_network_receive_bytes_total	Counter	Total bytes of received traffic (unit: bytes)
node_network_transmit_bytes_total	Counter	Total bytes of transmitted traffic (unit: bytes)
node_network_receive_packets_total	Counter	Total packets of received traffic (unit: packets)
node_network_transmit_packets_total	Counter	Total packets of transmitted traffic (unit: packets)
node_network_receive_drop_total	Counter	Total dropped packets of received traffic (unit: packets)
node_network_transmit_drop_total	Counter	Total dropped packets of transmitted traffic (unit: packets)

Metric name	Type	Description
cpu_load_1m	Gauge	CPU load average (1 minute)
cpu_load_5m	Gauge	CPU load average (5 minutes)
cpu_load_15m	Gauge	CPU load average (15 minutes)
cpu_runnable_task_count	Gauge	Average number of runnable tasks (over the past minute)
cpu_usage_time	Counter	CPU usage time (by state)

Memory

Metric name	Type	Description
memory_bytes_used	Gauge	Memory usage (by state)
memory_anonymous_used	Gauge	Anonymous memory usage (by state)
memory_dirty_used	Gauge	Dirty page memory usage
memory_page_cache_used	Gauge	Page cache memory usage (by state)
memory_unevictable_used	Gauge	Unevictable memory usage

System

Metric name	Type	Description
host_uptime	Gauge	Operating system uptime
system_cpu_stat	Counter	CPU time in different states
system_interrupts_total	Counter	Total interrupt service count (cumulative)
system_os_feature	Gauge	Operating system feature enabled state
system_processes_total	Counter	Total number of processes since boot (cumulative)
system_procs_blocked	Gauge	Number of currently blocked processes
system_procs_running	Gauge	Number of currently running processes

Problem count

Metric name	Type	Description
problem_counter	Counter	Number of occurrences of a specific problem type

Problem state

Metric name	Type	Description
problem_gauge	Gauge	State of a specific problem type

5. process-exporter

process-exporter is mainly used to monitor the state of processes on a host, but unfortunately it has not been updated for a long time.

Project address https://github.com/mumoshu/prometheus-process-exporter , dashboard https://grafana.com/grafana/dashboards/8378-system-processes-metrics/

5.1 Installation

Install

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/observation/process-exporter.yaml

How the metrics are configured

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: v1
data:
  process-exporter.yaml: |-
    process_names:
    - name: "{{.Matches}}"
      cmdline:
      - 'python'
    - name: "{{.Matches}}"
      cmdline:
      - 'triton'    
kind: ConfigMap
metadata:
  name: process-exporter-config
  namespace: monitor

Match template	Description
{{.Comm}}	Contains the name of the original executable, i.e. /proc//stat
{{.ExeBase}}	Contains the name of the executable (default)
{{.ExeFull}}	Contains the path of the executable
{{.Username}}	Contains the username
{{.Matches}}	Contains all matches produced by the regular expressions (recommended)
{{.PID}}	Contains the PID of the process; one PID contains only one process (not recommended)
{{.StartTime}}	Contains the start time of the process (not recommended)

5.2 Metrics

Metric name	Description
namedprocess_namegroup_num_procs	Number of running processes
namedprocess_namegroup_states	Number of processes in Running/Sleeping/Other/Zombie state
namedprocess_namegroup_cpu_seconds_total	Reads the CPU utime and stime state time of the process from /proc/[pid]/stat
namedprocess_namegroup_read_bytes_total	Reads the bytes read by the process from /proc/[pid]/io
namedprocess_namegroup_write_bytes_total	Reads the bytes written by the process from /proc/[pid]/io
namedprocess_namegroup_memory_bytes	Reads the memory bytes used by the process
namedprocess_namegroup_open_filedesc	Reads the number of file descriptors used by the process
namedprocess_namegroup_thread_count	Number of running threads
namedprocess_namegroup_thread_cpu_seconds_total	Reads the CPU state time of the thread
namedprocess_namegroup_thread_io_bytes_total	Reads the IO bytes of the thread

6. nvidia gpu exporter

nvidia gpu exporter is a tool that obtains GPU metrics through nvidia-smi

Project address https://github.com/utkuozdemir/nvidia_gpu_exporter , dashboard address https://grafana.com/grafana/dashboards/14574-nvidia-gpu-metrics/

6.1 Installation

Add the Helm chart repository

1
helm repo add utkuozdemir https://utkuozdemir.org/helm-charts

1
helm repo update

Install

1
2
3
4
helm install nvidia-gpu-exporter utkuozdemir/nvidia-gpu-exporter \
      --namespace monitor --create-namespace \
      --set nodeSelector."accelerator\/provider"=nvidia-gpu \
      --set serviceMonitor.enabled=false

You need to label the NVIDIA GPU nodes

1
kubectl label node <node-name> accelerator/provider=nvidia-gpu

6.2 Metrics

Metric name	Type	Unit	Description
nvidia_smi_accounting_buffer_size	gauge		Accounting buffer size
nvidia_smi_accounting_mode	gauge		Accounting mode
nvidia_smi_clocks_applications_graphics_clock_hz	gauge	MHz	Application graphics clock frequency
nvidia_smi_clocks_applications_memory_clock_hz	gauge	MHz	Application memory clock frequency
nvidia_smi_clocks_current_graphics_clock_hz	gauge	MHz	Current graphics clock frequency
nvidia_smi_clocks_current_memory_clock_hz	gauge	MHz	Current memory clock frequency
nvidia_smi_clocks_current_sm_clock_hz	gauge	MHz	Current streaming multiprocessor clock frequency
nvidia_smi_clocks_current_video_clock_hz	gauge	MHz	Current video clock frequency
nvidia_smi_clocks_default_applications_graphics_clock_hz	gauge	MHz	Default application graphics clock frequency
nvidia_smi_clocks_default_applications_memory_clock_hz	gauge	MHz	Default application memory clock frequency
nvidia_smi_clocks_max_graphics_clock_hz	gauge	MHz	Maximum graphics clock frequency
nvidia_smi_clocks_max_memory_clock_hz	gauge	MHz	Maximum memory clock frequency
nvidia_smi_clocks_max_sm_clock_hz	gauge	MHz	Maximum streaming multiprocessor clock frequency
nvidia_smi_clocks_throttle_reasons_active	gauge		Active throttle reason
nvidia_smi_clocks_throttle_reasons_applications_clocks_setting	gauge		Application clocks setting throttle reason
nvidia_smi_clocks_throttle_reasons_gpu_idle	gauge		GPU idle throttle reason
nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown	gauge		Hardware power brake slowdown throttle reason
nvidia_smi_clocks_throttle_reasons_hw_slowdown	gauge		Hardware slowdown throttle reason
nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown	gauge		Hardware thermal slowdown throttle reason
nvidia_smi_clocks_throttle_reasons_supported	gauge		Supported throttle reasons
nvidia_smi_clocks_throttle_reasons_sw_power_cap	gauge		Software power cap throttle reason
nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown	gauge		Software thermal slowdown throttle reason
nvidia_smi_clocks_throttle_reasons_sync_boost	gauge		Sync boost throttle reason
nvidia_smi_command_exit_code	gauge		Exit code of the last scrape command
nvidia_smi_compute_cap	gauge		Compute capability
nvidia_smi_compute_mode	gauge		Compute mode
nvidia_smi_count	gauge		Count
nvidia_smi_display_active	gauge		Display active state
nvidia_smi_display_mode	gauge		Display mode
nvidia_smi_ecc_errors_corrected_aggregate_device_memory	gauge		Total corrected device memory ECC errors
nvidia_smi_ecc_errors_corrected_aggregate_dram	gauge		Total corrected DRAM ECC errors
nvidia_smi_ecc_errors_corrected_aggregate_total	gauge		Total corrected ECC errors
nvidia_smi_ecc_errors_corrected_volatile_device_memory	gauge		Corrected volatile device memory ECC errors
nvidia_smi_ecc_errors_corrected_volatile_dram	gauge		Corrected volatile DRAM ECC errors
nvidia_smi_ecc_errors_corrected_volatile_total	gauge		Total corrected volatile ECC errors
nvidia_smi_ecc_errors_uncorrected_aggregate_device_memory	gauge		Total uncorrected device memory ECC errors
nvidia_smi_ecc_errors_uncorrected_aggregate_dram	gauge		Total uncorrected DRAM ECC errors
nvidia_smi_ecc_errors_uncorrected_aggregate_total	gauge		Total uncorrected ECC errors
nvidia_smi_ecc_errors_uncorrected_volatile_device_memory	gauge		Uncorrected volatile device memory ECC errors
nvidia_smi_ecc_errors_uncorrected_volatile_dram	gauge		Uncorrected volatile DRAM ECC errors
nvidia_smi_ecc_errors_uncorrected_volatile_total	gauge		Total uncorrected volatile ECC errors
nvidia_smi_ecc_mode_current	gauge		Current ECC mode
nvidia_smi_ecc_mode_pending	gauge		Pending ECC mode
nvidia_smi_encoder_stats_average_fps	gauge		Average encoder FPS
nvidia_smi_encoder_stats_average_latency	gauge		Average encoder latency
nvidia_smi_encoder_stats_session_count	gauge		Number of encoder sessions
nvidia_smi_enforced_power_limit_watts	gauge	W	Enforced power limit
nvidia_smi_gpu_info	gauge		GPU information
nvidia_smi_index	gauge		Index
nvidia_smi_inforom_ecc	gauge		ECC information ROM
nvidia_smi_inforom_oem	gauge		OEM information ROM
nvidia_smi_memory_free_bytes	gauge	MiB	Free memory
nvidia_smi_memory_reserved_bytes	gauge	MiB	Reserved memory
nvidia_smi_memory_total_bytes	gauge	MiB	Total memory
nvidia_smi_memory_used_bytes	gauge	MiB	Used memory
nvidia_smi_name	gauge		Name
nvidia_smi_pci_bus	gauge		PCI bus number
nvidia_smi_pci_device	gauge		PCI device number
nvidia_smi_pci_device_id	gauge		PCI device ID
nvidia_smi_pci_domain	gauge		PCI domain
nvidia_smi_pci_sub_device_id	gauge		PCI sub-device ID
nvidia_smi_pcie_link_gen_current	gauge		Current PCIe link generation
nvidia_smi_pcie_link_gen_gpucurrent	gauge		Current GPU PCIe link generation
nvidia_smi_pcie_link_gen_gpumax	gauge		Maximum GPU PCIe link generation
nvidia_smi_pcie_link_gen_hostmax	gauge		Maximum host PCIe link generation
nvidia_smi_pcie_link_gen_max	gauge		Maximum PCIe link generation
nvidia_smi_pcie_link_width_current	gauge		Current PCIe link width
nvidia_smi_pcie_link_width_max	gauge

7. juicefs-exporter

When a JuiceFS file system is mounted on a host, Metrics data is already available on port 9567 by default.

Dashboard address https://github.com/juicedata/juicefs/blob/be5b6935975ea665c37f9cf5f5827e6f9474e28f/docs/en/grafana_template.json

7.1 Installation

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/observation/host-9567-expoorter.yaml

7.2 Metrics

File system

Metric name	Description	Unit
`juicefs_used_space`	Total space used	bytes
`juicefs_used_inodes`	Total number of inodes

Operating system

Metric name	Description	Unit
`juicefs_uptime`	Total uptime	seconds
`juicefs_cpu_usage`	CPU usage	seconds
`juicefs_memory`	Memory usage	bytes

Metadata engine

Metric name	Description	Unit
`juicefs_transaction_durations_histogram_seconds`	Latency distribution of transactions	seconds
`juicefs_transaction_restart`	Number of transaction restarts

FUSE

Metric name	Description	Unit
`juicefs_fuse_read_size_bytes`	Size distribution of read requests	bytes
`juicefs_fuse_written_size_bytes`	Size distribution of write requests	bytes
`juicefs_fuse_ops_durations_histogram_seconds`	Latency distribution of all requests	seconds
`juicefs_fuse_open_handlers`	Number of open files and directories

Metric name	Description	Unit
`juicefs_sdk_read_size_bytes`	Size distribution of read requests	bytes
`juicefs_sdk_written_size_bytes`	Size distribution of write requests	bytes
`juicefs_sdk_ops_durations_histogram_seconds`	Latency distribution of all requests	seconds

Cache

Metric name	Description	Unit
`juicefs_blockcache_blocks`	Total number of cached blocks
`juicefs_blockcache_bytes`	Total size of cached blocks	bytes
`juicefs_blockcache_hits`	Total number of cached block hits
`juicefs_blockcache_miss`	Total number of cached block misses
`juicefs_blockcache_writes`	Total number of cache block writes
`juicefs_blockcache_drops`	Total number of cache block drops
`juicefs_blockcache_evicts`	Total number of cache block evictions
`juicefs_blockcache_hit_bytes`	Total size of hit cached blocks	bytes
`juicefs_blockcache_miss_bytes`	Total size of missed cached blocks	bytes
`juicefs_blockcache_write_bytes`	Total size of cache block writes	bytes
`juicefs_blockcache_read_hist_seconds`	Latency distribution of cache block reads	seconds
`juicefs_blockcache_write_hist_seconds`	Latency distribution of cache block writes	seconds
`juicefs_staging_blocks`	Number of blocks in the staging path
`juicefs_staging_block_bytes`	Total bytes of blocks in the staging path	seconds
`juicefs_staging_block_delay_seconds`	Total seconds of staging block delay	seconds

Object storage

Metric name	Description	Unit
`juicefs_object_request_durations_histogram_seconds`	Latency distribution of object storage requests	seconds
`juicefs_object_request_errors`	Total number of failed requests
`juicefs_object_request_data_bytes`	Total data size of object storage requests	bytes

Internal features

Metric name	Description	Unit
`juicefs_compact_size_histogram_bytes`	Size distribution of compacted data	bytes
`juicefs_used_read_buffer_size_bytes`	Size of the buffer currently used for reads

Data synchronization

Metric name	Description	Unit
`juicefs_sync_scanned`	Number of all objects scanned from the source
`juicefs_sync_handled`	Number of objects from the source already handled
`juicefs_sync_pending`	Number of objects waiting to be synchronized
`juicefs_sync_copied`	Number of objects already synchronized
`juicefs_sync_copied_bytes`	Total size of data already synchronized	bytes
`juicefs_sync_skipped`	Number of objects skipped during synchronization
`juicefs_sync_failed`	Number of objects that failed during synchronization
`juicefs_sync_deleted`	Number of objects deleted during synchronization
`juicefs_sync_checked`	Number of objects whose checksum was verified during synchronization
`juicefs_sync_checked_bytes`	Total size of data whose checksum was verified during synchronization	bytes

8. kube-state-metrics

kube-state-metrics generates Metrics data for the state of various resources by watching the Kubernetes API server.

Project address https://github.com/kubernetes/kube-state-metrics , dashboard https://grafana.com/grafana/dashboards/13332-kube-state-metrics-v2/ .

8.1 Installation

When Prometheus Server is installed, kube-state-metrics is usually already installed by default, so there is no need to install it again.

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/observation/v2.10.0-kube-state-metrics.yaml

8.2 Metrics

Node

Metric name	Type	Description
kube_node_info	Gauge	Queries information about all nodes in the cluster; the sum() function gives the total number of nodes in the cluster.
kube_node_spec_unschedulable	Gauge	Queries whether a node can be scheduled new Pods. The sum() function gives the total number of Pods that can be scheduled in the cluster.
kube_node_status_allocatable	Gauge	Queries the total amount of resources available for scheduling on the node, including CPU, memory, Pods, and so on. Allows filtering by label to view a node’s specific resource capacity.
kube_node_status_capacity	Gauge	Queries the total amount of all resources on the node, including CPU, memory, Pods, and so on. Allows filtering by label to view a node’s specific resource capacity.
kube_node_status_condition	Gauge	Queries the node’s state; based on states such as OutOfDisk, MemoryPressure, and DiskPressure you can find nodes in an abnormal state.

Metric name	Type	Description
kube_pod_info	Gauge	Queries information about all Pods; the sum() function gives the total number of Pods in the cluster.
kube_pod_status_phase	Gauge	Queries the startup state of all Pods. The states include: _ True: started successfully. _ Failed: failed to start. * Unknown: state unknown.
kube_pod_status_ready	Gauge	Queries all Pods in the Ready state. The sum() function gives the total number of Pods in the cluster.
kube_pod_status_scheduled	Gauge	Queries all Pods in the scheduled state. The sum() function gives the total number of Pods in the cluster.

Container

Metric name	Type	Description
kube_pod_container_info	Gauge	Queries information about all Containers. The sum() function gives the total number of Containers in the cluster.
kube_pod_container_status_ready	Gauge	Queries information about all Containers in the Ready state. The sum() function gives the total number of Containers in the cluster.
kube_pod_container_status_restarts_total	Count	Queries the cumulative number of Container restarts in the cluster. The irate() function gives the Container restart rate in the cluster.
kube_pod_container_status_running	Gauge	Queries information about all Containers in the Running state. The sum() function gives the total number of Containers in the cluster.
kube_pod_container_status_terminated	Gauge	Queries information about all Containers in the Terminated state. The sum() function gives the total number of Containers in the cluster.
kube_pod_container_status_waiting	Gauge	Queries information about all Containers in the Waiting state. The sum() function gives the total number of Containers in the cluster.
kube_pod_container_resource_requests	Gauge	Queries the container’s resource requests. Allows filtering by label to view a container’s specific resource requests.
kube_pod_container_resource_limits	Gauge	Queries the container’s resource limits. Allows filtering by label to view a container’s specific resource limits.