1. Interface Path Differences Between cgroup v1 and v2 v1 1 2 3 4 /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpuacct.usage /sys/fs/cgroup/memory/memory.limit_in_bytes /sys/fs/cgroup/memory/memory.usage_in_bytes v2 1 2 3 4 /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current cgroup v2 is the upgraded version of v1, with advantages such as more unified resource hierarchy management and precise resource isolation. But it also means that when writing code, the paths for reading the relevant interface files are different, so compatibility handling is required.

1. What DCGM Is DCGM (Data Center GPU Manager) is a toolset NVIDIA provides for managing and monitoring GPUs in the data center. It offers the following capabilities: GPU behavior monitoring GPU configuration management GPU policy supervision GPU health and diagnostics GPU accounting and process statistics NVSwitch configuration and monitoring

1. Installing Dependencies MindCluster ToolBox is a system-level toolset for cluster operations and hardware management, mainly used for operations such as device monitoring, performance testing, and log collection. MindCluster ToolBox includes the Ascend DMI tool, the log collection tool, and the Ascend Cert tool. MindCluster ToolBox works by calling low-level DCMI (Device Control Management Interface) and AscendCL (Ascend Computing Language) interfaces to perform the relevant detection functions; system-level information queries are implemented by calling the general-purpose libraries provided by the system.

1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.

Content referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.