1. What Is npu-smi npu-smi is a command-line tool provided by Huawei, purpose-built for managing and monitoring the status and performance of Huawei Ascend series Neural Processing Units (NPUs), similar to NVIDIA’s nvidia-smi. 2. Meaning of npu-smi Fields 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 npu-smi info +------------------------------------------------------------------------------------------------+ | npu-smi 23.

I have been studying the infrastructure around model training lately, and I found that AI chip interconnect topology determines the scheduling and resource allocation of training cluster jobs. So I spent a little time putting together the common AI chip interconnect solutions. 1. Point-to-Point Interconnect In a traditional PCIe system, data transfer between AI chips goes over PCIe, which cannot meet the demands of large-scale data transfer.

Content referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.

1. What Is nvidia-smi nvidia-smi stands for NVIDIA System Management Interface, the interface NVIDIA provides for managing and monitoring GPUs. nvidia-smi calls NVML. NVML stands for NVIDIA Management Library, a library that provides a set of C APIs for NVIDIA GPU monitoring and management. 1.1 Queryable State ECC error counts GPU utilization Active compute processes Clocks and PState Temperature and fan speed Power management Hardware identification 1.

1. Disk Handling 1.1 Inspect Disks Inspect the new disk 1 fdisk -l Disk /dev/nvme1n1: 3.91 TiB, 4294967296000 bytes, 8388608000 sectors 1.2 Build a RAID0 If you have several small disks, a better approach is to build a RAID0 — this gives you not only a larger storage directory but also higher speed.