1. Introduction to Volcano Volcano is a Kubernetes-based resource scheduling system open-sourced by Huawei. Compared with the native scheduler, its notable features are: Supports gang scheduling Scheduling batch jobs easily runs into deadlock: for example, two jobs each need 10 Pods running simultaneously to start. When both jobs are submitted at the same time, it is possible that only part of the Pods of each are scheduled, so neither job can run properly and they wait on each other.

1. What Is npu-smi npu-smi is a command-line tool provided by Huawei, purpose-built for managing and monitoring the status and performance of Huawei Ascend series Neural Processing Units (NPUs), similar to NVIDIA’s nvidia-smi. 2. Meaning of npu-smi Fields 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 npu-smi info +------------------------------------------------------------------------------------------------+ | npu-smi 23.

I have been studying the infrastructure around model training lately, and I found that AI chip interconnect topology determines the scheduling and resource allocation of training cluster jobs. So I spent a little time putting together the common AI chip interconnect solutions. 1. Point-to-Point Interconnect In a traditional PCIe system, data transfer between AI chips goes over PCIe, which cannot meet the demands of large-scale data transfer.

Content referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.

1. Keep Trying to Ship the AI Application Side Based on my understanding of operations, I developed an open-source operations tool, https://github.com/shaowenchen/ops . The Ops tool divides operations work into two categories — script execution and file distribution — and both operations targets, hosts and Kubernetes clusters, implement both of