1. Installing Dependencies
MindCluster ToolBox is a system-level toolset for cluster operations and hardware management, mainly used for operations such as device monitoring, performance testing, and log collection.
MindCluster ToolBox includes the Ascend DMI tool, the log collection tool, and the Ascend Cert tool.
MindCluster ToolBox works by calling low-level DCMI (Device Control Management Interface) and AscendCL (Ascend Computing Language) interfaces to perform the relevant detection functions; system-level information queries are implemented by calling the general-purpose libraries provided by the system.
1
| dmidecode -s system-product-name
|
1
| Atlas 900 RCK A2 Compute Node
|
- Check the operating system
1
| uname -m && cat /etc/*release
|
1
2
| aarch64
BigCloud Enterprise Linux For Euler release 21.10 (LTS-SP2)
|
- Check whether the current environment is supported
Go to https://www.hiascend.com/document/detail/zh/mindx-dl/600/toolbox/ascenddmi/toolboxug_0003.html
The toolbox depends on the libascendcl.so library provided by Ascend Toolkit.
Go to https://www.hiascend.com/developer/download/community/result?module=dl%2Bcann to download the MindCluster ToolKit installation package. Here we take Ascend-cann-toolkit_8.1.RC1_linux-aarch64.run as an example.
1
| chmod +x Ascend-cann-toolkit_8.1.RC1_linux-aarch64.run
|
1
| ./Ascend-cann-toolkit_8.1.RC1_linux-aarch64.run --install --quiet
|
- Set the toolkit environment variables
1
2
| echo "source /usr/local/Ascend/ascend-toolkit/set_env.sh" >> ~/.bashrc
source ~/.bashrc
|
1
| cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg
|
Go to https://www.hiascend.com/developer/download/community/result?module=dl%2Bcann to download the MindCluster ToolBox installation package. Here we take Ascend-mindx-toolbox_7.0.RC1_linux-aarch64.run as an example.
1
| chmod +x Ascend-cann-toolkit_8.1.RC1_linux-aarch64.run
|
1
| ./Ascend-mindx-toolbox_7.0.RC1_linux-aarch64.run --install --install-for-all --quiet
|
- Set the toolbox environment variables
1
2
| echo "source /usr/local/Ascend/toolbox/set_env.sh" >> ~/.bashrc
source ~/.bashrc
|
1.3 Verifying the Installation
1
| ascend-dmi version: 7.0.RC1
|
This is all you need to start using it.
- View the real-time status of all NPUs
You can refer to information such as memory usage, temperature, and power consumption.
- Check HCCS/PCIe signal quality
The normal range is
1
2
3
4
5
| Normal range:
B (bottom) <= -17
T (top) >= 17
L (left) <= -3
R (right) >= 3
|
Use -d to specify the device ID; the default is 0. Use --all to test all devices.
- Verify whether the compute throughput meets the target
1
| ascend-dmi --flops --quiet
|
- Test host-device bandwidth
1
| ascend-dmi --bw -t h2d --quiet
|
- Test device-host bandwidth
1
| ascend-dmi --bw -t d2h --quiet
|
- Test device-device bandwidth
1
| ascend-dmi --bw -t p2p --ds 0 --dd 1 --quiet
|
- Test the bandwidth from each device to every other device
1
| ascend-dmi --bw -t p2p --quiet
|
4. Fault Diagnosis
For CANN/driver/device/network/signalQuality/on-chip memory
1
| ascend-dmi --dg --se healthCheck --quiet
|
Check items such as driver, device, network, signalQuality, hbm, and cann will appear.
- Performance specifications
For BandWidth/Aiflops
1
| ascend-dmi --dg --se performanceCheck --quiet
|
For Aicore/on-chip memory (full stress test)/P2P stress test/power stress test
1
| ascend-dmi --dg --se stressTest --quiet
|
5. NPU Environment Recovery
Used to recover the NPU environment when a voltage anomaly or card drop occurs. Use -d to specify the device ID; the default is all devices.
