Content referenced or recorded while troubleshooting, continuously updated
1. Mounting Devices into a Container
1
| export IMAGE=ascendai/pytorch:2.1.0
|
1
2
3
4
5
6
7
8
9
10
| nerdctl run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -it --ipc=host \
--device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
${IMAGE} \
/bin/bash
|
2. Creating a Pod
1
2
| export IMAGE=ascendai/pytorch:2.1.0
export NodeName=
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: test-ascend-pod
namespace: default
spec:
restartPolicy: Never
nodeName: ${NodeName}
containers:
- name: ascend-container
image: ${IMAGE}
command: ["/bin/bash"]
args: ["-c", "sleep infinity"]
resources:
limits:
huawei.com/Ascend910: "1"
requests:
huawei.com/Ascend910: "1"
EOF
|
3. Driver Upgrade
1
2
3
4
5
6
7
8
9
| mkdir -p /data/paascontainer/ops
cd /data/paascontainer/ops
rm -f *.run
wget http://mirrors-internal.cmecloud.cn/coca/huawei/910b/Ascend-hdk-910b-npu-firmware_7.7.0.1.231.run
wget http://mirrors-internal.cmecloud.cn/coca/huawei/910b/Ascend-hdk-910b-npu-driver_25.0.rc1.1_linux-aarch64.run
chmod +x *.run
./Ascend-hdk-*-npu-firmware_*.run --upgrade
./Ascend-hdk-*-npu-driver_*.run --upgrade
reboot
|
4. Device Cannot Register with the Cluster
- Cannot schedule to certain cards
1
| DeviceInfoCfg: '{"DeviceInfo":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-10,Ascend910-11,Ascend910-12,Ascend910-13,Ascend910-14,Ascend910-8,Ascend910-9","huawei.com/Ascend910-Fault":"[{\"fault_type\":\"CardUnhealthy\",\"npu_name\":\"Ascend910-15\",\"large_model_fault_level\":\"ManuallySeparateNPU\",\"fault_level\":\"ManuallySeparateNPU\",\"fault_handling\":\"ManuallySeparateNPU\",\"fault_code\":\"\",\"fault_time_and_level_map\":{}}]","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Recovering":"","huawei.com/Ascend910-Unhealthy":"Ascend910-15"}},"SuperPodID":-1,"ServerIndex":1023}'
|
- After repairing the card, delete the cm file
After repairing the card, the configuration file that records the state must be deleted manually.
1
| kubectl -n kube-system delete cm mindx-dl-deviceinfo-
|
