How to Add Huawei NPU Nodes to a Kubernetes Cluster

1. Disk Handling

1.1 Inspect Disks

Inspect the new disk

1
fdisk -l

Disk /dev/nvme1n1: 3.91 TiB, 4294967296000 bytes, 8388608000 sectors

1.2 Build a RAID0

If you have several small disks, a better approach is to build a RAID0 — this gives you not only a larger storage directory but also higher speed.

Create the RAID

1
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=3 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1

Inspect the RAID

1
mdadm --detail /dev/md0

1.3 Mount the Disk

Create the filesystem

1
mkfs.xfs -f /dev/nvme1n1

mkfs.ext4 /dev/nvme1n1

xfs is suited to large-file workloads, ext4 to small and medium files.

For a RAID0 you can run mkfs.xfs -f /dev/md0 .

Create the mount directory

1
mkdir -p /data

Get the UUID

1
2
UUID=$(blkid -s UUID -o value /dev/nvme1n1)
echo $UUID

Configure fstab

1
grep -q "$UUID" /etc/fstab || echo "UUID=$UUID /data xfs defaults,nofail 0 2" >> /etc/fstab

1
grep -q "$UUID" /etc/fstab || echo "UUID=$UUID /data ext4 defaults,nofail 0 2" >> /etc/fstab

Mount the storage

1
mount -a

Check the mount point

1
2
3
df -h  |grep data

/dev/nvme1n1    4.0T   28G  3.9T   1% /data

2. Drivers and Configuration

2.1 Check Whether the System Recognizes the NPU

1
lspci |grep Huawei

1
18:00.0 Processing accelerators: Huawei Technologies Co., Ltd. Device d802 (rev 20)

The device is identified as a Huawei Ascend NPU.

2.2 Install the Driver

Create the HwHiAiUser user

1
2
groupadd -g 1000 HwHiAiUser
useradd -g HwHiAiUser -u 1000 -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash

Add directory permissions

1
2
chown -R HwHiAiUser /usr/local/Ascend
chmod -R 755 /usr/local/Ascend

Download the driver and firmware

Go to https://www.hiascend.com/hardware/firmware-drivers/community?product=4&model=26&cann=9.0.0-beta.1&driver=Ascend+HDK+25.5.1 to find the matching driver and firmware.

Install the driver

1
bash ./Ascend-hdk-910b-npu-driver_25.5.1_linux-x86-64.run --full --install-for-all

Install the firmware

1
bash ./Ascend-hdk-910b-npu-firmware_7.8.0.6.201.run --full

Verify that the installation succeeded

1
npu-smi info

2.3 Disable TLS

Disable

1
for i in {0..15};do hccn_tool -i $i -tls -s enable 0;done

Check

1
for i in {0..15};do hccn_tool -i $i -tls -g| grep 'switch';done

2.4 Check the IP

Check the IP

1
for i in {0..15};do hccn_tool  -i $i  -ip -g;done

Check the gateway

1
for i in {0..15};do hccn_tool -i $i -gateway -g;done

2.5 NIC Health Status

Card health status:

1
for i in {0..15};do npu-smi info -t health -i $i -c 0| grep 'Health Status';done

To see the specific reason: npu-smi info -t health -i -c 0

Link status

1
for i in {0..15};do hccn_tool -i $i -link -g;done

Network health status

1
for i in {0..15};do hccn_tool -i $i -net_health -g;done

1
for i in {0..15};do npu-smi info -t ecc -i $i;done

ECC counts cause program error 0x80E18005.

3. Installing ascend-docker-runtime

3.1 Install ascend-docker-runtime

Download ascend-docker-runtime

Go to https://gitcode.com/Ascend/mind-cluster/releases to find the download link for the matching architecture.

1
wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.1.0.beta.2/Ascend-docker-runtime_26.1.0.beta.2_linux-x86_64.run

Install ascend-docker-runtime

1
bash ./Ascend-docker-runtime_26.1.0.beta.2_linux-x86_64.run --install

3.2 Install Containerd

Add the repository

1
2
apt-get update
apt-get install -y ca-certificates curl gnupg lsb-release

In China:

1
2
mkdir -p /etc/apt/keyrings
curl -fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

1
2
3
4
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/ \
 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install containerd

1
2
apt update
apt install containerd.io=1.6.31-1

Generate the containerd config file (toml)

1
2
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

Modify the containerd config file

1
2
3
4
sed -i 's#root = "/var/lib/containerd"#root = "/data/containerd"#g' /etc/containerd/config.toml
sed -i 's#state = "/run/containerd"#state = "/data/run/containerd"#g' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:[^"]*"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#g' /etc/containerd/config.toml
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml

Change the runtime to ascend

1
sed -i 's#runtime = "runc"#runtime = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime"#g' /etc/containerd/config.toml

Restart containerd

1
2
systemctl daemon-reload
systemctl restart containerd

3.3 Docker Configuration [Optional]

Choose either Docker or Containerd.

Update the Docker configuration

1
vim /etc/docker/daemon.json

Add the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "default-runtime": "ascend",
  "data-root": "/data/docker",
  "runtimes": {
    "ascend": {
      "path": "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime",
      "runtimeArgs": []
    }
  }
}

Restart Docker

1
2
systemctl daemon-reload
systemctl restart docker

3.4 Verifying with hccl_test

1
export IMAGE=shaowenchen/demo:hccl-test-8.0.RC2-ubuntu22.04

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
nerdctl run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -it --ipc=host \
      --device=/dev/davinci0 \
      --device=/dev/davinci1 \
      --device=/dev/davinci2 \
      --device=/dev/davinci3 \
      --device=/dev/davinci4 \
      --device=/dev/davinci5 \
      --device=/dev/davinci6 \
      --device=/dev/davinci7 \
      --device=/dev/davinci8 \
      --device=/dev/davinci9 \
      --device=/dev/davinci10 \
      --device=/dev/davinci11 \
      --device=/dev/davinci12 \
      --device=/dev/davinci13 \
      --device=/dev/davinci14 \
      --device=/dev/davinci15 \
      --device=/dev/davinci_manager \
      --device=/dev/devmm_svm \
      --device=/dev/hisi_hdc \
      -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
      -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
      -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
      -v /var/log/npu/conf/slog/slog.conf:/var/log/npu/conf/slog/slog.conf \
      -v /var/log/npu/slog/:/var/log/npu/slog \
      -v /var/log/npu/profiling/:/var/log/npu/profiling \
      -v /var/log/npu/dump/:/var/log/npu/dump \
      -v /var/log/npu/:/usr/slog \
      ${IMAGE} \
      /bin/bash

1
2
3
4
5
export HCCL_RDMA_TC=100
export HCCL_RDMA_SL=3
export HCCL_BUFFSIZE=2048
export HCCL_PROC=16
mpirun -n ${HCCL_PROC} /usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test/bin/all_reduce_test -b 8K -e 64M -f 2 -d fp32 -o sum -p ${HCCL_PROC}

4. Joining the K8s Cluster

4.1 Change the Hostname

1
export HOSTNAME=k8s-worker-npu-01

1
hostnamectl set-hostname ${HOSTNAME}

Create the working directory for the device management plugin

1
mkdir -p /var/log/mindx-dl/devicePlugin

4.2 Initialize Kernel Parameters

1
opscli task -f ~/.ops/tasks/set-host.yaml

4.3 Install the Base K8s Components

Add the K8s repository

https://developer.aliyun.com/mirror/kubernetes/ — add this for versions below 1.28

1
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/kubernetes-aliyun.gpg

1
2
3
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

1
apt-get update

Symlink the kubelet directory

1
2
mkdir -p /data/kubelet
ln -s /data/kubelet /var/lib/kubelet

Install the base K8s components

1
export K8S_VERSION=1.27.6

1
apt-get install kubeadm=${K8S_VERSION}-00 kubelet=${K8S_VERSION}-00 kubectl=${K8S_VERSION}-00 -y

Configure kubelet

1
vim /var/lib/kubelet/kubeadm-flags.env

Add the following:

1
--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 --resolv-conf=/etc/resolv.conf  --authentication-token-webhook=true --authorization-mode=Webhook --system-reserved=cpu=1,memory=2Gi --kube-reserved=cpu=1,memory=2Gi --cpu-manager-policy=static

NUMA support

1
--topology-manager-policy=best-effort --topology-manager-scope=container --feature-gates=TopologyManager=true,CPUManager=true,MemoryManager=true

topology-manager-policy can be best-effort (satisfy if possible), restricted (one resource does not cross nodes), or single-numa-node (no resource crosses nodes).
topology-manager-scope can be container (satisfied at the container level) or all containers within the pod.
feature-gates can be TopologyManager=true,CPUManager=true,MemoryManager=true to enable topology management, CPU management, and memory management.

Multiple NICs

In the kubelet configuration above, add the --node-ip= parameter to specify the node IP address and the --hostname-override= parameter to specify the node Hostname.

1
--node-ip=x.x.x.x --hostname-override=my-hostname

With Flannel, you need to add the --iface= parameter to the DeamonSet to specify the NIC.

1
2
3
4
5
6
7
containers:
  - args:
      - --ip-masq
      - --kube-subnet-mgr
      - --iface-regex=^10\.0\.
    command:
      - /opt/bin/flanneld

Restart kubelet

1
2
3
4
systemctl enable kubelet
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet

4.4 Join the Cluster

Generate a token

Generate the token on the master node

1
kubeadm token create --print-join-command

Join the cluster

1
2
kubeadm join x.x.x.x:6443 --token xxx \
    --discovery-token-ca-cert-hash sha256:xxx

In a Docker environment, you need to pass the --cri-socket parameter.

1
--cri-socket unix:///var/run/cri-dockerd.sock

In a Containerd environment:

1
--cri-socket unix:///run/containerd/containerd.sock -v5

If you get the error [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist, run this.

1
2
3
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward

6. Deploying k8s Plugins

6.1 Install the device-plugin to register huawei.com/Ascend910

Install device-plugin

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/ascend/v7.1.RC1-device-plugin-910.yaml

Edit

1
kubectl -n kube-system edit ds ascend-device-plugin-daemonset-910

1
-useAscendDocker=false -volcanoType=false

The plugin contains runtime- and volcano-related configuration that needs to be adjusted for your environment

6.2 Install rdma-shared-dev-plugin to register rdma/ib

To let Kubernetes discover RDMA devices such as InfiniBand and share them among multiple Pods, you need to install k8s-rdma-shared-dev-plugin.

Install k8s-rdma-shared-dev-plugin

1
kubectl apply -f https://raw.githubusercontent.com/shaowenchen/ops-hub/master/network/k8s-rdma-shared-dev-plugin.yaml

Modify the configuration file

1
kubectl -n kube-system edit cm rdma-devices

6.3 Create a Test Pod

Create the Pod

1
export IMAGE=shaowenchen/demo:hccl-test-8.0.RC2-ubuntu22.04

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  generateName: npu-demo-
  labels:
    app: npu-demo
spec:
  nodeName: ${HOSTNAME}
  hostIPC: true
  containers:
    - name: npu-demo
      image: ${IMAGE}
      command: ["/bin/bash", "-c", "sleep infinity"]
      securityContext:
        privileged: true
      resources:
        requests:
          huawei.com/Ascend910: 4
        limits:
          huawei.com/Ascend910: 4
      volumeMounts:
      - mountPath: /usr/local/bin/npu-smi
        name: npu-smi
        readOnly: true
      - mountPath: /usr/local/Ascend/driver
        name: ascend-driver
        readOnly: true
  volumes:
    - name: npu-smi
      hostPath:
        path: /usr/local/bin/npu-smi
    - name: ascend-driver
      hostPath:
        path: /usr/local/Ascend/driver
EOF

Check the Pod status

1
kubectl get pod -l app=npu-demo

Check the cards

1
kubectl exec -it npu-demo-pm28m npu-smi info

Delete the Pod

1
kubectl delete pod -l app=npu-demo

7. Performance Optimization

Lock the CPU to the high-performance frequency mode; the downside is higher power consumption

Temporarily set the CPU to the locked high-performance mode

1
2
3
for cpu in /sys/devices/system/cpu/cpu[0-9]*; do
  echo performance | sudo tee $cpu/cpufreq/scaling_governor
done

Permanently set the CPU to the locked high-performance mode

1
apt-get install tuned -y

1
tuned-adm profile throughput-performance

Check the CPU operating mode

1
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

System THP tuning; the downside is reduced memory utilization

Disable or tune THP (Transparent Huge Pages) automatic compaction to prevent kernel background defrag from causing latency stalls and periodic latency jitter.

Temporarily disable THP

1
2
echo defer > /sys/kernel/mm/transparent_hugepage/defrag
echo 'defer+madvise' > /sys/kernel/mm/transparent_hugepage/defrag

Permanently disable THP

1
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Check the THP status

1
cat /sys/kernel/mm/transparent_hugepage/defrag

NIC ring buffer (packet loss issue); the downside is a larger buffer and increased network latency

The default value is usually 256 or 512, which is too small for high-speed 10G/100G NICs.

Temporarily set the NIC ring buffer

1
2
ethtool -G eth0 rx 8192 tx 8192
ethtool -G eth1 rx 8192 tx 8192

Permanently set the NIC ring buffer

1
2
3
4
5
6
cat >/etc/ethtool-ring.sh <<'EOF'
#!/bin/bash
ethtool -G eth0 rx 8192 tx 8192 || true
ethtool -G eth1 rx 8192 tx 8192 || true
EOF
chmod +x /etc/ethtool-ring.sh

Or configure bond0

1
2
3
4
5
6
7
8
9
cat >/etc/ethtool-ring.sh <<'EOF'
#!/bin/bash
SLAVES=$(cat /sys/class/net/bond0/bonding/slaves 2>/dev/null)

for eth in $SLAVES; do
    ethtool -G $eth rx 8192 tx 8192 || true
done
EOF
chmod +x /etc/ethtool-ring.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
cat >/etc/systemd/system/ethtool-ring.service <<'EOF'
[Unit]
Description=Set NIC ring buffer size
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/ethtool-ring.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable ethtool-ring.service
systemctl start ethtool-ring.service

Check the NIC ring buffer

1
2
ethtool -g eth0
ethtool -g eth1

Check the bond0 NIC ring buffer

1
2
3
4
5
SLAVES=$(cat /sys/class/net/bond0/bonding/slaves 2>/dev/null)

for eth in $SLAVES; do
    ethtool -g $eth || true
done

Change the machine clocksource from hpet to tsc; the downside is possible desynchronization on old CPUs or multi-CPU systems

TSC (Time Stamp Counter) is a CPU built-in timer with extremely fast access and low latency (nanosecond level)

Temporarily set the clocksource

1
echo tsc | tee /sys/devices/system/clocksource/clocksource0/current_clocksource

Permanently set the clocksource

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
cat >/etc/systemd/system/clocksource-tsc.service <<'EOF'
[Unit]
Description=Set clocksource to tsc
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource'

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable clocksource-tsc.service
systemctl start clocksource-tsc.service

Check the current clocksource

1
cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Set fan speed to maximum

This is usually adjusted in the BMC (out-of-band management) or the BIOS.

Kernel memory and scheduler tuning

Lower the swap tendency so training is not slowed down by swapping; disable automatic NUMA balancing to reduce the jitter caused by pages migrating across nodes.

Temporarily set

1
2
3
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
echo 50000 > /sys/kernel/debug/sched/migration_cost_ns

Permanently set vm.swappiness and kernel.numa_balancing

1
2
3
4
5
6
cat >/etc/sysctl.d/99-npu-perf.conf <<'EOF'
vm.swappiness = 0
kernel.numa_balancing = 0
EOF

sysctl -p /etc/sysctl.d/99-npu-perf.conf

Permanently set migration_cost_ns

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
cat >/etc/systemd/system/sched-migration-cost.service <<'EOF'
[Unit]
Description=Set sched migration_cost_ns
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo 50000 > /sys/kernel/debug/sched/migration_cost_ns'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable sched-migration-cost.service
systemctl start sched-migration-cost.service

Check the current configuration

1
2
sysctl vm.swappiness kernel.numa_balancing
cat /sys/kernel/debug/sched/migration_cost_ns