1. What RDMA Is

RDMA (Remote Direct Memory Access) is a technology created to solve the delay of server-side data processing in network transmission.

With TCP/IP transmission, data passes through the network stack, then through the NIC to be sent; after the receiver receives it, the data is reassembled according to sequence numbers.

With DMA transmission, data can be transferred directly between the device and memory, without going through the network stack.

With RDMA transmission, DMA data transfer across nodes can be achieved.

The comparison among the three is as follows:

2. The RDMA Communication Process

RDMA uses a queue mechanism for data communication, and its basic communication unit is the QP (Queue Pair). A QP consists of one SQ (Send Queue) and one RQ (Receive Queue).

The general communication process is as follows:

1. The HOST submits a work request WR (Work Request), placing the WR into the work queue WQ (Work Queue).
2. The RDMA hardware consumes the WR in the WQE and performs the data transfer.
3. After the RDMA hardware finishes consuming, it generates a CQE (Completion Queue Entry), places the CQE into the CQ (Completion Queue), and waits for the HOST to consume it.
4. The HOST consumes a WC (Work Completion) from the CQ.

3. RDMA Technology Implementations

3.1 InfiniBand

Proposed in 1999 by the InfiniBand Trade Association (IBTA), it aims to provide high-bandwidth, low-latency connectivity for high-performance computing (HPC) and large-scale data centers. It is mainly driven by Intel, NVIDIA, Mellanox (now an NVIDIA subsidiary), IBM, and others, and is widely used in supercomputers and enterprise storage solutions.

InfiniBand has become one of the standards for data centers and HPC, supports multiple protocols (such as RDMA and NVMe over Fabrics), and is gradually expanding into the AI and machine learning fields.

InfiniBand offers high bandwidth (up to 400Gbps), low latency (microsecond level), support for large-scale scaling, and advanced traffic management and quality of service (QoS) features.

3.2. iWARP

Proposed in 2000 by the IETF’s RDMA Working Group, it aims to implement RDMA over existing TCP/IP networks. It is mainly driven by companies such as Intel, Cisco, and Broadcom, and is widely applied particularly in cloud computing and virtualization environments.

iWARP has seen some adoption in enterprise networks, especially for RDMA needs under traditional Ethernet architectures. It is gradually gaining market recognition, but its competitiveness is relatively weak.

iWARP has strong compatibility (it can run on existing Ethernet infrastructure) and supports the TCP/IP protocol, lowering the barrier to RDMA deployment and making it suitable for small and medium-sized data centers.

3.3. RoCE (RDMA over Converged Ethernet)

Proposed in 2008 by IBM and other companies, it aims to integrate RDMA into Ethernet environments and achieve efficient data transfer through DCB technology. It is mainly driven by Mellanox (now NVIDIA), Cisco, Intel, and others, and is gradually becoming widespread in data center and cloud computing environments.

RoCE has two versions. The RoCE v1 protocol is an Ethernet link-layer protocol, while the RoCEv2 protocol is built on top of UDP/IPv4 or UDP/IPv6 and can form larger networks.

Compared with InfiniBand, RoCE’s advantage is that it can run on existing Ethernet infrastructure, which can reduce part of the hardware cost, but it is also hard to get below 50% of InfiniBand’s cost.

4. RDMA under Ascend

The RDMA of the Atlas 800 relies on RoCE NICs, as shown below:

A single Board has four SF216D-H NICs, and each SF216D-H has two 200GE NICs. Each NIC connects to one NPU card and is used for networking.

When Huawei Altas 800 clusters are networked, an external switch connects the various NPU cards together to form a parameter plane network. Since the NIC ports on each switch are limited, a larger network requires a Spine-Leaf network topology.

4.1 Common Configuration Commands

• Set the IP address and netmask

1

hccn_tool -i 0 -ip -s address 10.52.11.2 netmask 255.255.255.0

-i specifies the card index.

• Set the default gateway of the RoCE NIC

1

hccn_tool -i 0  -gateway -s gateway 10.52.11.1

• Set the network detection target

1

hccn_tool -i 0 -netdetect -s address 10.52.11.1

This IP is mainly used to detect network status; it is generally set to the gateway address within the subnet.

• Enable TLS

1

for i in {0..7}; do hccn_tool -i $i -tls -s enable 0; done

4.2 Common Inspection Commands

• View the networking IPs

1

cat /etc/hccn.conf

Or

1

for i in {0..7}; do hccn_tool -i $i -ip -g; done

• Check the connectivity of NIC IPs within a single node

1

for i in {0..7};do hccn_tool -i $i -net_health -g;done

• View the connection status of the communication ports

1

for i in {0..7};do hccn_tool -i ${i} -link -g;done

• Card health status

1

for i in {0..7};do npu-smi info -t health -i $i -c 0;done

• ECC

1

for i in {0..7};do npu-smi info -t ecc -i $i;done

• pci

1

lspci |grep d80

4.3 Common Networking Commands

• View the optical module status

1

for i in {0..7};do hccn_tool -i ${i} -optical -g;done

• Query the tls status

1

for i in {0..7}; do hccn_tool -i $i -tls -g; done |grep switch

• Test connectivity

1
2

export ADDRESS=10.52.41.10
for i in {0..7}; do hccn_tool -i $i -ping -g address ${ADDRESS} ; done

• Full-mesh test between two nodes

1

echo "ADDRESSES=$(grep -oP '^address_\d+=\K[\d.]+' /etc/hccn.conf | paste -sd,)"

Obtain the ADDRESSES variable and set it on the other host.

1
2
3
4
5

IFS=',' read -ra ADDR_ARRAY <<< "$ADDRESSES"
for addr in "${ADDR_ARRAY[@]}"; do
  echo "testing ${addr}"
  for i in {0..7}; do hccn_tool -i $i -ping -g address ${addr} ; done
done

4.4 Testing Cross-Host Bandwidth

• Get the IP of the receiving-side card

1
2
3
4

hccn_tool -i 0 -ip -g

ipaddr:10.52.11.3
netmask:255.255.255.0

• Receiver

1
2

hccn_tool -i 0 -roce_test reset
hccn_tool -i 0 -roce_test ib_send_bw -s 4096000 -n 1000 -tcp

• Sender

1
2

hccn_tool -i 0 -roce_test reset
hccn_tool -i 0 -roce_test ib_send_bw -s 4096000 -n 1000 address 10.52.41.20 -tcp

5. Some References

CANN Commercial Edition 8.0.RC3 Collective Communication User Guide 01