1. Problems Facing Distributed Training Estimating training resources is difficult and cannot be automated How much compute, how much time, how much bandwidth, how many CPUs, how much memory — without enough accumulated experience it is hard to estimate accurately. The result is over-requesting and over-allocation, causing enormous resource waste.

1. Building the hccl-test Image Download the dependency packages Python-3.8.18.tgz Ascend-cann-toolkit_8.0.RC2_linux-x86_64.run Ascend-cann-kernels-910b_8.0.RC2_linux.run mpich-3.2.1.tar.gz If downloading is inconvenient, you can also copy them straight out of the image I built. Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 FROM ubuntu:22.

1. Building the nccl-test Image Check the CUDA version 1 2 3 nvidia-smi | grep "CUDA Version" | awk '{print $9}' 12.2 Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > Dockerfile << EOF FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04 ENV DEBIAN_FRONTEND=noninteractive ARG CONDA_VERSION WORKDIR /workspace ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt install -y openmpi-bin libopenmpi-dev ssh openssh-server net-tools vim git iputils-ping nfs-common RUN git clone https://github.

This article is compiled from part of an internal sharing session I gave. 1. Core Elements of a Storage System 1.1 Security Credentials for object storage buckets, authorization when using storage PVCs, and control over access sources — these are all issues security needs to address. But they are also very easy to overlook, and when something goes wrong it is a big problem.

1. Using Memory as a Storage Medium The image above is the storage pyramid, showing the relationship between the price of a storage medium and its speed. Today, enterprises widely adopt tape and disks as storage media mainly because of their price advantage. Market prices are driven by supply and demand: even for an expensive storage medium, if the production cost is low, the market is large enough, and the yield rate in the production process is high, its market price will come down.