https://www.chenshaowen.com/en/tags/volcano/Recent content in Volcano on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 11 Aug 2024 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/use-volcano-to-run-hccl-test.htmlSun, 11 Aug 2024 00:00:00 +0000Sun, 11 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-volcano-to-run-hccl-test.html1. Building the hccl-test Image Download the dependency packages Python-3.8.18.tgz Ascend-cann-toolkit_8.0.RC2_linux-x86_64.run Ascend-cann-kernels-910b_8.0.RC2_linux.run mpich-3.2.1.tar.gz If downloading is inconvenient, you can also copy them straight out of the image I built. Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 FROM ubuntu:22.微信公众号VolcanoHCCLAscendAITestinghttps://www.chenshaowen.com/en/blog/use-volcano-to-run-nccl-test.htmlSun, 11 Aug 2024 00:00:00 +0000Sun, 11 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-volcano-to-run-nccl-test.html1. Building the nccl-test Image Check the CUDA version 1 2 3 nvidia-smi | grep "CUDA Version" | awk '{print $9}' 12.2 Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > Dockerfile << EOF FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04 ENV DEBIAN_FRONTEND=noninteractive ARG CONDA_VERSION WORKDIR /workspace ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt install -y openmpi-bin libopenmpi-dev ssh openssh-server net-tools vim git iputils-ping nfs-common RUN git clone https://github.微信公众号VolcanoNCCLNvidiaAITestinghttps://www.chenshaowen.com/en/blog/the-basic-of-volcano.htmlSun, 24 Mar 2024 00:00:00 +0000Sun, 24 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-basic-of-volcano.html1. Introduction to Volcano Volcano is a Kubernetes-based resource scheduling system open-sourced by Huawei. Compared with the native scheduler, its notable features are: Supports gang scheduling Scheduling batch jobs easily runs into deadlock: for example, two jobs each need 10 Pods running simultaneously to start. When both jobs are submitted at the same time, it is possible that only part of the Pods of each are scheduled, so neither job can run properly and they wait on each other.微信公众号VolcanoAIKubernetesSchedulingOperations