The DLRover version used in this article is 0.3.7 1. DLRover Operator 1.1 Starting the ElasticJob and ScalePlan Controllers Implementation code: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // 创建 ElasticJob 的控制器 if err = controllers.NewElasticJobReconciler(mgr, masterImage).SetupWithManager(mgr); err

1. Problems Facing Distributed Training Estimating training resources is difficult and cannot be automated How much compute, how much time, how much bandwidth, how many CPUs, how much memory — without enough accumulated experience it is hard to estimate accurately. The result is over-requesting and over-allocation, causing enormous resource waste.

1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.

1. Storage Options There are three storage approaches: Isolate by directory and share a single JuiceFS Elasticsearch nodes share one JuiceFS instance, mounting different Elasticsearch nodes through subdirectories. /0/ corresponds to node Node-0 /1/ corresponds to node Node-1 /2/ corresponds to node Node-2 The main advantage of this approach is that it is easy to scale and simple to configure.

1. Packaging the Fluid Runtime Image Create the fluid_config_init.py script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!/usr/bin/env python import json rawStr = "" with open("/etc/fluid/config.json", "r") as f: rawStr = f.