DLRover
DLRover Workflow Analysis on Kubernetes
· ☕ 11 min read
The DLRover version used in this article is 0.3.7 1. DLRover Operator 1.1 Starting the ElasticJob and ScalePlan Controllers Implementation code: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // 创建 ElasticJob 的控制器 if err = controllers.NewElasticJobReconciler(mgr, masterImage).SetupWithManager(mgr); err

Elastic, Fault-Tolerant Training with DLRover-Managed Jobs
· ☕ 10 min read
1. Problems Facing Distributed Training Estimating training resources is difficult and cannot be automated How much compute, how much time, how much bandwidth, how many CPUs, how much memory — without enough accumulated experience it is hard to estimate accurately. The result is over-requesting and over-allocation, causing enormous resource waste.