Fault-Tolerant Training on Shaowen Chen's Websitehttps://www.chenshaowen.com/en/tags/fault-tolerant-training/Recent content in Fault-Tolerant Training on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sat, 17 Aug 2024 00:00:00 +0000weeklyElastic, Fault-Tolerant Training with DLRover-Managed Jobshttps://www.chenshaowen.com/en/blog/use-dlrover-to-manage-training-job.htmlSat, 17 Aug 2024 00:00:00 +0000Sat, 17 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-dlrover-to-manage-training-job.html1. Problems Facing Distributed Training Estimating training resources is difficult and cannot be automated How much compute, how much time, how much bandwidth, how many CPUs, how much memory — without enough accumulated experience it is hard to estimate accurately. The result is over-requesting and over-allocation, causing enormous resource waste.微信公众号DLRoverAITrainingKubernetesElastic TrainingFault-Tolerant TrainingOperations