1. Data Delivery Based on Object Storage
As shown above, model R&D mainly involves three sub-platforms:
- Data platform
The data platform is mainly responsible for data-related management, such as data ingestion and data processing, ultimately producing the data needed for training.
The data platform stores raw data in object storage. During processing, it fetches data from object storage, processes it, and stores the result back in object storage.
- Training platform
The training platform is mainly responsible for model training, such as providing the model development environment and supporting distributed model training.
The training platform provides the algorithm engineers with a Jupyter environment equipped with AI compute cards to support their online development and debugging. They store their code on storage devices with high real-time performance, and access files such as checkpoints and models by mounting them.
When a large distributed training job is launched, the data is warmed up layer by layer all the way into the VRAM of the AI compute cards, then the model is trained, and finally a model is produced.
- Inference platform
The inference platform is mainly responsible for bringing models online and managing traffic, such as model quantization, model conversion, and model deployment.
The inference platform fetches models from object storage, starts the workflow, deploys the models, and implements governance of inference traffic.
The data platform, training platform, and inference platform usually have different technical owners, so we cannot force all platforms to use a unified data definition and usage convention.
Therefore what is emphasized here is data delivery: we only need to define data delivery well. As for how each platform implements things internally, what mounting method or acceleration component it chooses, and so on, each platform decides for itself.
Considering cost, performance, and delivery convenience, data delivery based on object storage is a very good choice. The underlying layer of many data lake solutions is also based on object storage.
2. Data Platform
Although AI accelerator cards are good at data processing, the data platform usually does not use these expensive devices to process data.
AI jobs mainly consume AI compute card resources. In our production clusters, the VRAM usage of GPU and NPU cards often reaches 90%, while the CPU and memory usage on the nodes is only around 10%.
And accelerated compute nodes are high-spec machines; they usually come with hundreds of GB, or even several TB, of memory, and CPU compute power starting at 200 cores. This means that on accelerated compute nodes we still have plenty of resources available.
The workload of the data platform is very well suited to being co-deployed with AI jobs.
As shown above, the storage of the data platform falls mainly into two categories:
- Persistent storage, such as ingested data and the data needed for training
- Temporary storage, such as the data accelerated during processing, and the ES storage volumes that support full-text search over datasets
3. Training Platform
As shown above, the training process mainly involves three storage interactions:
- Algorithm development, storing the algorithm engineers’ code
The code consists of small files, and it also has relatively high real-time requirements, so it is not suitable for storing in JuiceFS. NFS storage can be used instead, with different directories isolating and distinguishing individuals and teams.
- Training data, storing the training data
Training data is characterized by a particularly large number of small files and high demands on metadata access speed, so the JuiceFS multi-zone enterprise edition can be used.
- Model data and checkpoint data
While checkpoints are being stored, the training job is in a paused state. The speed of checkpoint storage affects training efficiency.
The temporary storage of model data and checkpoint data should also use JuiceFS, but it can be downgraded to the single-zone enterprise edition or community edition.
After training completes, we should set up an archiving job to store the models, data, and so on produced during training into object storage.
JuiceFS should serve as a kind of temporary storage, as an acceleration layer, and should not store data for long periods.
4. Inference Platform
We usually train in one region, but need to run inference in multiple regions.
The reasons for multi-region inference may include:
- Upper-layer services are deployed across multiple clouds and need nearby inference
- A single cloud does not have enough AI compute card resources
- Multi-cloud high availability
But this also brings a new problem: how to distribute models across multiple cloud regions.
As shown above, the raw model data should be stored in object storage, and the model is distributed across clouds over the public network or a dedicated line. For example, syncing model data from Alibaba Cloud’s OSS to Tencent Cloud’s COS β object storage sync between public clouds is relatively fast.
In each region, with the help of the JuiceFS single-zone enterprise edition or community edition, the model is loaded into JuiceFS, mounted into the cluster’s inference Pods via a PVC, and inference is performed.
Multiple Kubernetes clusters in the same region can share a single JuiceFS store.
