This article is compiled from part of an internal sharing session I gave.
1. Core Elements of a Storage System
1.1 Security
Credentials for object storage buckets, authorization when using storage PVCs, and control over access sources β these are all issues security needs to address.
But they are also very easy to overlook, and when something goes wrong it is a big problem.
1.2 Lifecycle Management
A storage system exists to serve the business’s use of data.
The business’s lifecycle management of data directly shapes the functional requirements of the storage system.
Data generation - cleaning - usage - circulation - archiving: behind each of these processes is the need for support from the storage system and various components.
1.3 Performance
Performance is an important metric for measuring a storage system. Across the various storage systems, throughput, latency, and IOPS are all performance metrics we need to pay attention to.
With tools such as dd and fio, we can evaluate the performance of various storage systems. Then we tune, modify parameters, and increase bandwidth to reach optimal performance.
1.4 Storage Cost
Cost most directly influences a decision-maker’s storage choice.
Generally, the better the performance, the higher the cost the storage requires, while actual needs require us to evaluate for ourselves and make a balanced decision between performance and cost.
For many companies, storage cost accounts for a large portion of the overall IT cost. As a result, they are very sensitive to storage services used at scale, and decisions come with an extra non-technical consideration.
1.5 Operations Cost
Because of the rapid changes in infrastructure, traditional storage systems are not necessarily suited to cloud-native application scenarios. Using and operating these storage systems on Kubernetes is costly, and sometimes even deploying them is difficult.
A cloud-native-friendly storage system may need only a single helm install command to complete deployment, and it has good observability and self-healing characteristics β things not every storage system has.
At the same time, the day-to-day tedious business configuration consumes a great deal of the operations team’s energy and time; reducing the cost of manpower is a challenge.
2. The Stage of Storage Construction
We can always find plenty of best practices and get plenty of reference advice, but implementing them step by step directly will be fraught with difficulties, and may even provoke widespread anger.
These best practices are right, and we work hard too, yet we still do not get good results. That is because the stage we are at differs, the manpower invested differs, and the business form differs β many of the people who summarize best practices may themselves have overlooked these factors.
A successful summary is not the whole of success, but the parts the author considered difficult.
Analyzing the stage of the storage system lets us accurately identify the issues that most urgently need attention right now.
2.1 Early Stage: Operations Cost, Performance
In the early stage of a project, going from 0 to 1, we do not consider too much; as long as it can be implemented quickly, that is enough β the project needs to go live urgently.
This stage mainly relies on manual operations and configuration, and the opinions of the operations staff matter more than those of the decision-makers.
As long as performance is good enough and configuration is convenient, it can be landed and tried out first; other issues have relatively low priority.
2.2 Middle Stage: Storage Cost, Lifecycle Management
After a period of experimentation, everyone gains some understanding of the storage’s characteristics, and there are many things they want to optimize and adjust.
Meanwhile the storage scale grows larger, storage cost grows higher, and the pressure from the boss grows greater. At the same time, various business functions and cross-product, cross-team storage and circulation needs raise the complexity of the storage system and push up maintenance cost.
At this point the biggest problem is reducing storage cost based on hot/cold data tiering and on the differing performance requirements of different needs. Components such as Fluid are used to manage data and meet the needs of the layers above.
2.3 Late Stage: Security, Lifecycle Management
When you start paying attention to data security, the storage scale should already be very large, and the functional build-out should be fairly complete.
Data security is an issue you cannot avoid, and it carries enormous latent risk. A single major security incident can directly render all the previous effort worthless, and even cause the people involved to resign outright.
How to avoid these risks is the question the late stage urgently needs to consider.
3. Directions for Storage System Evolution
3.1 Object-Storage-Oriented Data Delivery
Currently our way of working is that the training and inference platforms share and copy data directly in JuiceFS. The current problems are:
- JuiceFS storage space is limited; although we purchased 1P and 500TB of enterprise-edition storage, it is still not enough
- JuiceFS costs more than object storage
- Data in JuiceFS lacks lifecycle management
Object-storage-oriented data delivery, by contrast, emphasizes that during delivery we must archive and seal the data. Data delivery for the data platform, training platform, and inference platform should be based on object storage, not JuiceFS.
Problems that can be solved after the evolution:
- Each platform manages its JuiceFS cache according to its own hot/cold data
- Helps decouple the platforms
- Removes the dependency on the dedicated intranet line; Alibaba Cloud accesses Kingsoft Cloud’s JuiceFS directly
- Each platform uses the cache service of the respective cloud vendor; JuiceFS on Kingsoft Cloud, Jindo on OSS
3.2 Central-Storage-Oriented R&D Workflow
A central-storage-oriented R&D workflow emphasizes that data in an intermediate state should be stored and circulated centrally.
The R&D workflow is an intermediate state with the following characteristics:
- No need to deliver data
- No need to deliver models
- Fast iteration
Advantages of a JuiceFS-oriented R&D workflow:
- JuiceFS has an absolute speed advantage
- Higher circulation efficiency, avoiding frequent data export
