1. What Is MLOps
MLOps is short for Machine Learning Operations, and it describes the standardization and engineering of the entire lifecycle around model development.
MLOps includes the following key steps:
- Data management: storing, accessing, cleaning, and transforming data
- Model development: algorithm development and model construction
- Model training and tuning: training models on data, adjusting hyperparameters to optimize the model, and fine-tuning models
- Model evaluation: testing a model’s accuracy, generalization ability, and performance metrics
- Model deployment: deploying the model to the target environment and converting the model
- Model monitoring: monitoring the model’s behavior, performance degradation, data drift, and robustness
Another similar concept is ModelOps. ModelOps covers a broader scope: it includes not only the machine learning models that MLOps focuses on, but all models in the field of artificial intelligence.
2. The Benefits of MLOps
2.1 Improving Model Development Efficiency
- Automation tools accelerate processes such as model training, deployment, and evaluation
- Promoting collaboration improves a team’s working efficiency
2.2 Improving Consistency
- Through version control, every step — data, model parameters, experiments — can be traced
- Standardized processes reduce the inconsistency introduced by manual operations
- Environment management guarantees that every environment is reproducible and deliverable
2.3 Promoting Division of Labor and Collaboration
- Clear responsibilities: algorithm, data, and operations engineers each own their own work
- Specialized division of labor, with different forms of collaboration
3. The Core Problems of MLOps
- Shortening the model development iteration cycle
Many companies take a long time to get a model into production, often 1 to 3 months. MLOps aims to accelerate this process through standardized, automated workflows.
- Seamless collaboration between roles
The model development process involves many roles: business, data, algorithms, and operations. MLOps aims to build a unified platform for collaboration among all these roles during model development.
4. DevOps VS MLOps
| DevOps | MLOps | |
|---|---|---|
| Focus | Software development and IT operations | Development, deployment, and operations of machine learning models |
| Team composition | Software engineers and IT operations staff | Data scientists, machine learning engineers, and IT operations staff |
| Development tasks | Writing new feature code or fixing existing bugs | Building and training machine learning models, exploratory experimentation |
| Testing scope | Running unit and integration tests, focusing on code quality | Beyond routine testing, also validating data and model quality |
| Deployment process | Deploying code and applications | Deploying machine learning pipelines, automatically retraining and deploying models |
| Production environment | Monitoring errors and performance of applications and the production environment | Monitoring model performance, tracking data changes and model decay |
| Continuous Integration (CI) | Testing and validating code and components | Testing and validating data, data schemas, and models |
| Continuous Delivery (CD) | Deploying software packages or services | Deploying machine learning training pipelines, automatically deploying model prediction services |
| Continuous Training (CT) | - | Automatically retraining and serving models, unique to ML systems |
| Goal | Accelerating software delivery and improving reliability | Improving the efficiency and reliability of machine learning models |
| Challenges | The challenges of continuous integration and automated deployment | Model drift, data bias, model interpretability, etc. |
5. The Stages MLOps Includes
- Data extraction
Determine the data you need and acquire the raw dataset.
- Data analysis
Explore the statistical characteristics of the data, such as value ranges and distributions; check data quality, looking for missing values and outliers; identify data features and patterns.
- Data preparation
Split the data into training, validation, and test sets; perform the necessary preprocessing on the data, such as normalization and encoding; convert the data into a format the model can accept.
- Model training
Choose an appropriate algorithm and model architecture, train the model, and save the trained model.
- Model evaluation
Evaluate the model’s performance on the test set, analyze its strengths and weaknesses, and compare its performance against a baseline model.
- Model validation
Confirm whether the model reaches a deployable level of performance, verify its robustness, and assess quality metrics such as interpretability and fairness.
- Model serving
Deploy the model to the production environment and expose an API interface.
- Model monitoring
Continuously collect and monitor the performance metrics of the model’s online service; detect data drift and assess whether the model needs retraining; optimize or retrain the model based on monitoring results.
6. MLOps Levels
6.1 Level 0: Manual Process
Applicable scenarios:
- The model does not need to be updated frequently
Characteristics:
- Relies on manual operations
- Script-driven interactive manual process: data analysis, data preparation, model training, and validation are entirely manual
- Machine learning separated from operations: data scientists are separated from engineers, and the trained model is handed off to the engineering team as an artifact
- Infrequent release iterations: new model versions are deployed only a few times a year
- No CI/CD: continuous integration and continuous deployment are neglected because changes are infrequent
- Only the trained model is deployed as a service, without the need for a complete machine learning platform
- Model predictions and operations are not tracked or logged
Problems that exist:
- The model may fail to adapt to dynamic changes in the environment or in the data
- Model quality needs to be actively monitored in the production environment
- The model needs to be retrained frequently with the latest data
- New implementations must be constantly tried in order to produce models
6.2 Level 1: Machine Learning Pipeline Automation
Applicable scenarios:
- Automatically training models with new data
- Continuously delivering model services
Characteristics:
- Fast experimentation: machine learning experiment steps are orchestrated, transformations run automatically, and iterations are fast
- Automatically training models with new data
- Smooth switching between different environments
- Modular code for components and pipelines: code is modularized and components are containerized to achieve reproducibility
- Continuous delivery of models: the pipeline automatically runs the model deployment steps
Requirements:
- Data and model validation: automated data validation and model validation steps
- Feature store: a centralized store that standardizes the definition, storage, and access of features.
- Metadata management: records information about each pipeline execution, enabling data and artifact lineage and reproducibility
- Machine learning pipeline triggers: automatically execute pipelines on a schedule or in response to triggers
6.3 Level 2: CI/CD Pipeline Automation
Applicable scenarios:
- The model needs to be trained quickly on new data and new business environments
- There are high demands on the speed and quality of model iteration
Characteristics:
- Source code control: manage the source code of pipelines and components
- Testing and build services: automatically build, test, and package pipelines and their components
- Deployment services: automatically deploy pipeline implementations to the target environment
- Model registry: manage different versions of models
- Feature store and metadata store: provide feature and metadata management
- Pipeline orchestration: provide pipeline orchestration capabilities
