A token is a unit tightly bound to data. It can be used to measure how much corpus is needed to train a model, and also to measure the input and output length at inference time. 1. What a token is A token can be a whole word, a subword,

1. About FLOPs FLOPs (Floating Point Operations Per Second) refers to the number of floating-point operations performed per second. Specifically: A single floating-point addition, such as a + b, counts as one floating-point operation. A single floating-point multiplication, such as a * b, also counts as one floating-point operation. Other basic floating-point operations, such as division and square root, can also be counted as one floating-point operation.

1. Definition There are two completely different phases in LLM inference, and PD separation is The compute-intensive Prefill phase, LLM processes all users’ input and computes the corresponding KV Cache The memory-intensive Decode phase, sequentially producing tokens one by one, computing only a single token per memory access 2. Metrics 2.

1. What Is MLOps MLOps is short for Machine Learning Operations, and it describes the standardization and engineering of the entire lifecycle around model development. MLOps includes the following key steps: Data management: storing, accessing, cleaning, and transforming data Model development: algorithm development and model construction Model training and tuning: training models on data, adjusting hyperparameters to optimize the model, and fine-tuning models Model evaluation: testing a model’s accuracy, generalization ability, and performance metrics Model deployment: deploying the model to the target environment and converting the model Model monitoring: monitoring the model’s behavior, performance degradation, data drift, and robustness Another similar concept is ModelOps.

1. Keywords Machine Learning (ML) The technology of automatically acquiring knowledge from data. Neural Network (NN) A model that imitates the structure and learning mechanism of biological neural networks; one of the branches of machine learning. The structure of a neural network consists of an input layer, hidden layers, and an output layer.