A token is a unit tightly bound to data. It can be used to measure how much corpus is needed to train a model, and also to measure the input and output length at inference time. 1. What a token is A token can be a whole word, a subword,

1. About FLOPs FLOPs (Floating Point Operations Per Second) refers to the number of floating-point operations performed per second. Specifically: A single floating-point addition, such as a + b, counts as one floating-point operation. A single floating-point multiplication, such as a * b, also counts as one floating-point operation. Other basic floating-point operations, such as division and square root, can also be counted as one floating-point operation.

1. Definition There are two completely different phases in LLM inference, and PD separation is The compute-intensive Prefill phase, LLM processes all users’ input and computes the corresponding KV Cache The memory-intensive Decode phase, sequentially producing tokens one by one, computing only a single token per memory access 2. Metrics 2.

This post comes mainly from an internal sharing session, and is also a summary of some recent work. 1. The Common Failure Handling Process The figure above shows a typical operations exception handling process. Along the timeline, there are the following key points in time: Failure occurs Failure detected Failure

This article is compiled from part of an internal sharing session I gave. 1. Core Elements of a Storage System 1.1 Security Credentials for object storage buckets, authorization when using storage PVCs, and control over access sources — these are all issues security needs to address. But they are also very easy to overlook, and when something goes wrong it is a big problem.