1. Definition
There are two completely different phases in LLM inference, and PD separation is
- The compute-intensive Prefill phase,
LLM processes all users’ input and computes the corresponding KV Cache
- The memory-intensive Decode phase,
sequentially producing tokens one by one, computing only a single token per memory access
2. Metrics
2.1 prefill performance evaluation metrics
TTFT (Time To First Token), the time taken to generate the 1st token
P90 TTFT SLO = 0.4s, meaning our requirement for this system is: the TTFT of 90% of requests must be <= 0.4
2.2 decode performance evaluation metrics
TPOT (Time Per Output Token), the time taken to produce each response token
P90 TPOT SLO = 0.04s, meaning our requirement for this system is that the TPOT of 90% of requests must be <= 0.04s
3. Advantages Brought by PD Separation
In the context of long context, the prefill and decode phases have very imbalanced demands on compute and memory.
- Make full use of device resources
prefill uses high-compute GPUs, decode uses low-compute, large-memory GPUs
- Optimize separately, improving both TTFT and TPOT metrics at once
the prefill phase should limit the Batch Size, the decode phase should increase the Batch Size
4. batching strategy
- prefill phase
Because the prefill phase is compute-intensive, as batch size increases compute becomes the bottleneck and the throughput growth trend flattens out.
- decode phase
Because the decode phase is bandwidth- and memory-intensive, as batch size increases the throughput growth trend becomes more and more pronounced.
