In a PD disaggregated deployment, heterogeneous GPU models are often used to deploy the model across machines, which multiplies the cross-machine communication pressure. RDMA devices are usually brought in to accelerate kvcache transfer between nodes so as to achieve a lower FTTL. This post describes how to test eRDMA devices

1. Why Deploy LLM Applications with PD Disaggregation In the process of LLM inference, there are two serial stages: Process the entire input context and generate the KV Cache (Prefill stage) Incrementally generate new tokens (Decode stage) These two stages have different resource requirements. The Prefill stage has to compute