This page looks best with JavaScript enabled

Deploying PD-Disaggregated Applications with vLLM

 ·  ☕ 4 min read

1. Why Deploy LLM Applications with PD Disaggregation

In the process of LLM inference, there are two serial stages:

  1. Process the entire input context and generate the KV Cache (Prefill stage)
  2. Incrementally generate new tokens (Decode stage)

These two stages have different resource requirements. The Prefill stage has to compute a large amount of KV Cache, while the Decode stage has to read and store a large amount of KV Cache.

If the Prefill and Decode stages run simultaneously on one device, that device needs strong enough compute and large enough VRAM, and its price is correspondingly higher.

From a chip design perspective, the more compute cores there are, the larger the silicon area they occupy, the smaller the area left for memory interfaces, and the smaller the maximum memory capacity that can be supported.
Chip manufacturers, constrained by cost, yield, process, power consumption, and the international situation, have limited the area of a single chip, so the requirements that Prefill and Decode place on the chip are contradictory.

Since this contradiction cannot be resolved in the short term, the only option is to deploy the Prefill and Decode processes on different devices. Choose a chip with strong compute for Prefill, and a chip with large memory capacity for Decode.

Of course, if the model is small and the text is short, a single device may be able to provide enough compute and memory, and PD disaggregated deployment is unnecessary.

2. Request Traffic Analysis for a PD-Disaggregated Application

A PD-disaggregated application needs three services:

  1. API Proxy service, which receives user requests and forwards them to the P node and D node
  2. P node, responsible for the Prefill stage, generating the KV Cache
  3. D node, responsible for the Decode stage, reading and storing the KV Cache

During deployment, the P node and D node can have identical configurations, differing only in the compute device used. For example, the P node uses H200 and the D node uses H20. The API Proxy service uses only CPU, and only needs low-latency networking to the P node and D node.

The API Proxy service is the entry and exit point for traffic. A PD-disaggregated application processes as follows:

  1. The API Proxy sets max_tokens=1 and forwards the user request to the P node
  2. The P node processes the request, generates the KV Cache, and returns the first token to the API Proxy
  3. The API Proxy discards the P node’s response
  4. The API Proxy forwards the user request to the D node
  5. The D node processes the request, reads the KV Cache, and continues generating tokens
  6. The D node returns the generated tokens to the API Proxy

There are some interesting considerations here:

  1. Must P always come before D? Can we go straight to D, in case it hits the Cache? This leads to NVIDIA’s open-source dynamo project
  2. What are the ways for P and D to share the KV Cache? This leads to the different Connectors, which use remote storage, NIXL, and NCCL for high-speed transfer
  3. How do multiple P nodes and D nodes work together? This leads to the design of the Proxy service and how to improve the cache hit rate
  4. Tiering around the cache. On both P and D nodes, multi-level caches for VRAM, memory, local, and remote can be configured — how do we coordinate them
  5. Ultra-large-capacity storage pools. https://lmcache.ai/kv_cache_calculator.html can calculate that a long-text Cache block for an LLM is over 200 MB. How do we build a cache pool with ultra-large capacity
  6. KV Cache expiration and cleanup policies
  7. Load balancing and fault tolerance. How do multiple P and D nodes do load balancing, and how are node failures handled

In real production, all of these questions may need to be considered. This also brings new challenges and opportunities to AI Infra deployment components. Let’s first deploy a simple application to get a feel for PD disaggregation and relax a bit.

3. Deploying a PD-Disaggregated Application with vLLM

3.1 Connectors Supported by vLLM

A connector is the way P and D nodes share the KV Cache. vLLM currently supports 5 types of connectors:

  • SharedStorageConnector, shares KV Cache via a shared storage path
  • LMCacheConnectorV1, combines LMCache caching with NIXL to transfer KV Cache
  • NixlConnector, transfers KV Cache based on NIXL
  • P2pNcclConnector, uses NVIDIA NCCL to transfer KV Cache
  • MultiConnector, a combination of multiple connectors

Reference https://docs.vllm.ai/en/latest/features/disagg_prefill.html?h=prefill#why-disaggregated-prefilling

Here we use SharedStorageConnector for the demonstration.

3.2 Starting the vLLM Container Environment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

nerdctl run -it \
        --security-opt apparmor=unconfined \
        --security-opt seccomp=unconfined \
        -p 8000:8000 \
        -p 8100:8100 \
        -p 8200:8200 \
        --gpus all \
        --ipc=host \
        --ulimit memlock=-1 \
        --ulimit stack=67108864 \
        --name vllm \
        --volume /data/models:/data/models \
        --entrypoint /bin/bash \
        vllm/vllm-openai:v0.10.1.1

The remaining commands are all executed inside the container.

3.3 Deploying the P Node

1
2
3
4
5
6
7
8

export CUDA_VISIBLE_DEVICES=0
vllm serve /data/models/Qwen2.5-7B-Instruct \
    --port 8100 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.8 \
    --trust-remote-code \
    --kv-transfer-config \
        '{"kv_connector":"SharedStorageConnector","kv_role":"kv_producer","kv_connector_extra_config":{"shared_storage_path":"/data/models/pd_local_storage"}}'

Here kv_role is set to kv_producer, indicating that the P node is the producer of the KV Cache. shared_storage_path specifies the path of the shared storage; the P node stores the KV Cache it generates under this path.

  • Deploy the D node
1
2
3
4
5
6
7
8

export CUDA_VISIBLE_DEVICES=1
vllm serve /data/models/Qwen2.5-7B-Instruct \
    --port 8200 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.8 \
    --trust-remote-code \
    --kv-transfer-config \
        '{"kv_connector":"SharedStorageConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"shared_storage_path":"/data/models/pd_local_storage"}}'

You can also set the role of both nodes to kv_both, acting as both producer and consumer of the KV Cache.

3.4 Starting the AI Proxy Service

  • Download the Proxy service code
1
2
3

wget https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py -O /data/models/disagg_prefill_proxy_server.py
wget https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/disagg_benchmarks/rate_limiter.py -O /data/models/rate_limiter.py
wget https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/disagg_benchmarks/request_queue.py -O /data/models/request_queue.py
  • Install dependencies
1

python3 -m pip install --ignore-installed blinker quart -i https://pypi.tuna.tsinghua.edu.cn/simple
  • Start the Proxy service
1

python3 /data/models/disagg_prefill_proxy_server.py
  • Send a request
1
2
3
4
5
6
7
8

curl -X POST "http://127.0.0.1:8000/v1/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/data/models/Qwen2.5-7B-Instruct",
    "prompt": "介绍一下量子计算的基本原理",
    "temperature": 0.7,
    "max_tokens": 4000
  }'

1

{"id":"cmpl-b46522dbab4c4f9abba2f0e5ff0c9cd7","object":"text_completion","created":1758250381,"model":"/data/models/Qwen2.5-7B-Instruct","choices":[{"index":0,"text":" 量子计算是一种基于量子力学原理的计算方式，它利用量子比特（qubit）代替传统计算机中的二进制位（bit）。量子比特可以处于0和1的叠加态，这意味着一个量子比特可以同时表示0和1，这种特性称为“叠加”。此外，量子比特之间还可以发生纠缠，即两个或多个量子比特之间的状态可以相互关联，即使它们相隔很远，一个量子比特的状态改变也会立即影响到另一个量子比特的状态。\n\n这些特性使得量子计算机在某些特定问题上具有超越经典计算机的优势。例如，在因子分解、搜索算法等领域，量子计算机可以实现指数级加速。量子计算的核心在于如何设计和控制量子比特，以及如何进行量子门操作，以实现特定的量子算法。目前，量子计算仍处于研究和发展阶段，但其潜力巨大，有望在未来带来革命性的变革。","logprobs":null,"finish_reason":"stop","stop_reason":151643,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":8,"total_tokens":190,"completion_tokens":182,"prompt_tokens_details":null},"kv_transfer_params":null}
  • Check the D node logs
1

(EngineCore_0 pid=4206) INFO 09-18 19:53:35 [shared_storage_connector.py:265] External Cache Hit!
  • Check the shared storage path
1
2
3
4
5
6
7

ls /data/models/pd_local_storage/d41d8cd98f00b204e9800998ecf8427e/

model.layers.0.self_attn.attn.safetensors   model.layers.21.self_attn.attn.safetensors
model.layers.1.self_attn.attn.safetensors   model.layers.22.self_attn.attn.safetensors
model.layers.10.self_attn.attn.safetensors  model.layers.23.self_attn.attn.safetensors
model.layers.11.self_attn.attn.safetensors  model.layers.24.self_attn.attn.safetensors
...

4. Summary

This article covers the following:

  • For LLM inference with large models and long text, the Prefill and Decode stages have contradictory requirements for compute and memory, which can be resolved by PD-disaggregated deployment
  • From the request traffic analysis of a PD-disaggregated application, the P node generates the KV Cache with max_tokens=1 set and does not directly respond to the request
  • Using vLLM to deploy a PD-disaggregated application and experience the P node, D node, and Proxy service working together
微信公众号
WRITTEN BY
微信公众号

See Also