1. Testing the Original Model Start the environment 1 2 3 4 5 6 7 8 9 10 11 12 nerdctl run -it \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --gpus all \ --ipc=host \ -p 8000:8000 \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --name vllm \ --volume /data/models:/data/models \

1. Why Deploy LLM Applications with PD Disaggregation In the process of LLM inference, there are two serial stages: Process the entire input context and generate the KV Cache (Prefill stage) Incrementally generate new tokens (Decode stage) These two stages have different resource requirements. The Prefill stage has to compute

1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.

1. What Is Prefix Cache In model inference scenarios, caching mechanisms are often used to improve throughput and performance. There are two common caching mechanisms: Key-Value Cache (KV Cache), which is oriented toward the internals of a single request, caching the intermediate computation results (Key and Value) of the Transformer model to avoid redundant computation.

1. Installing the Driver Download the driver Go to https://www.nvidia.com/en-us/drivers/ and select the corresponding driver version to download. 1 wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.76.05/NVIDIA-Linux-x86_64-580.76.05.run Install the driver 1 bash NVIDIA-Linux-x86_64-580.76.05.run Check the GPUs 1 nvidia-smi 1 2 3 GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-92fcdc58-4754-73c7-af6c-56740936817d) GPU 1: NVIDIA GeForce RTX 5090 (UUID: