https://www.chenshaowen.com/en/tags/vllm/Recent content in vLLM on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sat, 20 Sep 2025 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/using-vllm-to-deploy-pd-disagg-app.htmlSat, 20 Sep 2025 00:00:00 +0000Sat, 20 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-vllm-to-deploy-pd-disagg-app.html1. Why Deploy LLM Applications with PD Disaggregation In the process of LLM inference, there are two serial stages: Process the entire input context and generate the KV Cache (Prefill stage) Incrementally generate new tokens (Decode stage) These two stages have different resource requirements. The Prefill stage has to compute微信公众号vLLMDeploymentPDDisaggregationApplicationhttps://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.htmlWed, 17 Sep 2025 00:00:00 +0000Wed, 17 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.html1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.微信公众号AILLMvLLMInferencelmcacheOperationshttps://www.chenshaowen.com/en/blog/using-vllm-benchmark-to-test-model-performance.htmlTue, 05 Aug 2025 00:00:00 +0000Tue, 05 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-vllm-benchmark-to-test-model-performance.htmlVLLM Benchmark is a tool provided by VLLM for testing model performance, supporting a variety of inference backends. This article mainly records some of the process of using VLLM Benchmark to test model performance. 1. Starting the Model Service 1 2 3 4 5 6 7 8 9 10 11 12 13 14 python -m vllm.微信公众号AIVLLMBenchmarkingModel PerformanceTestinghttps://www.chenshaowen.com/en/blog/use-vllm-for-inference.htmlSat, 18 Jan 2025 00:00:00 +0000Sat, 18 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-for-inference.html1. Environment Preparation Download Miniforge 1 wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Install Miniforge 1 bash Miniforge3-$(uname)-$(uname -m).sh 1 2 echo "export PATH=$HOME/miniforge3/bin:$PATH" >> ~/.bashrc source ~/.bashrc Create the environment 1 conda create -n vllm python=3.12 vLLM currently requires Python 3.9+. Activate the environment 1 conda activate vllm Install dependencies 1 conda install微信公众号vLLMAILLMInferenceOperationshttps://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.htmlThu, 16 Jan 2025 00:00:00 +0000Thu, 16 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.html1. Building the Image To make testing convenient, the model files are packaged into the image here. Download the model 1 2 3 4 git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat cd Qwen1.5-1.8B-Chat && git lfs pull rm -rf .git cd .. Write the Dockerfile 1 2 3 4 5 cat <<EOF > Dockerfile微信公众号vLLMInferenceAILLMOperations