https://www.chenshaowen.com/en/tags/inference/Recent content in Inference on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Wed, 17 Sep 2025 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.htmlWed, 17 Sep 2025 00:00:00 +0000Wed, 17 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.html1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.微信公众号AILLMvLLMInferencelmcacheOperationshttps://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.htmlTue, 09 Sep 2025 00:00:00 +0000Tue, 09 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.html1. Installing the Driver Download the driver Go to https://www.nvidia.com/en-us/drivers/ and select the corresponding driver version to download. 1 wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.76.05/NVIDIA-Linux-x86_64-580.76.05.run Install the driver 1 bash NVIDIA-Linux-x86_64-580.76.05.run Check the GPUs 1 nvidia-smi 1 2 3 GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-92fcdc58-4754-73c7-af6c-56740936817d) GPU 1: NVIDIA GeForce RTX 5090 (UUID:微信公众号AIGPUNVIDIALLMInferencePerformance TestingOperationshttps://www.chenshaowen.com/en/blog/what-is-ray.htmlSun, 09 Feb 2025 10:00:00 +0000Sun, 09 Feb 2025 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-ray.html1. What Ray Is In 2016, UC Berkeley’s RISELab released a new distributed computing framework called Ray. In 2017, after the Ray paper was published, it drew broad attention across the industry; in China it was mainly Ant Group that adopted and contributed to Ray. In 2020, Ray released version微信公众号Distributed SystemsComputing FrameworkRayTrainingInferencehttps://www.chenshaowen.com/en/blog/use-vllm-for-inference.htmlSat, 18 Jan 2025 00:00:00 +0000Sat, 18 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-for-inference.html1. Environment Preparation Download Miniforge 1 wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Install Miniforge 1 bash Miniforge3-$(uname)-$(uname -m).sh 1 2 echo "export PATH=$HOME/miniforge3/bin:$PATH" >> ~/.bashrc source ~/.bashrc Create the environment 1 conda create -n vllm python=3.12 vLLM currently requires Python 3.9+. Activate the environment 1 conda activate vllm Install dependencies 1 conda install微信公众号vLLMAILLMInferenceOperationshttps://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.htmlThu, 16 Jan 2025 00:00:00 +0000Thu, 16 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.html1. Building the Image To make testing convenient, the model files are packaged into the image here. Download the model 1 2 3 4 git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat cd Qwen1.5-1.8B-Chat && git lfs pull rm -rf .git cd .. Write the Dockerfile 1 2 3 4 5 cat <<EOF > Dockerfile微信公众号vLLMInferenceAILLMOperationshttps://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.htmlTue, 06 Feb 2024 00:00:00 +0000Tue, 06 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.html1. What Is TensorRT TensorRT is a C++ library mainly used for high-performance inference acceleration on NVIDIA GPUs. It provides both a C++ API and a Python API for integration. The mainstream deep learning frameworks TensorRT supports are: Caffe, where TensorRT can read the prototxt format directly TensorFlow, where the微信公众号AITensorRTNVIDIAGPUInferenceLLMhttps://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.htmlSat, 03 Feb 2024 08:05:48 +0000Sat, 03 Feb 2024 08:05:48 +0000https://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.html1. Compiling Models with TensorRT-LLM 1.1 Introduction to TensorRT-LLM When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server. But this conversion process is not simple, and you often run into微信公众号AITritonTensorRTLLMInferenceGPUOperationsContainersLearninghttps://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.htmlSat, 16 Sep 2023 00:00:00 +0000Sat, 16 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.html1. Running Locally in Containers Start the LLM 1 docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -p 8000:8000 shaowenchen/chinese-alpaca-2-7b-gguf:Q2_K The API documentation is available at http://localhost:8000/docs, as shown below: Deploy a simple Chat UI Note that the OPENAI_API_HOST parameter must be set to your host machine’s IP address rather than localhost or 127.微信公众号CPULLMAIInferenceOperationshttps://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.htmlTue, 05 Sep 2023 00:00:00 +0000Tue, 05 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.html1. LLM Deployment Tool llama.cpp Research on large models is split into two parts: training and inference. The training process is essentially the process of finding model parameters that minimize the model’s loss function and optimize the inference results. Once training is complete, the model’s parameters are fixed, and at微信公众号AILLMToolsllama.cppInferenceDeployment