1. Testing the Original Model Start the environment 1 2 3 4 5 6 7 8 9 10 11 12 nerdctl run -it \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --gpus all \ --ipc=host \ -p 8000:8000 \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --name vllm \ --volume /data/models:/data/models \

1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.

1. What Is Prefix Cache In model inference scenarios, caching mechanisms are often used to improve throughput and performance. There are two common caching mechanisms: Key-Value Cache (KV Cache), which is oriented toward the internals of a single request, caching the intermediate computation results (Key and Value) of the Transformer model to avoid redundant computation.

1. Installing the Driver Download the driver Go to https://www.nvidia.com/en-us/drivers/ and select the corresponding driver version to download. 1 wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.76.05/NVIDIA-Linux-x86_64-580.76.05.run Install the driver 1 bash NVIDIA-Linux-x86_64-580.76.05.run Check the GPUs 1 nvidia-smi 1 2 3 GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-92fcdc58-4754-73c7-af6c-56740936817d) GPU 1: NVIDIA GeForce RTX 5090 (UUID:

1. What Is Model Quantization Model quantization is the process of converting the weights and activations of a high-precision model (usually 32-bit floating point FP32 or 16-bit floating point FP16) into a low-precision model (such as 8-bit integer INT8). The value range of FP32 is -3.4*10^38 to 3.4*10^38, with 4 billion values.