1. What Is Prefix Cache

In model inference scenarios, caching mechanisms are often used to improve throughput and performance. There are two common caching mechanisms:

• Key-Value Cache (KV Cache), which is oriented toward the internals of a single request, caching the intermediate computation results (Key and Value) of the Transformer model to avoid redundant computation.
• Prefix Cache, which is oriented toward multiple requests, exploiting the common prefix of prompts to avoid redundant computation.

The principle of Prefix Cache is to detect the common prefix of requests through structures such as hashing and radix trees, and reuse the previous computation results during the prefill stage to improve inference performance.

The figure above is a schematic of a Radix Tree (base tree) building the Prefix Cache across multiple requests.

As requests increase, the cache tree is constantly adjusted: it both adds new nodes and removes nodes according to a strategy.

The insight that Prefix Cache offers AI application developers is:

• Put the invariant part of the prompt first and the variable part last.
• When there are multiple prompts, try to keep them sharing the same prefix.

2. Starting the Environment

To be able to use the built-in benchmark tool directly, the vLLM OpenAI API Server image is used here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

nerdctl run -it \
        --security-opt apparmor=unconfined \
        --security-opt seccomp=unconfined \
        -p 8000:8000 \
        --gpus all \
        --ipc=host \
        --ulimit memlock=-1 \
        --ulimit stack=67108864 \
        --name vllm \
        --volume /data/models:/data/models \
        --entrypoint /bin/bash \
        vllm/vllm-openai:v0.10.1.1

3. Starting the Service

3.1 Inference Service

• Disable prefix cache

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

export CUDA_VISIBLE_DEVICES=7
python3 -m vllm.entrypoints.openai.api_server \
  --model /data/models/Qwen2.5-7B-Instruct \
  --served-model-name /data/models/Qwen2.5-7B-Instruct \
  --port 8000 \
  --gpu_memory_utilization 0.8 \
  --max-model-len 4096 \
  --max-seq-len-to-capture 8192 \
  --max-num-seqs 128 \
  --enforce-eager \
  --no-enable-prefix-caching

• Enable prefix cache

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

export CUDA_VISIBLE_DEVICES=7
python3 -m vllm.entrypoints.openai.api_server \
  --model /data/models/Qwen2.5-7B-Instruct \
  --served-model-name /data/models/Qwen2.5-7B-Instruct \
  --port 8000 \
  --gpu_memory_utilization 0.8 \
  --max-model-len 4096 \
  --max-seq-len-to-capture 8192 \
  --max-num-seqs 128 \
  --enforce-eager \
  --enable-prefix-caching

3.2 Starting the Client

• Enter the environment

1

nerdctl exec -it vllm /bin/bash

• Use a random dataset

1
2
3
4
5
6
7

vllm bench serve \
  --backend openai \
  --model /data/models/Qwen2.5-7B-Instruct \
  --dataset-name random \
  --random-input-len 1024 \
  --num-prompts 1024 \
  --request-rate 16

• Use the ShareGPT dataset

1
2
3
4
5
6
7
8

vllm bench serve \
  --backend openai \
  --model /data/models/Qwen2.5-7B-Instruct \
  --dataset-name sharegpt \
  --dataset-path /data/models/ShareGPT_V3_unfiltered_cleaned_split.json \
  --random-input-len 1024 \
  --num-prompts 1024 \
  --request-rate 16

4. Test Results

• Disable prefix cache

During the test, there is a small amount of GPU KV cache hit.

• Enable prefix cache

The device used is an NVIDIA A100-SXM4-80GB. From the test results, it can be seen that with Prefix Cache on the ShareGPT dataset, the Prefix Cache hit rate of the second test reached 96.0%, and P99 TTFT dropped from 138.34ms to 47.05ms — a very noticeable improvement.

This shows that when there is sufficient GPU memory, if a prompt can hit the cache, both TTFT and TPOT of inference will improve significantly.