Using vLLM for Model Inference

1. Environment Preparation

Download Miniforge

1
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"

Install Miniforge

1
bash Miniforge3-$(uname)-$(uname -m).sh

1
2
echo "export PATH=$HOME/miniforge3/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

Create the environment

1
conda create -n vllm python=3.12

vLLM currently requires Python 3.9+.

Activate the environment

1
conda activate vllm

Install dependencies

1
conda install vllm

2. Inference Testing

2.1 Model Preparation

Set the model URL

Overseas

1
export MODEL_REPO=https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat

China

1
export MODEL_REPO=https://hf-mirror.com/Qwen/Qwen1.5-1.8B-Chat

Download the model

1
nerdctl run --rm -v ./:/runtime shaowenchen/git lfs clone $MODEL_REPO

2.2 Offline Batched Inference

This inference mode suits offline scenarios, such as batch inference.

Specify the card index

1
export CUDA_VISIBLE_DEVICES=1

Run inference with generate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from vllm import LLM, SamplingParams

model = LLM(
    model="Qwen1.5-1.8B-Chat",
    task="generate",
    enforce_eager=True,
    dtype="half",
)
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
datas = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
outputs = model.generate(datas, sampling_params)
[output.outputs[0].text for output in outputs]

Here the task value defaults to auto; the available values are auto, generate, embedding, embed, classify, score, reward.

2.3 API Server For Online Serving

This inference mode suits online scenarios, such as chatbots, and provides a request interface compatible with the OpenAI API.

Specify the card index

1
export CUDA_VISIBLE_DEVICES=1

Start the service

1
2
3
4
5
6
7
python3 -m vllm.entrypoints.openai.api_server \
        --model Qwen1.5-1.8B-Chat \
        --served-model-name  Qwen1.5-1.8B-Chat \
        --trust-remote-code \
        --dtype=half \
        --host 0.0.0.0 \
        --port 30000

Since the test device is a V100, half precision is used here.

Inspect the model

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
curl http://127.0.0.1:30000/v1/models | jq .
{
    "object": "list",
    "data": [
        {
        "id": "Qwen1.5-1.8B-Chat",
        "object": "model",
        "owned_by": "vllm",
        ...
        }
    ]
}

Test inference

1
2
3
4
5
6
7
8
9
curl http://127.0.0.1:30000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
         "model": "Qwen1.5-1.8B-Chat",
         "messages": [
             {"role": "user", "content": "什么是大模型"}
         ],
         "max_tokens": 1024
     }'

Inspect the Metrics

1
curl http://127.0.0.1:30000/metrics

Multi-GPU inference

If a single card cannot load the model, you can shard it across multiple cards via tensor parallelism.

1
export CUDA_VISIBLE_DEVICES=1,3

1
2
3
4
5
6
7
8
python3 -m vllm.entrypoints.openai.api_server \
        --model Qwen1.5-1.8B-Chat \
        --served-model-name  Qwen1.5-1.8B-Chat \
        --tensor-parallel-size 2 \
        --trust-remote-code \
        --dtype=half \
        --host 0.0.0.0 \
        --port 30000

3. Vectorized Embedding

In RAG applications, data needs to be vectorized for retrieval and recall. Let’s test the vectorization capability here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from vllm import LLM
model = LLM(
    model="Qwen1.5-1.8B-Chat",
    task="embed",
    enforce_eager=True,
    dtype="half",
)
datas = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
outputs = model.embed(datas)
[output.prompt_token_ids for output in outputs]

Using embed lets you vectorize data in batches:

1
2
3
4
[[9707, 11, 847, 829, 374],
 [785, 4767, 315, 279, 3639, 4180, 374],
 [785, 6722, 315, 9625, 374],
 [785, 3853, 315, 15235, 374]]

4. LoRA Support

4.1 Model Preparation

Set the model URL

Overseas

1
export MODEL_REPO=https://huggingface.co/Speeeed/Qwen1.5-1.8B-Chat-wsc-lora

China

1
export MODEL_REPO=https://hf-mirror.com/Speeeed/Qwen1.5-1.8B-Chat-wsc-lora

Download the model

In addition to the base model from before, download one more LoRA model.

1
nerdctl run --rm -v ./:/runtime shaowenchen/git lfs clone $MODEL_REPO

4.2 Loading LoRA

Specify the card index

1
export CUDA_VISIBLE_DEVICES=1

Start the service

1
2
3
4
5
6
7
8
9
python3 -m vllm.entrypoints.openai.api_server \
        --model Qwen1.5-1.8B-Chat \
        --served-model-name  Qwen1.5-1.8B-Chat \
        --trust-remote-code \
        --dtype=half \
        --host 0.0.0.0 \
        --port 30000 \
        --enable-lora \
        --lora-modules wsc-lora=Qwen1.5-1.8B-Chat-wsc-lora

Inspect the available models

1
2
3
4
5
6
curl http://127.0.0.1:30000/v1/models | jq . | grep id

      "id": "Qwen1.5-1.8B-Chat",
          "id": "modelperm-9d467dd1a39b4c81bf412255a9cc8729",
      "id": "wsc-lora",
          "id": "modelperm-137c35f3ad7d427a9d754e8ff87e953f",

Test inference

1
2
3
4
5
6
7
8
9
curl http://127.0.0.1:30000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
         "model": "wsc-lora",
         "messages": [
             {"role": "user", "content": "什么是大模型"}
         ],
         "max_tokens": 1024
     }'

4.3 Dynamic LoRA

vLLM allows loading and unloading LoRA dynamically through the API, without restarting the service.

Set environment variables

1
2
export CUDA_VISIBLE_DEVICES=1
export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Start the service

1
2
3
4
5
6
7
8
python3 -m vllm.entrypoints.openai.api_server \
        --model Qwen1.5-1.8B-Chat \
        --served-model-name  Qwen1.5-1.8B-Chat \
        --trust-remote-code \
        --dtype=half \
        --host 0.0.0.0 \
        --port 30000 \
        --enable-lora

At this point curl http://127.0.0.1:30000/v1/models shows only one model, Qwen1.5-1.8B-Chat.

Load LoRA

1
2
3
4
5
6
curl -X POST http://localhost:30000/v1/load_lora_adapter \
-H "Content-Type: application/json" \
-d '{
    "lora_name": "wsc-lora",
    "lora_path": "Qwen1.5-1.8B-Chat-wsc-lora"
}'

A return of Success: LoRA adapter 'wsc-lora' added successfully. means the load succeeded. At this point the wsc-lora model appears in ‘/v1/models’.

Unload LoRA

1
2
3
4
5
curl -X POST http://localhost:30000/v1/unload_lora_adapter \
-H "Content-Type: application/json" \
-d '{
    "lora_name": "wsc-lora"
}'

A return of Success: LoRA adapter 'wsc-lora' removed successfully. means the unload succeeded.

5. vLLm vs Triton Inference Server

5.1 Performance Metrics

Metric	vLLM	Triton Inference Server
Throughput	vLLM achieves high throughput, especially in multi-user scenarios, thanks to its efficient GPU memory utilization.	Triton shows strong throughput when paired with certain backends. However, it usually cannot reach the peak performance that vLLM demonstrates.
Latency	vLLM has low latency and performs better when handling longer outputs.	Triton shows low latency, although it is relatively higher than vLLM’s, mainly due to the lack of the specialized optimizations found in vLLM.
TTFT	vLLM is optimized for large language models and can generate the initial token faster, thus reducing TTFT.	Triton also has low TTFT, but it may vary with different configurations, which indirectly affects TTFT.

5.2 Features

Feature	vLLM	Triton Inference Server
Supported models	vLLM supports most open-source LLMs, mixture-of-experts and multimodal models such as LLaVA.	Triton supports a variety of frameworks and backends, allowing deployment of various model types, including LLMs.
Quantization options	vLLM supports AWQ, GPTQ, Marlin (combining GPTQ, AWQ and FP8), INT8 (W8A8), FP8 (W8A8), AQLM, bitsandbytes, DeepSpeedFP and GGUF.	Triton supports a variety of frameworks and backends and can accommodate a broad range of quantization techniques, including those vLLM supports.
LoRA adapters	vLLM has robust LoRA support, allowing adapters to be loaded and unloaded dynamically at runtime. It offers flexible model switching.	Triton supports LoRA through custom backends and can manage multiple model versions simultaneously.
Batching capability	vLLM supports continuous batching, where new requests are dynamically added to the existing batch.	Triton has dynamic batching algorithms that combine individual requests into optimized batches to improve throughput.
Streaming support	vLLM provides built-in streaming output support for real-time interaction with the LLM.	Triton supports streaming inference through gRPC and HTTP/REST APIs, suitable for real-time applications.
API compatibility	vLLM provides an HTTP server implementing OpenAI’s Completions and Chat APIs.	Triton provides RESTful API and gRPC endpoints, supporting various client integrations.
Community and support	vLLM has an active GitHub and a dedicated Discord community. The vLLM team encourages participation through regular meetups.	Triton gets community support through NVIDIA forums and GitHub repositories.
Documentation	vLLM documentation covers installation, configuration, and advanced usage such as model quantization and distributed inference, with tutorials and examples.	Triton has extensive documentation, but users are often overwhelmed by the complexity of its features and configuration.

5.3 Ease of Use

Ease of use	vLLM	Triton Inference Server
Installation	vLLM can be installed easily via pip and Docker.	Triton can be installed via Docker, but the process can be fairly complex due to the various configuration requirements and dependencies of different model frameworks.

5.4 GPU Parallelism

Aspect	vLLM	Triton Inference Server
GPU parallelism	vLLM excels at maximizing GPU parallelism through its innovative tensor and pipeline parallelism.	Triton supports multi-GPU and multi-node deployment using custom backends. It has model ensemble capabilities that can be used for pipeline parallelism.

5.5 Hardware Compatibility and Cloud Deployment Options

Aspect	vLLM	Triton Inference Server
Hardware compatibility	vLLM supports a wide range of options, including Nvidia CUDA GPUs, AMD ROCm GPUs, AWS Neuron, CPU, OpenVINO, TPU and XPU.	Triton is compatible with Nvidia CUDA GPUs and Intel CPUs.
Cloud deployment options	vLLM is designed to be cloud-agnostic, offering flexibility to deploy across various cloud providers.	Triton can be deployed on any cloud platform and integrates with Kubernetes to enable scalable microservice architectures.

Source https://www.inferless.com/learn/vllm-vs-triton-inference-server-choosing-the-best-inference-library-for-large-language-models

6. Integration with Triton Inference Server

Triton Server is an inference framework that provides the following capabilities:

Model orchestration, deploying multiple models simultaneously in a single Triton Server instance and assembling business logic in one pass. For example, downloading, processing and inferring an image.
Support for multiple backends, such as TensorRT, ONNX Runtime, PyTorch, etc.
Dynamic batching, which adjusts the batch size dynamically based on the input length of requests to improve inference efficiency. However, Triton Server’s batching mechanism is mainly aimed at deep learning models with fixed input length, whereas vLLM targets large models with variable-length input; see https://www.anyscale.com/blog/continuous-batching-llm-inference

The integration of vLLM with Triton Server is provided by https://github.com/triton-inference-server/vllm_backend, and vLLM is one of the backends that Triton Server supports.

Documentation https://docs.vllm.ai/en/latest/
Blog https://blog.vllm.ai/
Performance testing https://perf.vllm.ai/
roadmap http://roadmap.vllm.ai/

8. Summary

This post mainly tested vLLM’s inference capabilities, including single-GPU, multi-GPU, vectorized Embedding, LoRA support, and dynamic LoRA.

It also put together a comparison of vLLM and Triton Inference Server, covering performance metrics, features, ease of use, GPU parallelism, hardware compatibility and cloud deployment options.

Compared with the earlier test of Triton Inference Server, vLLM is easier to use, which brings great convenience to deployment.