1. Compiling Models with TensorRT-LLM

1.1 Introduction to TensorRT-LLM

When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server.

But this conversion process is not simple, and you often run into various errors. It requires a certain grasp of model structure and platform operators, as well as the ability to convert and debug. The goal of TensorRT-LLM is to reduce the complexity of this process so that large models can more easily run on the TensorRT engine.

Note that TensorRT targets specific hardware: different GPU models require compiling different TensorRT format models. This is markedly different from the generality that the ONNX model format aims for.

At the same time, TensorRT-LLM does not support all GPU models; it only supports cards such as H100, L40S, A100, A30, and V100.

1.2 Configuring the Compilation Environment

1

docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --gpus device=0 -v $PWD:/app/tensorrt_llm/models -it --rm shaowenchen/nvidia-tensorrt-llm:v0.7.1 bash

--gpus device=0 means using the GPU card numbered 0, and here shaowenchen/nvidia-tensorrt-llm:v0.7.1 corresponds to the Release version of TensorRT-LLM v0.7.1.

Since building images yourself is very troublesome, here are a few images for optional versions:

• shaowenchen/nvidia-tensorrt-llm:v0.7.1
• shaowenchen/nvidia-tensorrt-llm:v0.7.0
• shaowenchen/nvidia-tensorrt-llm:v0.6.1

1.3 Compiling and Generating a TensorRT Format Model

In the container environment described above, run the command:

1
2
3
4
5
6
7
8
9

python examples/baichuan/build.py --model_version v2_7b \
                --model_dir ./models/Baichuan2-7B-Chat \
                --dtype float16 \
                --parallel_build \
                --use_inflight_batching \
                --enable_context_fmha \
                --use_gemm_plugin float16 \
                --use_gpt_attention_plugin float16 \
                --output_dir ./models/Baichuan2-7B-trt-engines

There are mainly three generated files:

• baichuan_float16_tp1_rank0.engine, the model computation graph file with embedded weights
• config.json, the file with detailed configuration information such as model structure, precision, and plugins
• model.cache, the compilation cache file, which can speed up subsequent compilation

1.4 Inference Test

1
2
3
4

python examples/run.py --input_text "世界上第二高的山峰是哪座？" \
                 --max_output_len=200 \
                 --tokenizer_dir ./models/Baichuan2-7B-Chat \
                 --engine_dir=./models/Baichuan2-7B-trt-engines

1
2
3
4

[02/03/2024-10:02:58] [TRT-LLM] [W] Found pynvml==11.4.1. Please use pynvml>=11.5.0 to get accurate memory usage
Input [Text 0]: "世界上第二高的山峰是哪座？"
Output [Text 0 Beam 0]: "
珠穆朗玛峰（Mount Everest）是地球上最高的山峰，海拔高度为8,848米（29,029英尺）。第二高的山峰是喀喇昆仑山脉的乔戈里峰（K2），海拔高度为8,611米（28,251英尺）。"

1.5 Verifying There Is No Serious Degradation

Model inference optimization can use techniques such as replacing operators, quantization, and pruning backpropagation, but there is one baseline that must be met: the model must not degrade much.

Only when the precision loss is within an acceptable range does model inference optimization make sense. The summarize.py provided by the TensorRT-LLM project can run some tests and score the model. rouge1, rouge2, and rougeLsum are metrics used to evaluate the quality of text generation, and they can be used to assess model inference quality.

• Get the Rouge metrics for the original format model

1

pip install datasets nltk rouge_score -i https://pypi.tuna.tsinghua.edu.cn/simple

Since optimum currently does not support the Baichuan model, you need to edit examples/summarize.py and comment out model.to_bettertransformer(). This problem has already been resolved in the latest TensorRT-LLM code; I am using the latest Release version (v0.7.1).

1
2
3
4

python examples/summarize.py --test_hf \
                    --hf_model_dir ./models/Baichuan2-7B-Chat \
                    --data_type fp16 \
                    --engine_dir ./models/Baichuan2-7B-trt-engines

Output:

1
2
3
4
5
6

[02/03/2024-10:21:45] [TRT-LLM] [I] Hugging Face (total latency: 31.27020287513733 sec)
[02/03/2024-10:21:45] [TRT-LLM] [I] HF beam 0 result
[02/03/2024-10:21:45] [TRT-LLM] [I]   rouge1 : 28.847385241217726
[02/03/2024-10:21:45] [TRT-LLM] [I]   rouge2 : 9.519352831698162
[02/03/2024-10:21:45] [TRT-LLM] [I]   rougeL : 20.85486489462602
[02/03/2024-10:21:45] [TRT-LLM] [I]   rougeLsum : 24.090111126907733

• Get the Rouge metrics for the TensorRT format model

1
2
3
4

python examples/summarize.py --test_trt_llm \
                    --hf_model_dir ./models/Baichuan2-7B-Chat \
                    --data_type fp16 \
                    --engine_dir ./models/Baichuan2-7B-trt-engines

Output:

1
2
3
4
5
6

[02/03/2024-10:23:16] [TRT-LLM] [I] TensorRT-LLM (total latency: 28.360705375671387 sec)
[02/03/2024-10:23:16] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[02/03/2024-10:23:16] [TRT-LLM] [I]   rouge1 : 26.557043897453102
[02/03/2024-10:23:16] [TRT-LLM] [I]   rouge2 : 8.28672928021811
[02/03/2024-10:23:16] [TRT-LLM] [I]   rougeL : 19.13639628365737
[02/03/2024-10:23:16] [TRT-LLM] [I]   rougeLsum : 22.0436013250798

For the model compiled with TensorRT-LLM, rougeLsum dropped from 24 to 22, which shows that capability does degrade somewhat. But as long as it is within an acceptable range, it is still usable, because inference speed improves considerably.

After this step, you can exit the container; inference is carried out in a different container.

2. Triton Server Configuration Notes

2.1 Introduction to Triton Server

Triton Server is an inference framework that gives users the ability to run inference at scale. Specifically:

• It supports multiple backends — tensorrt, onnxruntime, pytorch, python, vllm, tensorrtllm, and so on — and you can also customize a backend, needing only the corresponding shared library.
• It provides HTTP and GRPC interfaces externally.
• Batching capability: it supports inference in batches, and once Dynamic batching is enabled, multiple batches can be merged and inferred simultaneously, achieving higher throughput.
• Pipeline capability: a single Triton Server can run inference for multiple models at once, and models can be orchestrated with each other, supporting Concurrent Model Execution for pipelined parallel inference.
• Observability: it provides Metrics for real-time monitoring of various inference indicators.

The above is the architecture diagram of Triton Server. Simply put, Triton Server is an end (model) to end (application) inference framework that provides lifecycle process management around inference; once the model is configured, it can directly provide services to the application layer.

2.2 Triton Server Usage Configuration

In Triton community examples, there are usually four directories like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

.
├── ensemble
│   ├── 1
│   └── config.pbtxt
├── postprocessing
│   ├── 1
│   │   └── model.py
│   └── config.pbtxt
├── preprocessing
│   ├── 1
│   │   └── model.py
│   └── config.pbtxt
└── tensorrt_llm
    ├── 1
    └── config.pbtxt

9 directories, 6 files

For Triton Server, the directory format above actually defines four models: preprocessing, tensorrt_llm, postprocessing, and ensemble, except that ensemble is a composite model that defines multiple models to fuse them together.

The reason ensemble exists is that tensorrt_llm inference is not text2text. With the Pipeline capability of Triton Server, preprocessing tokenizes the input and postprocessing detokenizes the output, which together deliver end-to-end inference capability. Otherwise, when using TensorRT-LLM directly on the client side, you would still need to handle the bidirectional mapping between words and indices yourself.

The specific roles of these four models are as follows:

• preprocessing, used for preprocessing the input text, including tokenization and word vectorization, implementing preprocessing similar to text2vec.

• tensorrt_llm, used for vec2vec inference of the TensorRT format model

• postprocessing, used for post-processing the output text, including post-processing of the generated text such as alignment and truncation, implementing post-processing similar to vec2text.

• ensemble, which fuses the three models above to provide text2text inference

Each of the models defined above has a 1 directory representing version 1. Model files go in the version directory, and config.pbtxt goes in the model directory to describe inference parameters such as input, output, and version.

2.3 Control and Management of Model Loading

Triton Server controls how models are loaded through the --model-control-mode parameter. There are currently three loading modes:

• none, load all models in the directory
• explicit, load specified models in the directory, loading the specified models through the --load-model parameter
• poll, periodically poll and load all models in the directory, configuring the polling period through the --repository-poll-secs parameter

2.4 Control and Management of Model Versions

Triton Server provides a Version Policy in the model configuration file config.pbtxt, and each model can have multiple versions coexisting. By default it uses the model with version number 1. There are currently three version policies:

• Use all versions simultaneously

version_policy: { all: {}}

• Use only the most recent n versions

version_policy: { latest: { num_versions: 3}}

• Use only specified versions

version_policy: { specific: { versions: [1, 3, 5]}}

3. Using TensorRT-LLM in Triton Server

3.1 Cloning the Configuration Files

The configuration related to the examples in this article has been organized into a repository on GitHub. After copying the model to the specified directory, you can run inference directly.

1

git clone https://github.com/shaowenchen/demo

3.2 Organizing the Inference Directory

• Copy the TensorRT format model

1

cp Baichuan2-7B-trt-engines/* modelops/triton-tensorrtllm/Baichuan2-7B-Chat/tensorrt_llm/1/

• Copy the source model

1

cp -r Baichuan2-7B-Chat modelops/triton-tensorrtllm/downloads

At this point the file directory structure is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

tree modelops/triton-tensorrtllm

modelops/triton-tensorrtllm
├── Baichuan2-7B-Chat
│   ├── end_to_end_grpc_client.py
│   ├── ensemble
│   │   ├── 1
│   │   └── config.pbtxt
│   ├── postprocessing
│   │   ├── 1
│   │   │   ├── model.py
│   │   │   └── __pycache__
│   │   │       └── model.cpython-310.pyc
│   │   └── config.pbtxt
│   ├── preprocessing
│   │   ├── 1
│   │   │   ├── model.py
│   │   │   └── __pycache__
│   │   │       └── model.cpython-310.pyc
│   │   └── config.pbtxt
│   └── tensorrt_llm
│       ├── 1
│       │   ├── baichuan_float16_tp1_rank0.engine
│       │   ├── config.json
│       │   └── model.cache
│       └── config.pbtxt
└── downloads
    └── Baichuan2-7B-Chat
        ├── Baichuan2 模型社区许可协议.pdf
        ├── Community License for Baichuan2 Model.pdf
        ├── config.json
        ├── configuration_baichuan.py
        ├── generation_config.json
        ├── generation_utils.py
        ├── modeling_baichuan.py
        ├── pytorch_model.bin
        ├── quantizer.py
        ├── README.md
        ├── special_tokens_map.json
        ├── tokenization_baichuan.py
        ├── tokenizer_config.json
        └── tokenizer.model

13 directories, 26 files

3.3 Starting the Inference Service

1
2
3
4
5
6

docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --gpus device=0 --rm -p 38000:8000 -p 38001:8001 -p 38002:8002 \
    -v $PWD/modelops/triton-tensorrtllm:/models \
    shaowenchen/nvidia-triton-trt-llm:v0.7.1 \
    tritonserver --model-repository=/models/Baichuan2-7B-Chat \
    --disable-auto-complete-config \
    --backend-config=python,shm-region-prefix-name=prefix0_:

If multiple triton servers are running on one machine, you need to use shm-region-prefix-name=prefix0_ to distinguish the shared memory prefix. For details, see https://github.com/triton-inference-server/server/issues/4145 .

Startup log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

I0129 10:27:31.658112 1 server.cc:619]
+-------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend     | Path                                                            | Config                                                                                                                                                                                              |
+-------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| python      | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","shm-region-prefix-name":"prefix0_:","default-max-batch-size":"4"}} |
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}}                                      |
+-------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0129 10:27:31.658192 1 server.cc:662]
+----------------+---------+--------+
| Model          | Version | Status |
+----------------+---------+--------+
| ensemble       | 1       | READY  |
| postprocessing | 1       | READY  |
| preprocessing  | 1       | READY  |
| tensorrt_llm   | 1       | READY  |
+----------------+---------+--------+
...
I0129 10:27:31.745587 1 grpc_server.cc:2513] Started GRPCInferenceService at 0.0.0.0:8001
I0129 10:27:31.745810 1 http_server.cc:4497] Started HTTPService at 0.0.0.0:8000
I0129 10:27:31.787129 1 http_server.cc:270] Started Metrics Service at 0.0.0.0:8002

Once all four models are in the READY state, inference can proceed normally.

• View model configuration parameters

1
2
3

curl localhost:38000/v2/models/ensemble/config

{"name":"ensemble","platform":"ensemble","backend":"","version_policy":{"latest":{"num_versions":1}},"max_batch_size":32,"input":[{"name":"text_input","data_type":"TYPE_STRING",...

You can view the model’s inference parameters. If auto-complete-config is used, this interface can be used to export the model inference parameters automatically generated by Triton Server, for modification and debugging.

• Check whether Triton is running normally

1
2
3
4
5

curl -v localhost:38000/v2/health/ready

< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain

3.4 Client Invocation

• Install dependencies

1

pip install tritonclient[grpc] -i https://pypi.tuna.tsinghua.edu.cn/simple

The performance of the Triton GRPC interface is significantly higher than that of the HTTP interface, and inside the container I could not find an example for the HTTP interface either, so I just used GRPC here.

• Inference test

1

wget https://raw.githubusercontent.com/shaowenchen/demo/master/triton-tensorrtllm/Baichuan2-7B-Chat/end_to_end_grpc_client.py

1
2
3
4

python3 ./end_to_end_grpc_client.py -u 127.0.0.1:38001 -p "世界上第三高的山峰是哪座？" -S -o 128


珠穆朗玛峰（Mount Everest）是世界上最高的山峰，海拔高度为8,848米（29,029英尺）。在世界上，珠穆朗玛峰之后，第二高的山峰是喀喇昆仑山脉的乔戈里峰（K2，又称K2峰），海拔高度为8,611米（28,251英尺）。第三高的山峰是喜马拉雅山脉的坎钦隆加峰（Kangchenjunga），海拔高度为8,586米（28,169英尺）。</s>

3.5 Viewing Metrics

Triton Server already provides inference metrics, listening on port 8002. In the example in this article, that is port 38002.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

curl -v localhost:38002/metrics

nv_inference_request_success{model="ensemble",version="1"} 1
nv_inference_request_success{model="tensorrt_llm",version="1"} 1
nv_inference_request_success{model="preprocessing",version="1"} 1
nv_inference_request_success{model="postprocessing",version="1"} 128
# HELP nv_inference_request_failure Number of failed inference requests, all batch sizes
# TYPE nv_inference_request_failure counter
nv_inference_request_failure{model="ensemble",version="1"} 0
nv_inference_request_failure{model="tensorrt_llm",version="1"} 0
nv_inference_request_failure{model="preprocessing",version="1"} 0
nv_inference_request_failure{model="postprocessing",version="1"} 0

You can import the dashboard https://grafana.com/grafana/dashboards/18737-triton-inference-server/ in Grafana to view the metrics, as shown below:

4. Summary

This article is mainly a record of the process of learning to use TensorRT and Triton Server for inference. The main content is as follows:

• TensorRT is a more efficient model inference engine for Nvidia GPU hardware
• TensorRT-LLM lets large models use the TensorRT engine faster
• Triton Server is an end-to-end inference framework that supports most model frameworks and helps users quickly implement inference services at scale
• An example of using TensorRT-LLM for inference under Triton Server

5. References

• https://mmdeploy.readthedocs.io/zh-cn/latest/tutorial/03_pytorch2onnx.html
• https://docs.nvidia.com/deeplearning/tensorrt/container-release-notes/running.html#running
• https://github.com/NVIDIA/TensorRT-LLM
• https://github.com/triton-inference-server/triton-tensorrtllm
• https://zhuanlan.zhihu.com/p/663748373