https://www.chenshaowen.com/en/tags/tensorrt/Recent content in TensorRT on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Tue, 06 Feb 2024 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.htmlTue, 06 Feb 2024 00:00:00 +0000Tue, 06 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.html1. What Is TensorRT TensorRT is a C++ library mainly used for high-performance inference acceleration on NVIDIA GPUs. It provides both a C++ API and a Python API for integration. The mainstream deep learning frameworks TensorRT supports are: Caffe, where TensorRT can read the prototxt format directly TensorFlow, where the微信公众号AITensorRTNVIDIAGPUInferenceLLMhttps://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.htmlSat, 03 Feb 2024 08:05:48 +0000Sat, 03 Feb 2024 08:05:48 +0000https://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.html1. Compiling Models with TensorRT-LLM 1.1 Introduction to TensorRT-LLM When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server. But this conversion process is not simple, and you often run into微信公众号AITritonTensorRTLLMInferenceGPUOperationsContainersLearning