Local runtime · Server engine

TensorRT LLM

NVIDIA inference and serving engine for language, multimodal, and visual generation models, combining custom GPU kernels, a PyTorch-native runtime, and an optional ahead-of-time compiled TensorRT engine path behind one Python LLM API and an OpenAI-compatible server.

Maintainer

NVIDIA

Accelerators

NVIDIA CUDA · Multi-GPU

Model formats

Safetensors · PyTorch checkpoints · FP8 · AWQ · GPTQ

Source model

Open core

Licenses

Apache License 2.0 · BSD 3-Clause License · MIT License · Commercial or production-restricted terms

Boundary

Represents the TensorRT LLM library and its trtllm-serve, trtllm-bench, and trtllm-eval tools, operated on GPUs the user controls. It is not TensorRT, the general-purpose NVIDIA deep-learning inference library and engine-builder it can compile through, and it is not NVIDIA's managed offerings, which hold their own serving contract and belong to the inference-service collection. NVIDIA Dynamo and Triton Inference Server orchestrate this engine and are separate boundaries, as is NVIDIA Model Optimizer, the offline quantization toolkit whose checkpoints this runtime loads.

Open in the directory → Open official documentation ↗ Repository ↗

Editorial ratings appear in the directory beside the profile they belong to and are never compared across collections.