NVIDIA TensorRT

NVIDIA TensorRT is a deep-learning inference optimisation SDK and ecosystem developed by NVIDIA that compiles trained neural networks and large language models into highly optimised runtime engines for NVIDIA GPUs. First released in 2017, it now spans the TensorRT compiler, the open-source TensorRT-LLM library, the TensorRT Model Optimizer, TensorRT for RTX and the cloud-based TensorRT Cloud service.^[1]

History

TensorRT grew out of NVIDIA's cuDNN library as a compiler and runtime for convolutional networks in data centres, releasing in 2017 with layer fusion and kernel autotuning as its core tricks.^[1] The large-language-model era changed the problem: transformer inference is dominated by memory bandwidth and attention kernels rather than convolutions, so NVIDIA released TensorRT-LLM publicly in October 2023 as a purpose-built optimisation layer for LLMs.^[3] The library moved fully open source on GitHub in March 2024^[4] and reached version 1.0 in September 2025, adding PyTorch-native model authorship, a modular Python runtime and a stable production API, alongside what NVIDIA called a record-setting 8× inference speed-up.^[5]

Key Concepts and Technology

TensorRT works by optimising at build time for one specific GPU: the model is parsed, quantised (reduced to lower numeric precision such as INT8, FP8 or FP4), its layers and tensors are fused into fewer kernel launches, and candidate kernels are benchmarked on the actual hardware to pick the fastest — NVIDIA claims up to a 36× speed-up over CPU-only platforms.^[1]

TensorRT-LLM extends this to language models with a Python API for defining model architectures and adds the optimisations that matter at scale: FP8 and NVFP4 quantisation, speculative decoding techniques such as EAGLE-3 and multi-token prediction, disaggregated prefill and decode serving, wide expert parallelism for mixture-of-experts models, and KV-cache reuse.^[2][4] Compiled engines are typically served through NVIDIA Triton Inference Server or distributed as NIM containers.^[1]

Applications and Impact

TensorRT sits underneath much of the deployed LLM economy: NVIDIA has used it to publish world-record inference results for models such as DeepSeek-R1 on Blackwell hardware^[4], cloud providers use it inside managed inference endpoints, and developers use it on desktop RTX GPUs and Jetson edge devices. Its practical significance is economic rather than scientific — inference, not training, is the recurring cost of every AI product, and an optimisation stack that raises tokens-per-second-per-dollar directly changes what a service costs to run.

>See Also

• NVIDIA
• Inference
• Model Serving
• Quantisation
• NVIDIA Blackwell
• YTL AI Cloud