TensorRT-LLM

TensorRT-LLM compiles and optimizes language models for NVIDIA GPUs, using techniques such as quantization, in-flight batching and paged attention. It aims to squeeze the best performance out of the hardware.

It is more complex to set up than alternatives like vLLM, but delivers excellent speed for those who stay in NVIDIA's ecosystem. It integrates with Triton Inference Server.

Best for teams with dedicated infrastructure engineers.