SGLang

SGLang is an open-source engine for serving language models with high throughput. Features like RadixAttention reuse cached prefixes across requests, which speeds up workloads with shared prompts such as agents and chat.

It supports many popular open models and runs on NVIDIA and AMD GPUs. Several large AI companies use it in production.

Choose it if you host your own models and care about performance per GPU.