Text Generation Inference

Text Generation Inference, or TGI, is the serving toolkit Hugging Face built to power its Inference Endpoints. It supports continuous batching, quantization, streaming and tensor parallelism for popular open models.

You run it in a Docker container and talk to it through a simple HTTP API. It is stable and well documented, though vLLM has become the more common default in recent years.

A fine choice if you already use Hugging Face tooling.