BentoML

BentoML is an open-source framework for packaging trained machine learning models into production-ready services. It takes a model built in any framework, PyTorch, TensorFlow, scikit-learn, XGBoost, or a Hugging Face pipeline, and wraps it in a standardized "Bento" that bundles the model, inference code, and dependencies into a single deployable unit. From there teams can build a Docker container, deploy to Kubernetes, or push to BentoCloud, the company's managed platform for running inference at scale.

It's aimed at ML engineers and platform teams who need to turn a model into a working API without hand-rolling a serving layer. BentoML handles adaptive batching, async request handling, GPU scheduling, and multi-model pipelines where several models or preprocessing steps run together in one service. It has become a common piece of infrastructure for teams building LLM and generative AI applications, with support for serving large language models alongside more traditional tabular and vision models.

What sets it apart from a bare Flask or FastAPI wrapper is the amount of production concerns it takes off a team's plate: versioning models and their dependencies together, generating reproducible builds, and giving a consistent deployment path from a laptop to a cluster. The open-source core is free and widely used as the deployment layer underneath larger MLOps stacks, while BentoCloud offers a paid, managed alternative for teams that don't want to run their own Kubernetes infrastructure.