llama.cpp

llama.cpp is an open-source inference engine that runs large language models efficiently on consumer hardware. Its quantized GGUF format shrinks models so they fit in memory on laptops, phones and small servers.

Many popular tools, including Ollama and LM Studio, are built on top of it. It exposes an OpenAI-compatible server, making it easy to swap into existing apps.

Power users like the control it offers, though it is more hands-on than a packaged app.