All Tools
T
LLMFreeOpen Source
TABBYAPI
Run local LLMs with an OpenAI-compatible API
AGPL-3.0
ABOUT
Running open-source LLMs locally typically requires complex setup, model format conversion, and custom client code. TabbyAPI provides a drop-in OpenAI API replacement that works with any OpenAI-compatible client. It supports the ExLlamaV2 engine for fast inference, various quantization formats (GPTQ, EXL2, FP16), custom model configurations, and multi-user access with API key management. Deployment is a single Docker command, and it works on consumer GPUs.
INTEGRATION GUIDE
1. Deploy a local OpenAI-compatible endpoint for testing LLM integrations without cloud API costs
2. Run quantized LLMs on consumer GPUs for private inference of sensitive data
3. Set up a multi-user inference server for a team with per-user API keys and rate limits
4. Experiment with different model quantizations (GPTQ, EXL2, FP16) to find the best speed-quality tradeoff
5. Build a local RAG pipeline with embedding models and LLMs all running through one API endpoint
TAGS
llminferencelocal-aiopenai-compatibleapi-serverquantizationself-hosted