All Tools
P
LLMFreeOpen Source
POWERINFER
Fast local LLM serving on a consumer GPU
MIT
ABOUT
Full-precision LLM serving on a desktop GPU usually means swapping layers to CPU or shrinking the model. PowerInfer uses activation locality and sparse predictors to keep hot neurons on GPU and cold ones on CPU, so models far larger than VRAM still generate tokens quickly on one consumer card.
INTEGRATION GUIDE
1. Run sparse Llama and Mixtral-class models locally on a single RTX GPU without offloading the whole model
2. Serve ReLU-sparsified checkpoints faster than llama.cpp on the same desktop hardware
3. Prototype on-device inference for smartphones with the PowerInfer-2 stack
4. Convert Hugging Face sparse models to PowerInfer GGUF for local chat and coding assistants
TAGS
llminferencelocalgpucpusparse-modelsllama-cppopen-source