All Tools
D
LLMFreeOpen Source
DEEPSPEED-MII
High-throughput LLM inference powered by DeepSpeed
Apache-2.0
ABOUT
Serving large LLMs needs continuous batching and KV-cache tricks that training stacks do not provide. DeepSpeed-MII wraps DeepSpeed-Inference so teams deploy Llama or Mixtral with a pipeline or gRPC server instead of hand-tuning those kernels.
INSTALL
pip install deepspeed-miiINTEGRATION GUIDE
1. Serve Llama or Mixtral with continuous batching and blocked KV-cache
2. Run a persistent gRPC or REST endpoint for multi-client generation
3. Deploy a non-persistent Python pipeline for batch text generation
4. Compare DeepSpeed FastGen throughput against other LLM servers
TAGS
pythonllminferenceservingdeepspeedbatchingllama