All Tools
M
LLMFreeOpen Source
MOONCAKE
Disaggregated KV cache serving for large-scale LLM inference
Apache-2.0
ABOUT
Prefill and decode fight for the same GPUs, and KV cache is recomputed or copied over slow paths when instances cannot share it. Mooncake splits serving around a distributed KV cache store and RDMA transfer engine so vLLM and SGLang clusters reuse cache across nodes and raise tokens per second without breaking latency SLOs.
INSTALL
pip install mooncake-transfer-engineINTEGRATION GUIDE
1. Share KV cache across vLLM instances with Mooncake Store for higher throughput
2. Move hidden states and weights over RDMA during disaggregated RL rollout and training
3. Back SGLang P2P weight updates for trillion-parameter models without huge offline storage
4. Deploy Kimi-style prefill/decode disaggregation on a multi-node GPU cluster
TAGS
llminferencekv-cacherdmavllmsglangservingopen-source