All Tools
N
LLMFreeOpen Source
NVIDIA DYNAMO
Datacenter-scale orchestration above LLM inference engines
Apache-2.0
ABOUT
A single inference engine cannot schedule prefill versus decode, route across a fleet, or share KV cache at datacenter scale. NVIDIA Dynamo sits above vLLM, SGLang, and TensorRT-LLM to coordinate disaggregated serving, intelligent routing, multi-tier KV caching, and autoscaling so LLM and multimodal workloads use GPUs more efficiently.
INSTALL
uv pip install --prerelease=allow "ai-dynamo[sglang]"INTEGRATION GUIDE
1. Orchestrate multi-node vLLM or SGLang clusters with disaggregated prefill and decode
2. Route requests and share KV cache across GPU pools to raise tokens per second
3. Serve reasoning, multimodal, and video generation models with automatic scaling
4. Run TensorRT-LLM workers under one control plane instead of ad-hoc engine scripts
TAGS
llminferencenvidiaservingkv-cachevllmsglangtensorrt-llmopen-source