NVIDIA Dynamo
What is NVIDIA Dynamo?
NVIDIA Dynamo is open-source AI Model Serving & Inference software for multi-GPU and multi-node large language model (LLM) inference. It is the orchestration layer above inference engines: it does not replace SGLang, NVIDIA TensorRT-LLM, or vLLM. It coordinates those engines into one cluster that loads trained weights, runs prefill and decode, and returns tokens through an OpenAI-compatible HTTP API. A single model on a single GPU does not need Dynamo; the engine alone is enough.
Key Capabilities
- Disaggregated serving: Prefill (context) and decode (token generation) run on separate, independently scaled GPU pools so each phase can use different hardware and replica counts.
- KV-aware routing: The router sends a request to a worker whose key-value (KV) cache already overlaps the prompt, which avoids repeating prefill work.
- KV Block Manager: KV cache can spill from GPU memory to CPU RAM, local SSD, or remote/object storage so context length is not capped by on-device memory.
- Planner: An SLA-driven autoscaler profiles time-to-first-token and inter-token latency and resizes prefill/decode pools.
- Grove: A Kubernetes operator for topology-aware gang scheduling (including NVLink domains such as NVL72).
- AIConfigurator: Offline search over many serving topologies (parallelism, prefill/decode split) without burning a cluster to try each one.
- ModelExpress / NIXL: GPU-to-GPU weight streaming for faster replica cold-start.
- Frontends: Dynamo-native HTTP frontend, or a Kubernetes Gateway API Inference Extension path with an endpoint-picker plugin. Both expose OpenAI-compatible /v1/chat/completions.
- Workload types: LLM and reasoning models, multimodal encode/prefill/decode, and video-generation backends (FastVideo, SGLang Diffusion) in current 1.0-line docs.
Audience & Use Cases
- Audience: Platform and site reliability (SRE) teams and ML engineers who already run vLLM, TensorRT-LLM, or SGLang and need to scale past one node.
- Use Case: Production LLM serving on a GPU cluster; independently scaling prefill vs decode; KV reuse across a fleet; meeting latency SLOs without pinning every replica to full GPU memory.
Technical Specifications
- Source: github.com/ai-dynamo/dynamo. Apache-2.0. Core in Rust; Python bindings (ai-dynamo on PyPI).
- Runtimes: NGC containers (for example nvcr.io/nvidia/ai-dynamo/sglang-runtime, vllm-runtime, tensorrtllm-runtime); Kubernetes DynamoGraphDeploymentRequest CRDs; local --discovery-backend file for development.
- Discovery / messaging: Kubernetes-native for cluster deploy; optional etcd/NATS for non-K8s distributed modes. Inter-component traffic is TCP.
- Engines: SGLang, TensorRT-LLM, vLLM (not Triton as the default backend).
- Licensing: Open source. NVIDIA states production support and NIM packaging of Dynamo capabilities ship under NVIDIA AI Enterprise, which remains a separate suite.
Categories & Use Cases
Technical Details
| Mobile Application | No |
|---|
FAQs
What is NVIDIA Dynamo?
NVIDIA Dynamo is open-source AI Model Serving & Inference software for multi-GPU and multi-node large language model (LLM) inference. It is the orchestration layer above inference engines: it does not replace SGLang, NVIDIA TensorRT-LLM, or vLLM. It coordinates those engines into one cluster that loads trained weights, runs prefill and decode, and returns tokens through an OpenAI-compatible HTTP API. A single model on a single GPU does not need Dynamo; the engine alone is enough.