NVIDIA Dynamo · Developer intro

NVIDIA Dynamo

Open-source, distributed inference serving for the datacenter.

What is Dynamo

Engines optimize the GPU. Dynamo optimizes the system around them.

Open sourceRust + Python · Apache 2.0 · pip, containers, Kubernetes

Any enginevLLM · SGLang · TensorRT-LLM, and more

Modularscheduling, memory, data transfer as separate crates and wheels

Solves: disaggregation · config search · traffic spikes · fault tolerance beyond Kubernetes

What Dynamo focuses on

How Dynamo approaches performance, scale and resiliency

Performance

2×tokens/s · disagg vs agg

agg

disagg

Baseten · GLM-5.2

⇢

Route to the cacheKV events → the worker holding the prefix

⇄

Split prefill and decodeeach phase on its own GPUs

▤

Keep KV, don't recomputeoffload to host, disk, remote

Scalability

5×faster cold start

before

Dynamo

Photoroom · Snapshot · 220 s → 40 s

⚡

Start in secondsSnapshot + ModelExpress

◎

Scale to the SLAPlanner sizes prefill and decode separately

▦

Scale hierarchicallyGrove: gang, topology-aware

Resiliency

2.7×faster P95 TTFT in failover

before

Dynamo

KVCR-Guard · internal

⛨

Keep state aliveKV and weights outlive the engine

◉

Detect fastinhibition, discovery, canaries

↻

Keep streams alivemigrate requests with their tokens

Performance: Dynamo makes the engine do less work per token. The router sees exactly which KV blocks each worker holds (KV events) and weighs that against load; prefill and decode can run on separate pools; KV is offloaded to cheaper tiers instead of evicted. Proof (published): Baseten, GLM-5.2: "we observed 2x higher tokens per second on disaggregated inference" vs aggregated (head-to-head; hardware not stated).
Scalability: Planner autoscales prefill and decode separately against TTFT/ITL targets; Grove gang-schedules whole multi-node instances with topology awareness; Snapshot and ModelExpress make new workers serve in seconds. Proof (published): Photoroom cut GPU cold starts from ~220 s to 35–45 s with Dynamo Snapshot.
Resiliency: failures are routine at scale. Bad workers are detected in seconds (inhibition, discovery, canaries); in-flight streams migrate with their tokens; every router replica is active and rebuilds from worker events; KVCR-Guard and GMS keep KV and weights through engine crashes. Proof (internal benchmark): KVCR-Guard P95 TTFT during failover 26.5 s → 9.65 s.

Why Dynamo

Three reasons teams pick Dynamo

1

Production-grade performance

Routing, disagg and offloading speed up the engine; fault tolerance keeps it serving.

2

Best integration & interop

EnginesvLLMSGLangTRT-LLMTokenSpeedFastVideo

KV offloadLMCacheHiCacheFlexTensorKVCR

Networking & storage · NIXLUCXlibfabricGPUDirect StorageS3Azure Blob3FSMooncakePOSIX

Plug into the engines, caches and fabrics you already run.

3

Hardware neutral

NVIDIAGPU

IntelXPU

AMDGPU

GoogleTPU · via vLLM

AmazonTrainium · via vLLM

Run on the accelerators you have, without vendor lock-in.

Where Dynamo is today

In production at scale, built in the open

Customers

~100production customers

25%+serve on 1,000+ GPUsseveral on 10,000+ GPUs

25+ customers

v1.0 at GTC 2026 · v1.6 today · enterprise support since v1.4

GitHub activity

⭐ 8.4kstars · 1.75k forks

~175merged PRs / week

283external contributors

62% external

214external PRs merged · Oct 2026

33% of all

github.com/ai-dynamo/dynamo · Mar 2025 – Oct 21 2026

Where Dynamo is today · community

External contributions keep climbing

PRs external · Oct

Mar 2025Oct 2026

external contributors

Mar 2025Oct 2026

customer & partner blogs

Mar 2025Oct 2026

papers · deep dives + citations

Mar 2025Oct 2026

The Dynamo ecosystem

Use the whole stack, or pick the parts you need

Click a component for its slide.

Scheduling & scaling

Router: the right worker for every request

Sends each request to the worker that already holds its KV cache, weighed against live load, so cached prefixes are reused instead of recomputed.

1
Exact cache viewKV events from every engine, on every router replica
2
Admission controladmit, hold in class queues, or fail fast with 529
3
Customizableplug in your own filters, scorers and pickers

BasetenTTFT

2× faster

DigitalOceanTTFT

4.3× faster

Pinterestp99 TTFT

5.9× faster

Prime Intellectp90 ITL

1.7× faster

Results gray = before · green = with DynamoDeep dive: router.html →

Scheduling & scaling

Planner: capacity that follows the SLA

An autoscaler built for LLM serving: it watches live traffic and scales prefill and decode separately to hit your TTFT and ITL targets with the fewest GPUs.

1
Live traffic inrequest rate, prompt and output lengths as they arrive
2
Predict and sizeforecasts the next interval; sizes prefill and decode to TTFT / ITL
3
Scale up and downadds prefill or decode workers, removes idle ones, within the GPU budget

Alibaba Cloudwith Dynamo Planner

80% fewerSLA breaches

while using

5% fewerGPUs

Results gray = before · green = with DynamoDeep dive: planner-grove.html →

Scheduling & scaling

Grove: schedule whole inference systems

A Kubernetes API that describes a whole inference system — frontend, prefill, decode — as one custom resource. Unlike LeaderWorkerSet, which groups one leader with its workers, Grove gang-schedules, scales and orders every role together.

1
Scale at every levela role, a group, or the whole system
2
Hierarchical gang schedulingleader + workers start together or not at all
3
Topology-aware placementpack in one rack, spread replicas apart

Training

In production at multiple frontier labs training the newest models

Inference

In production for inference at multiple neoclouds

gray = before · green = with DynamoDeep dive: planner-grove.html →

Scheduling & scaling

Snapshot: warm workers in seconds

Checkpoint a warmed-up worker once, including GPU state, and restore it anywhere in seconds.

1
Capture a warm workerafter engine init, CRIU + cuda-checkpoint save CPU and GPU state
2
Restore from storageany node resumes the snapshot: seconds instead of a full cold start
3
Weights via ModelExpressrestored nodes pull weights GPU → GPU from a peer

PhotoroomGPU cold start

5× faster

Qwen3-Next-80Bto serving, + ModelExpress

13× faster

Published results gray = before · green = with DynamoDeep dive: mx-snapshot.html →

Memory management

KVCR: move KV to where it is needed

The router plans where KV should be; KVCR moves it across workers, host memory and storage, and keeps it through engine crashes.

1
Router plans the movehints say what to copy, move, prefetch or drop
2
KV across nodes and tiersallocate KV across DRAM and storage, peer to peer over NIXL
3
KVCR-Guarda sidecar process holds the KV pool and serves it if the engine fails

DeepSeek V4 Prop99 TTFT on burst · B200

6× faster

ProductionTTFT p50 · 22 B200 nodes

1.28× faster

Production customerKV cache reuse

+14%

KVCR-GuardP95 TTFT in failover

2.7× faster

Results gray = before · green = with DynamoDeep dive: kvcr.html →

Memory management

GMS: weights that outlive the engine

Model weights live in GPU memory owned by a service, so they outlive the engine process.

1
One copy of weightsowned by GMS, mapped read-only into engines
2
Shadow engineparked; takes over in seconds if the active engine crashes
3
Engines plug incustom PyTorch allocator on CUDA VMM

After an engine crashweights reloaded

0 GB

gpt-oss-120brestore with GMS + NVMe (PoC)

< 5 s

Key factsDeep dive: ft.html →

Memory management

FlexTensor: stream weights into HBM on demand

Streams model weights (tensors) from host DRAM into GPU HBM just before each layer runs, so models larger than HBM fit and run at full speed.

1
Wrap, don't rewriteplain PyTorch: wrap the nn.Module, no model or engine changes
2
Weights (tensors) in host DRAMpacked into pinned blocks, freeing HBM for KV and activations
3
Streamed just in timeprofiled prefetch hides the copy behind compute

LTX-2.3 outpaint · A100 80GBmax video length

+48.5% longer

LTX-2.3 outpaint · 8×A100 80GBmedian generation time

2.24× faster

Results gray = before · green = with DynamoDeep dive: flextensor.html →

Data transfer

NIXL: one API for every transfer

The inference transfer library: one API every engine (vLLM, SGLang, TRT-LLM) uses to move KV and weights over networks and to storage.

1
One async agentpoint to point, non-blocking; used by vLLM, SGLang and TRT-LLM
2
Pluggable backendsnetwork: UCX, libfabric, UCCL, GPUNetIO · storage: GDS, POSIX, 3FS, S3…
3
Every memory tierGPU, CPU, SSD and remote storage behind one API

VAST DataTTFT · KV from storage vs recompute

20× faster

CloudianP95 TTFT at 120K tokens · S3 over RDMA vs recompute

20× faster

Published results gray = before · green = with DynamoDeep-dive deck coming

Data transfer

ModelExpress: weights from peers, not storage

New workers get weights from GPUs that already have them, over NIXL. Storage is read once per cluster.

1
Read storage onceone seed per cluster, storage as fallback
2
Peer to peerGPU → GPU over NIXL / RDMA
3
Fan outevery new worker becomes a source

Prime IntellectRL weight update vs NCCL

22× faster

DeepSeek-V4-Proweight load from a peer

48× faster

DeepSeek-V4-ProE2E cold start time

4.6× faster

H Company62 GB GPU → GPU

1.4 s

Published results gray = before · green = with DynamoDeep dive: mx-snapshot.html →

Performance

AIPerf: benchmark any endpoint

Load-tests any inference endpoint with realistic traffic and reports the metrics that set your SLA.

1
Real workloadssynthetic, ShareGPT, production and agentic-trace replay
2
Realistic loadconcurrency, request rate, arrivals, multi-turn sessions
3
SLA-grade metricsTTFT, ITL, goodput at p50/p99, plus Pareto sweeps

SemiAnalysis InferenceX

Runs the AgentX agentic-coding benchmark on real Claude Code traces

AWS SageMaker AI

Benchmarking engine behind inference recommendations; AWS contributes upstream

Used by gray = before · green = with DynamoDeep dive: aiperf.html →

Performance

AISim: test deployments without GPUs

Simulates Dynamo and the engine on measured GPU timing: compare deployments before spending GPU hours.

1
Two layersDynamo router + planner over vLLM, SGLang or TRT-LLM
2
Predictreplay a trace, get every request's latency
3
Recommendfewest GPUs that meet the SLA

Config sweep on CPU onlyvs the same sweep on real GPUs

1000× faster

Self-service FPMonboard a new model on your own GPUs

weeks → ~1 hour

Key facts gray = before · green = with DynamoDeep dive: aisim.html →

Performance

AITune: the fastest backend for every module

No-code tuning for PyTorch: picks the fastest validated backend for every module in your model or pipeline.

1
Inspectfinds tunable modules in any PyTorch model or pipeline
2
Tune each moduletries TensorRT, Torch-TRT, Inductor, ONNX Runtime; keeps the fastest that matches outputs
3
Run or deployjust-in-time with one env var, or a saved .ait for Triton and Dynamo

~300 public models · mean speedup vs Torch eagerdiffusion · NLP · vision · ASR · recommenders, from Transformers, Diffusers, timm, MONAI, NeMo

1.85× faster

Internal benchmark gray = before · green = with DynamoDeep dive: aitune.html →

Putting it together

Use cases with their own deep dives

🔁

Get started

Try it, build on it, talk to us

# one-line install, then serve
pip install "ai-dynamo[vllm]"
python3 -m dynamo.frontend --router-mode kv &
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B

GitHubgithub.com/ai-dynamo/dynamo Docsdocs.nvidia.com/dynamo Slackai-dynamo.org/slack · every Dynamo dev is there More decksai-dynamo.org/slides

NVIDIA Dynamo

Engines optimize the GPU. Dynamo optimizes the system around them.

How Dynamo approaches performance, scale and resiliency

Performance

Scalability

Resiliency

Three reasons teams pick Dynamo

Production-grade performance

Best integration & interop

Hardware neutral

In production at scale, built in the open

External contributions keep climbing

Use the whole stack, or pick the parts you need

Router: the right worker for every request

Planner: capacity that follows the SLA

Grove: schedule whole inference systems

Snapshot: warm workers in seconds

KVCR: move KV to where it is needed

GMS: weights that outlive the engine

FlexTensor: stream weights into HBM on demand

NIXL: one API for every transfer

ModelExpress: weights from peers, not storage

AIPerf: benchmark any endpoint

AISim: test deployments without GPUs

AITune: the fastest backend for every module

Use cases with their own deep dives

RL rollouts

Agents

Omni

Try it, build on it, talk to us

Speaker notes