NVIDIA Dynamo · FlexTensor
FlexTensor
Run models bigger than your GPU. Weights stream in from host RAM, just in time.
PyTorch · no model editsv0.5.0 · Apache 2.0vLLM · integratedSGLang · coming soon
CUDA-event profilingadaptive strategyblock optimizationweight sharingcopyless pinningall modulesmulti-model supportweight compression
The problem
HBM is the scarce resource, not FLOPs
One 80 GB GPU
weights58 GB
activations14 GB
KV8 GB
host RAMweights waiting to stream in
Memory per GPU · HBM only
HBMhost DRAM share
HBM won’t catch the models.
The one rule that matters
Free when compute outlasts the copy
What one forward step processes
tokens / stepPer-layer timeline · 5 layers
bus speedArchitecture
Five automatic steps, then it just runs
Inside inference
Less GPU memory for weights, free until the knee
GPU memory · 80 GB
resident weightsrolling slotsfreed for activations / KV
weights kept on GPUUse it
One call for PyTorch, one worker for vLLM
Run with
max_gpu_mem_fractionNone = latency mode · float = memory modeinclude_patternswhich modules streamResults
Same GPU, bigger jobs, near-zero overhead
Status and roadmap
Shipped, in progress, next
Shipped · v0.1 → v0.5
- v0.5.0 shared host weights across processes (SHM), vLLM DP ≥ 2
- v0.4.0 vLLM worker v2 with CUDA graphs
- v0.3.0 torch.compile per offloaded unit
In progress
- AITune integration: auto-profile and stream inside AITune
- SGLang offloader integration
Planned
- Weight compression: decompress on the GPU (nvCOMP)
- Multi-model shared VRAM: GPU context switching
- MoE-aware selective prefetch