NVIDIA Dynamo · FlexTensor

FlexTensor

Run models bigger than your GPU. Weights stream in from host RAM, just in time.

PyTorch · no model editsv0.5.0 · Apache 2.0vLLM · integratedSGLang · coming soon
Host RAM · pinnedall model weights PCIe / C2C GPU HBM GPU never waits Extra memory for activations / KV cache
CUDA-event profilingadaptive strategyblock optimizationweight sharingcopyless pinningall modulesmulti-model supportweight compression
The problem

HBM is the scarce resource, not FLOPs

One 80 GB GPU
weights58 GB
activations14 GB
KV8 GB
host RAMweights waiting to stream in

Memory per GPU · HBM only
HBMhost DRAM share

HBM won’t catch the models.

The one rule that matters

Free when compute outlasts the copy

What one forward step processes
tokens / step
Per-layer timeline · 5 layers
bus speed
Architecture

Five automatic steps, then it just runs

Inside inference

Less GPU memory for weights, free until the knee

GPU memory · 80 GB
resident weightsrolling slotsfreed for activations / KV
weights kept on GPU
Use it

One call for PyTorch, one worker for vLLM

Run with
Results

Same GPU, bigger jobs, near-zero overhead

Status and roadmap

Shipped, in progress, next

Shipped · v0.1 → v0.5
  • v0.5.0 shared host weights across processes (SHM), vLLM DP ≥ 2
  • v0.4.0 vLLM worker v2 with CUDA graphs
  • v0.3.0 torch.compile per offloaded unit
Planned
  • Weight compression: decompress on the GPU (nvCOMP)
  • Multi-model shared VRAM: GPU context switching
  • MoE-aware selective prefetch

Speaker notes

← → step · N notes · F fullscreen