NVIDIA Dynamo · FlexTensor

FlexTensor

Run models bigger than your GPU. Weights stream in from host RAM, just in time.

PyTorch · no model editsv0.5.0 · Apache 2.0

The problem

HBM is the scarce resource, not FLOPs

One 80 GB GPU

weights58 GB

activations14 GB

KV8 GB

host RAMweights waiting to stream in

Memory per GPU · HBM only

HBMhost DRAM share

HBM won’t catch the models.

Right side: per-GPU share of HBM and system DRAM in reference systems: DGX H100 (8×80 GB HBM, 2 TB DRAM), DGX B200 (1,440 GB HBM, 4 TB DRAM), GB200 NVL72 and GB300 NVL72 (Grace LPDDR5X, ~17 TB per rack of 72 GPUs). With FlexTensor the DRAM share becomes usable weight capacity, at the speed the bus allows.
"Beyond GPU Limits" deck, "Why it matters?": HBM goes 80 → 192 → 288 → 288 GB from H100 to B200, B300 and Rubin. The scarce thing is resident HBM, not FLOPs. Weights, activations and KV fight for the same HBM.
The split on the left is illustrative, not a measurement. Toggle to see the weights move to host RAM and the space they free.
Compared with other weight streaming (vLLM offload, HF Accelerate, ComfyUI, hand-written): FlexTensor wraps the nn.Module with no model edit, packs weights into allocation blocks, prefetches from a measured profile, and re-plans on new hardware. Alternatives today: framework CPU offload (vLLM offload, HF Accelerate hooks, ComfyUI heuristics), hand-written streaming, or more GPUs. Quantization is complementary: FlexTensor keeps model size and precision unchanged.

The one rule that matters

Free when compute outlasts the copy

What one forward step processes

tokens / step

Per-layer timeline · 5 layers

bus speed

"The one rule that matters": free offload [GB] = compute time [s] × bus speed [GB/s]. A layer's weights can stream for free if they arrive before the previous layer finishes computing.
Compute per layer grows with the tokens processed in one forward step; copy time grows with layer size ÷ bus speed. Both scale with layer size, so the break-even is roughly GPU throughput ÷ bus bandwidth tokens per step (BF16): about 6k tokens on PCIe Gen5 for an H100-class GPU. That is why video diffusion (tens of thousands of latent tokens per step) streams for free, long LLM prefill is close, and batch-1 decode (one token per step) does not.
Toy model, illustrative: 0.9 GB BF16 layer, ~400 TFLOPS effective. Video preset ≈ Wan2.2 81 frames at 832×480 (~32k latent tokens). Real numbers come from FlexTensor's profiler on your GPU and bus.

Architecture

Five automatic steps, then it just runs

Inside inference

Less GPU memory for weights, free until the knee

GPU memory · 80 GB

resident weightsrolling slotsfreed for activations / KV

weights kept on GPU

Use it

One call for PyTorch, one worker for vLLM

Run with

max_gpu_mem_fractionNone = latency mode · float = memory mode

include_patternswhich modules stream

📦 github.com/ai-dynamo/flextensorpip install flextensor · examples: Wan2.2, LTX-2.3, Cosmos, vLLM → 📖 ai-dynamo.org/flextensorconfiguration · phases · troubleshooting →

Results

Same GPU, bigger jobs, near-zero overhead

Wan2.2-T2V-A14B (README, public): peak allocated memory 78.2 → 25.9 GB (−67%), median 483.8 s vs 479.4 s native, under 1% overhead, excluding discovery and profiling. RTX PRO 6000 Blackwell 96 GB; 81 frames, 832×480, 40 steps, 5 runs. On an RTX 5090 32 GB, native does not fit; with FlexTensor it completes at 25.9 GB.
LTX-2.3 outpaint (README, public): A100 80 GB max video 825 frames (34.4 s) → 1,225 frames (51.0 s), +48.5% longer. A100 40 GB: native OOM → 497 frames (20.7 s). 1280×704 at 24 fps, one worker.
Cosmos-Transfer2.5 (README, public): reference needs ~65 GB. RTX 5090 32 GB: native OOM → completes at 30.5 GB peak. A100 40 GB: native OOM → 38.2 GB peak. LPIPS ≈ 0.03 vs reference.
Customer (internal, not named on the slide): a speech customer on RTX PRO 6000, 71% VRAM saved, +0.6% overhead, 3× model density.
Not shown on purpose: the 207 s → 92 s (2.24×) LTX number compares 1-GPU vs 8-GPU context parallelism, not FlexTensor vs native.

Status and roadmap

Shipped, in progress, next

Shipped · v0.1 → v0.5

v0.5.0 shared host weights across processes (SHM), vLLM DP ≥ 2
v0.4.0 vLLM worker v2 with CUDA graphs
v0.3.0 torch.compile per offloaded unit

In progress

AITune integration: auto-profile and stream inside AITune
SGLang offloader integration

Planned

Weight compression: decompress on the GPU (nvCOMP)
Multi-model shared VRAM: GPU context switching
MoE-aware selective prefetch