NVIDIA Dynamo · RL post-training

Dynamo in RL – roadmap

H2 ’26
01Rollout performance

Runtime, Router, AISim: routing, resilience, simulation

02Weight transfer & cold start

ModelExpress, NIXL, NCCLM2N: delta weights, fast refit

03Kubernetes deployment

Planner, Grove, Agent Substrate: one CRD for train + infer

1 / 19

RL Post-Training

Significant portion of RL is driven by inference and has similar optimization problems.

70–90%
of RL wall-clock time is inference
Dynamo · sandboxes, rollout, weight sync, Kubernetes
RL framework · trainer
Harness · tools & reward
2 / 19

RL Post-Training

Significant portion of RL is driven by inference and has similar optimization problems.

Improving Rollout Performance

Goal: increase goodput/GPU and reduce e2e rollout latency!
  • Smarter load-balancing (inter + intra-cluster)
  • Admission control to keep GPUs below capacity so RL steps stay fast and steady
  • Strong resiliency to faults during large-scale runs
  • Profiling/simulation to debug bottlenecks
Runtime, Router, AISim
3 / 19

KV- and session-aware routing: reuse the prefix, skip the re-prefill

Every request resends the whole prefix. On a worker that already caches it, only the new tokens need prefill.

Workload
Router
Incoming requests
waiting for the first request
Router KV-aware
Prefill work, in blocks
Round-robin
0
KV-aware
0
reused from this worker's KVnew tokensprefilled again
~1/Nchance the next turn hits the same worker with round-robin (N workers)
85–97%cache hit on repeat calls to the same worker (agentic coding sessions)
+10%tok/s/GPU from KV routing alone · Kimi-K2.5, agentic trace, 24× GB200
4× lowerp50 TTFT with a custom agent-aware router built on Dynamo’s routing APIs
Published numbers are from agentic serving, not RL training runs. Demo block counts are illustrative.
4 / 19
02 · Global routingSimulation

Global routing: one global, KV-aware router instead of a static split

One RL step = 240 rollouts across 4 datacenters. Training waits for the last one. Each row is a datacenter over time.
Green = GPUs working, gray = idle and waiting. Press Run and compare the gray.

t = 0
100 concurrent multi-turn sessions · square = worker, shade = busy slots, number = queue · APAC = older GPUs
5 / 19

How the global router works

Next rollout turn prompt prefix Global router picks the best datacenter CACHE HITLOAD DC A✓ DC B DC C tiny cache summary + loadKV itself never leaves the DC send turn here DC A✓ cached + free GPUscached prefixLocal KV-aware routerGPUs · darker = busier DC Blittle cachecached prefixLocal KV-aware routerGPUs · darker = busier DC Ctoo busycached prefixLocal KV-aware routerGPUs · darker = busier 2 1 3 1Each DC shares a tiny cache summary + load 2Router weighs cache hit vs. load 3Local router picks the GPU in that DC
6 / 19

Admission control: the router decides admit, defer, or reject

Rollout batch arrives all at once Dynamo router Admission gate plugin decides per request busy limits · priority · deadlines Admit Defer Priority queue waits for free capacity Reject fast 429 → retry later GPUs stay below capacity GPU 0GPU 1GPU 2GPU 3
How it helps RLour hypotheses
capto GPUswait
Cap the burst

A whole batch lands at once. Bound the queue so engines don’t thrash KV or preempt.

↑ priorityslowest
Stragglers first

Put the slowest rollouts in a higher tier to shorten the RL step tail.

GPUpoolevaljumps aheadevalnew rollouts waitweight sync
Share GPUs across workloads

Rollouts and evals share one GPU pool. Evals jump ahead of a rollout batch; new rollouts wait while weights sync.

7 / 19
01 · Fault tolerance · fills Prime slide 06Simulation

Fault tolerance: a worker dies, the rollout keeps going

AHistory lives outside the worker

sandbox proxy every tokenlogged GPUcache

If the GPU dies, the session’s history survives in the proxy.

BSessions move, not restart

reroute one session’s progress kept: all earlier turnsredo 1 turn

Sent to the least-loaded healthy worker. Only the current turn is redone.

CReplacement catches up fast

Full weights (2 TB) cold start ~25 min Checkpoint + Δ replay ~5 min then sessions rebalance onto it

Loads a checkpoint plus small Δ updates, not the full 2 TB model.

t = 0
48 trajectories × 1–5 h · 8 workers · same batch in both lanes · illustrative
8 / 19

Profiling + simulation with AISim: find and fix rollout bottlenecks

1Capture the live runJoin request traces with RL records by session IDTrainer Rollout RL recordsrequest trace+2Profile the timelinePerfetto view: where does rollout time go?long tool call → slow stepprefilldecodetool call3Simulate with AISimvalidate the pick on real GPUs
GPUs
Routing
KV cache
RL step time
–
GPU-hours / step
–
Try it: change one setting at a time
Find the real bottleneck

Split rollout time into queueing, prefill/decode, tool calls and weight updates.

Test changes without GPUs

Replay a real rollout trace to compare workers, routing, KV cache and planner settings.

Right-size the rollout fleet

Pick the setup that shortens RL steps, then confirm it on real GPUs.

ExperimentalServing side only: not trainer, rewards or convergence. Results are directional until calibrated against a live run.
9 / 19

RL Post-Training

Significant portion of RL is driven by inference and has similar optimization problems.

Faster Weight Transfers & Cold Start

Goal: decrease weight refit and reduce idle workers!
  • P2P weight transfers <10 sec
  • Resilience to faults/elasticity during large-scale runs
  • Moving samples back to the trainer
  • Scaling up workers quickly
ModelExpress, NIXL, NCCLM2N
10 / 19

ModelExpress: refit in seconds, new workers in minutes

Trainer FSDP / Megatron shard 0shard 1shard 2shard 3 Rollout vLLM / SGLang TP 0TP 1TP 2TP 3 FAST REFIT NIXL RDMA GPU → GPU · each rank pulls its own slices FAST REFITRESILIENCE XOR Δ only changed bits, >99% smaller pull Δ only any DC, any new worker Object storage base v40 Δ v41 Δ v42 every version kept durably FAST REFIT NCCL M2N collective reshard open PR v42v42v42v42 Three ways to ship each new weight version to rollouts
Warm refit Fast refit
1.5–1.8 s
✓ well under the <10 s goal
Qwen3-30B-A3B on GB200 · 4 FSDP GPUs → TP4/EP4 rollout (prime-rl)
New worker startup · DeepSeek-V4-Pro Fast scale-up
Runningworker Newworker weights kernels storage fallback →
Cold start from storage
8m 01s
P2P weights only
7m 00s
P2P weights + kernels
1m 44s
Copying compiled kernels saves the most time
✓
P2P weight refit <10 s1.5–1.8 s warm refit
✓
Scale up workers quicklynew worker 8m01s → 1m44s
✓
Resilience & elasticityevery version durable in object storage
○
Samples back to the trainerroadmap: experience transfer
11 / 19

ModelExpress: what RL frameworks get

Fast refit
expertsrank
MoE-aware pulls

Each rank pulls only its TP/EP slices

#704 merged
boundreuse
Warm, bounded memory

Reuses buffers and plans; bounded GPU install

#735 merged · #749 open
FP32BF16FP16→
Precision control

Per-tensor FP32 / BF16 / FP16 on the wire

PR #747
FP8+ FP8 KV
Quantized peer refit

Quantized vLLM rollouts + FP8 KV cache

PR #778
Resilience & elasticity
v40v41v42immutable + lease
Versioned updates

Immutable versions: every rollout knows its policy

MX main
1peer2trainer3storage
Source fallback

Peer → trainer → object storage

MX main
Integrity checks

Optional digests verify every refit

prime-rl runs
Scale up workers quickly
peernew
P2P from a running peer

New workers copy weights + kernels from a serving peer, not storage

8m01s → 1m44s
NVFP4bytes as-is
Quantized P2P load

FP8 / NVFP4 copied as-is between same-family GPUs

supported
Not covered yet ◐Packed FP8/FP4 trainer wire ✗Cross-vendor quantized CUDA ↔ XPU ○Samples → trainer roadmap
12 / 19
03 · Delta weight transfer · fills Prime slide 05 “path TBC”Simulation

Delta weight sync: one trainer, rollouts on three continents

ATrainer publishes Δ

version N ⊕ N+1 (bit-by-bit XOR) 0 0 0 1 0 0 0 0 0 1 0 0 0 0 full weights 2 TB Δ after zstd ~20 GB · >99% smaller

Mostly zeros → >99% smaller

→

BVersioned in object storage

ckpt ΔΔΔΔ … full checkpoint + chain of deltas

One upload per step

→

CPeers fan out in-region

one pull across the WAN one region W0 W1W2W3 NIXL P2P over RDMA

Peers copy over NIXL, not the WAN

→

DApply at a safe point

rollout generation policy Npolicy N+1 3–4 s pause

Swap weights, clear KV cache

Same update, two ways. Trainer makes a new step every 2 min (1T model).
❗Demo values are illustrative
BF16 weights · 3 remote regions · analytic model, illustrative
13 / 19

RL Post-Training

Significant portion of RL is driven by inference and has similar optimization problems.

Improving Kubernetes Deployment

Goal: increase sandboxes/CPU and goodput/GPU!
  • Deploy training/inference workloads optimally with single CRD API
  • Fleet-level coordination of hardware
  • Oversubscription of Vera CPUs to maximize sandboxes
Planner, Grove, Agent Substrate*
14 / 19

Planner: right-size rollout GPUs as load changes

An LLM-aware autoscaler: scales on queues, KV cache use and latency targets, not CPU.

Rollout load vs. replicas rollout loadreplicasGPU budget RL step 1 startsRL step 2 starts burst → scale up in seconds long tail → scale down Global Planner: one GPU budget, many pools max 16 GPUs before decode queues build → shift GPUs after prefill pooldecode pooleval pool
Why it matters for RL
Follows the RL step

Load-based scaling (~5 s) catches the burst at step start; prediction (~180 s) sets a capacity floor.

One budget, shifted where needed

Global Planner caps total GPUs and moves capacity between prefill, decode and eval pools.

Safe to try first

Advisory mode recommends replica counts without changing the deployment.

Status: shipped in Dynamo · targets: throughput (default), latency, load, SLA · works with vLLM, SGLang, TensorRT-LLM · scale-down does not drain in-flight requests yet: set replica floors.
15 / 19

Grove: one Kubernetes API for the whole rollout fleet

Describe router, prefill and decode as one custom resource; Grove schedules, places, orders and scales it.

PodCliqueSet · whole rollout system in one spec Router / frontendpodpodpod PodClique (one role) one NVLink domainPrefill engineGANGleaderworkerworkerworker① workers ready → ② leader startsone NVLink domainDecode engineGANGleaderworkerworkerworker① workers ready → ② leader starts PodCliqueScalingGroup: scales and schedules as one unit add a replica = add a whole engine, never half of one Gang schedulingall-or-nothing Topology-awarekeep NVLink traffic local Startup orderingright init sequence Rolling updatesmulti-level autoscaling
Why it matters for RL
No half-started engines

Multi-node rollout engines are gang-scheduled all-or-nothing, so GPUs never sit idle waiting for missing pods.

Placed for speed

Topology-aware placement keeps each engine inside one NVLink domain for wide-EP and disaggregated serving.

Update without stopping

Rolling updates and multi-level autoscaling; next: Grove Jobs for rollout and training workers.

Status: open source (ai-dynamo/grove) · shipped: gang scheduling, multi-level autoscaling, startup ordering, rolling updates, topology-aware scheduling · 2026: resource-optimized rolling updates, topology spread, auto topology detection.
16 / 19

Agent Substrate: many more sandboxes per CPU

Google's open-source runtime suspends idle sandboxes and resumes them on any warm worker in under a second.

Router: request → its actor Actors (sandboxes) most are idle, waiting onthe model or a tool activesuspended Warm worker pool worker 1worker 2worker 3worker 4worker 5worker 6 assign Snapshots RAM + files ofevery suspended actor suspend resume <1 s 30x+ actorsper worker pool (demo) <500 msresume, isolated by gVisor
Why it matters for RL
More sandboxes per CPU

Rollout sandboxes sit idle while the model generates. Suspend them and multiplex many onto a few warm workers.

State kept across turns

RAM and files are snapshotted, so a long multi-turn rollout resumes exactly where it paused.

Fast and isolated

Sub-second resume; gVisor or microVM isolation for model-written code.

Status: Google open source (agent-substrate/substrate), early development · GKE: evaluation now, production by allowlist · CPU sandboxes only (no GPU passthrough) · open network connections are not kept across suspend.
17 / 19

Roadmap to Dec ’26

01 · Rollout performance

Serving and routing control surface

  • Pass-through engine APIs for framework integration
  • ThunderAgent + KV hints for KV transfer in long-running rollouts
  • NEWPartial rollouts: cancel, pause and continue a trajectory without losing progress
02 · Weight transfer & data plane

Model Express / NIXL: weights, artifacts, versions

  • Fast weight sync via NIXL / NCCLM2N, delta weights, resharding, cold start
  • NEWRollout data plane: routed experts and logprobs over multiple transports (vLLM AuxOutput connector, in design review)
  • NEWModel version provenance: served version carried through request, response and artifacts; version-aware KV cache and capacity
03 · Kubernetes

K8s, Planner, autoscaling, and fault tolerance

  • Shift fixed GPU budget and requests between pools
  • Long-horizon rollout worker recovery via Shadow Failover & Snapshot
  • Grove Jobs + rolling updates for rollout & training workers
  • CPU sandboxes via Agent Substrate
NEWFlagship recipes
Quick validationSmall model, single node: try Dynamo for RL in minutes
Quantization-aware RLMXFP8 / NVFP4 on Blackwell for shorter time-to-intelligence
FW integrations:MilesSkyRLRelaxReefand more
18 / 19
Backup

KV retention across weight updates

One long rollout: a turn, a long tool call, then new weights (v → v+1) before the next turn.
No option wins on all three; it depends on how much the lab trusts KV from the old policy.

turnlong tool callweight syncpolicy v → v+1No re-prefillafter syncFrees GPUin tool callKV matchesnew policyFlush KVdefaultdecodetool call · KV idle on GPUre-prefill alldecode✗✗✓Retain KVused in productiondecodetool call · KV held on GPUreuse v KVdecode✓✗✗Offload + P2P KVto investigatedecodeKV → CPU / storageP2P to any workerdecode✓✓✗
Why it helps RL
No re-prefill after each sync

Long agent rollouts carry tens of thousands of tokens. Reusing KV skips recomputing that history every time the weights change.

GPU memory back during tool calls

Tool calls can run for minutes. Offloading the idle KV lets other rollouts use that memory, so more rollouts run at once.

Rollouts can move freely

With P2P KV transfer, a rollout can resume on any worker after a sync, rebalance or failure, without rebuilding its context.

19 / 19