NVIDIA Dynamo · Agents

Dynamo for Agentic Inference

Serve the whole agent program, not just its requests.

Agent harness lead agent subagent subagent ⚙ tool call session-id: s-42agent_hints {…} Dynamo trace + replayprofile, simulate program schedulerThunderAgent programmatic KVKVCR GPU workers
The problem

An agent is a program, not a stream of chats

time → requests independent requests · no idea what comes next · every turn competes on its own leadsubagent 1subagent 2 tool subagent 1 endssubagent 2 ends waits for subagents compact one session: shared prefix grows every turn · subagents fork and finish · tool gaps are predictable

Prefix reuse

Big system prompt + every turn resends history.

Structure

Subagents, planner–executor, compaction.

Tool gaps

GPU idle while the agent waits on tools.

New SLA

End-to-end session throughput, not one TTFT.

Dynamo's approach

Three layers, each builds on the last

Click a layer to jump. Results: early internal benchmarks, May 2026.

Layer 1

Connect the harness

A session ID in. Traces, replays and what-ifs out.

Layer 1 · request metadata

Two channels: who you are, and what you need

Identity · header
Intent · agent hints 

    

  

  
Session ID is passive: tracing and replay, not routing. Hints are active intent. Neither forces sticky placement.

  





  
Layer 1 · harnesses
Point your harness at Dynamo. Sessions come for free.

  

    

      

      

      

    

    
Session tree

      lead · s-42
      
      s-42:research
      s-42:review
      parent-session-id
      session-final ✓
    

  

  





  
Layer 1 · profile
Every turn and tool call, on one timeline

  

    
export DYN_REQUEST_TRACE=1      # frontend writes jsonl.gz
python3 benchmarks/request_trace/convert_to_perfetto.py … # → Perfetto

      

    

    
prefilldecodetool

  

  





  
Layer 1 · replay & simulate
Predict what-if without GPUs. Validate by replaying on real GPUs.

  

    
Your trace
sessions · turns · tool gaps · prefix sharing

    
→

    

      
offline · no GPUs
AISim predict
aisimulate predict --stack dynamo \
  --config trace.yaml --output-dir out/
2× workers?KV router vs round-robin?more KV capacity?engine settings?

      
→

      
live · GPUs
AIPerf replay
aiperf synthesize dynamo-trace trace.jsonl \
  --output rp/
aiperf profile --fixed-schedule \
  --use-dynamo-session-control …
Same arrival schedule and session tree, against a real deployment. Final numbers come from here.

    

  

  
Replays the request schedule, not the agent: prompts are synthetic but keep block-sharing topology; tool time stays in the gaps.

  





  
Layer 2
Schedule the program, not the request






  
Layer 2 · priority
One priority hint, honored at the router and in the engine

  

    
Router pending queue

      

      
lead · 10subagent · 5background · 0

    

      
Gate 1 · router
priority orders the queue · strict_priority tiers · osl sizes the pick

      
Gate 2 · engine
priority scheduling · priority-based KV eviction

        
vLLMSGLangTRT-LLM
routing✓✓✓
evictionplan✓plan
spec. prefill✓✓✓

    

  

  





  
Layer 2 · the high-concurrency cliff
Past KV capacity, admitting every turn thrashes

  

    
concurrent agent programsGPU KV fits 32

    

      
Admit every turn
request-level
turns / min
GPU time re-prefilling

      
ThunderAgent
program-level
turns / min
GPU time re-prefilling

    

    
KV in use, cache hitevicted → re-prefill whole historypaused between turns (waiting)

  

  
Illustrative model shaped by the ThunderAgent paper: parity at low concurrency, request-level collapses once KV is oversubscribed.

  





  
Layer 2 · ThunderAgent experimental
ThunderAgent: pause programs between turns, not mid-request

  

    
Each session is a program in ThunderAgent's table. Press → to step.1 · 10 programs run. Turns arrive, tool calls return, KV grows.2 · KV hits 95%: programs waiting on tools are paused between turns until usage is back at 80%. Running generations finish.3 · Room frees up: paused programs resume on their preferred worker, so their prefix is still a cache hit.4 · X-Dynamo-Session-Final releases a program's state; idle ones expire after 30 min.

    

      worker KV
      
      
      
      0.95 pause
      0.80 target
      running programspaused (KV kept / offloaded)
      
      
    

  

  





  
Layer 2 · results
The more agents in flight, the bigger the win

  

    
Speedup vs concurrent programs · paper, mini-SWE-agent

      

    

      
Dynamo+16%
throughput vs KV-aware routing
128 concurrent sessions · MiniMax-M2.5 · 2× TP4 on 8×H100 · mini-swe-agent · early

      
paper1.5–3.6×
serving throughput
vs vLLM and Continuum · agentic workloads

      
paper1.8–3.9×
RL rollout throughput
GLM-4.6 · up to 3.92× with OpenHands

    

  

  





  
Layer 2 · enable
One policy file and three flags

  

    
# thunderagent.yaml
request_classifier:
  type: thunderagent
worker_selection:
  aggregated: thunderagent
  instances:
    - {name: thunderagent, type: thunderagent}

python3 -m dynamo.frontend --router-mode kv \
  --router-policy-config thunderagent.yaml \
  --router-session-affinity-ttl-secs 1800 \
  --router-session-affinity-mode soft

    
Know before you ship
    
      
  • Experimental · tested on aggregated vLLM
    • 
      
  • Program state is per frontend: keep a session on one frontend
    • 
      
  • Send X-Dynamo-Session-Final when a program ends
    • 
      
  • Under pressure, deferred turns wait (up to 30 min), not 529

  

  





  
Layer 3
Programmatic KV cache
The agent knows when KV matters. Let it say so.






  
Layer 3 · events → actions
Agent events become KV actions

  

    

      
      
      
      
      
      
    

    

      

      
    

  

  





  
Layer 3 · try it
Free GPU memory while the agent waits

  

    

      
GPU KV held by one session

      
    

    
GPU KV·time freed
more sessions per GPU at the same KV budget
Illustrative session: 3 turns, 2 long tool calls, 1 subagent.

  

  





  
Recap & roadmap
Tell Dynamo about the program. It does the rest.

  

    

      
1
Connect & observe
Session IDs + hints. Trace, replay, simulate.
Next automatic hint injection from profiled workflows

      
2
Schedule the program
Priority end to end. ThunderAgent wins big at high concurrency.
Next ThunderAgent with KV offload, request migration, server-side tool envs

      
3
Programmatic KV
Prefill ahead, evict subagents, copy and move with KVCR.
Next tool-call offload + prefetch, pinning, block-level priority

    

  

  
Also next: pluggable model routing · agent-context spec with Mooncake (ATIF-compatible) · see the KVCR and Router decks.

  







Speaker notes





  ← → step · N notes · F fullscreen