NVIDIA Dynamo · Hands-on guide

Deploy and tune Dynamo

From a laptop to a Kubernetes cluster, then faster: recipes, the right knobs, and agents that tune for you.

👆 Click a stop on the map to start from there

Start here

Four ways in. Pick the one that matches where you are.

Local · install

Install: one container or one wheel

Method

Backend

# verify the install
python3 -m dynamo.frontend --help
python3 dev/sanity_check.py
# optional: etcd + NATS for KV events and routing
docker compose -f dev/docker-compose.yml up -d

Requirements · v1.6.0

OS: Ubuntu 22.04 / 24.04 · x86_64 or ARM64
CUDA: 12.9+ or 13.0+ (B300 / GB300 need 13)
Driver: 575.51.03+ (CUDA 12) · 580.00.03+ (CUDA 13)
Python: 3.11 / 3.12 (TRT-LLM: 3.12)
Engines: vLLM 0.31.0 · SGLang 0.5.21 · TRT-LLM 1.3.0rc29

No etcd or NATS needed to start: --discovery-backend file uses the filesystem. Add them when you want KV-aware routing locally.

Local · first request

Two processes and a curl

# terminal 1 · frontend on :8000
python3 -m dynamo.frontend --discovery-backend file

# terminal 2 · a worker (pick your engine)
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B \
    --discovery-backend file
#   dynamo.sglang / dynamo.trtllm take --model-path

# terminal 3 · ask it something
curl -sf localhost:8000/health && echo OK
curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"Hello!"}],
       "max_tokens":50}'

Press → to start the worker, then send a request.

Local · scale the shape

Launch presets: from one worker to disaggregated with a KV router

quickstart · 1 GPU

One aggregated worker does prefill and decode. Fine for a smoke test and small models.

launch/agg_router.sh · 2 GPUs

Two workers behind the KV-aware router: requests go to the worker that already holds the prefix. vLLM needs PYTHONHASHSEED=0 for matching block hashes.

TRT-LLM preset: 1 worker.

launch/disagg.sh · 2 GPUs

1 prefill + 1 decode worker. KV moves prefill → decode over NIXL. Workers take --disaggregation-mode prefill|decode; vLLM needs a unique VLLM_NIXL_SIDE_CHANNEL_PORT per worker.

launch/disagg_router.sh · 4 GPUs

2 prefill + 2 decode with the KV router picking the prefill worker. TRT-LLM preset: 1P + 1D on 2 GPUs.

# presets live per backend
cd examples/backends/vllm   # or sglang, trtllm
./launch/disagg_router.sh

Kubernetes · install

Install the platform once, add what your workload needs

# 1 · GPU Operator
helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace --wait
# 2 · Dynamo platform: operator + NATS
helm upgrade --install dynamo-platform \
  "$CHARTS/dynamo-platform-${DYNAMO_VERSION}.tgz" \
  -n dynamo-system --create-namespace --wait
# 3 · Hugging Face token (this exact name)
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN=...
# 4 · check
kubectl get crd | grep dynamo
./deploy/pre-deployment/pre-deployment-check.sh

I need

Pick what you need to see which add-on it brings in, and why.

From the Kubernetes quickstart and install guide. Add --set driver.enabled=false to the GPU Operator install when the cloud provider manages drivers. $CHARTS is https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts. Prerequisites: Kubernetes ≥1.30 with GPU nodes, kubectl ≥1.30, Helm v3+. Verify: CRDs present, dynamo-operator-* and nats-* pods running.
Add-ons: multinode needs Grove + KAI Scheduler or LWS + Volcano (the operator errors without one); production disaggregation needs RDMA via the Network Operator (TCP fallback: ~98 s TTFT vs ~200–500 ms with RDMA, per the install guide); Planner sla mode and KEDA/HPA need kube-prometheus-stack; models over 70B benefit from an RWX PVC model cache.
Docs caveat: the install guide still pins RELEASE_VERSION=1.2.1 while the quickstart uses 1.6.0. Use the current release.

Kubernetes · DynamoGraphDeployment

One DGD describes the whole serving graph

Backend

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata: {name: qwen3-32b-agg}
spec:
  components:
  - name: Frontend
    type: frontend
  - name: worker
    type: worker
    replicas: 1
    podTemplate:
      spec:
        containers:
        - name: main
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0
          envFrom: [{secretRef: {name: hf-token-secret}}]
          command: [python3, -m, dynamo.vllm]
          args: [--model, Qwen/Qwen3-32B, --tensor-parallel-size, "2"]
          resources: {limits: {nvidia.com/gpu: "2"}}

GPU limit = TP × PPcontainer must be named mainTP inside a node · PP + multinode.nodeCount across

Hover the YAML to see what each part creates.

Kubernetes · auto-deploy

Not sure how to size it? Ask for an SLA and let the profiler pick

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata: {name: qwen3-quickstart}
spec:
  model: Qwen/Qwen3-0.6B   # the only required field
  backend: auto
  searchStrategy: rapid     # or thorough
  workload: {isl: 4000, osl: 1000}
  sla: {ttft: 2000, itl: 30}  # ms
  autoApply: true

rapid~30 ssimulated with AISim · no GPUs

thorough2–4 hreal GPUs + AIPerf · disagg only

# review before deploying: set autoApply: false, then
kubectl get dgdr my-model -o jsonpath=\
  '{.status.profilingResults.selectedConfig}' > my-dgd.yaml

Recipes

Recipes: a tested deployment for a model, a backend and a mode

recipes/
└─ qwen3-32b/                # model
   ├─ model-cache/
   │  ├─ model-cache.yaml    # PVC
   │  └─ model-download.yaml # Job
   └─ vllm/                  # backend
      ├─ agg-round-robin/     # mode
      └─ disagg-kv-router/
         ├─ deploy.yaml       # full DGD
         └─ perf.yaml         # AIPerf Job

15model providers

31models

191tested targets

# 1 · token + model cache (edit storageClassName first)
kubectl create secret generic hf-token-secret \
  --from-literal=HF_TOKEN=... -n $NS
kubectl apply -f qwen3-32b/model-cache/ -n $NS
kubectl wait --for=condition=Complete \
  job/model-download -n $NS --timeout=6000s

# 2 · deploy
kubectl apply -n $NS \
  -f qwen3-32b/vllm/disagg-kv-router/deploy.yaml

# 3 · talk to it
kubectl port-forward svc/<name>-frontend 8000:8000

# 4 · benchmark it (optional)
kubectl apply -n $NS -f .../disagg-kv-router/perf.yaml

Recipes · browse

Find your model: providers in the docs' order, most popular first

Backend	Mode	Hardware	Numbers

Deploy this target

Recipe page in the docs →

Recipes · expected numbers

Recipes publish numbers: compare shapes before you deploy

GLM-5.3Z.ai · SGLang · B200

agg · 16 GPUs

190disagg · 20 GPUs

324

+70%tok/s/GPU with disagg (3P1D) at ~62 tok/s/user

Kimi-K2.6Moonshot AI · vLLM agg

8× H200

66.54× B200

166.5

2.5×tok/s/GPU on B200 vs H200 at ~56 tok/s/user

DeepSeek-V4-FlashDeepSeek · vLLM · B200

agg · 4 GPUs

362disagg · 12 GPUs

410

+13%tok/s/GPU with disagg (2P1D) at ~50 tok/s/user

Qwen3.6-35B-A3BQwen · SGLang agg · 1 GPU

H200

1,332B200

2,989GB200

3,036

2.3×tok/s/GPU on GB200 vs H200, same recipe

output tokens/s per GPU from each recipe's expected performance · agentic trace replays at the stated per-user speed · compare within a card, not across models

Same provider order as the Recipes tab: Z.ai, Moonshot AI, DeepSeek, Qwen. Numbers are the expected_performance summaries in the docs recipe catalog (Oct 24 2026).
GLM-5.3 on B200 (SGLang): aggregated 16 GPUs 190.05 tok/s/GPU at 61.92 user tok/s P50 (c64) vs disaggregated 20 GPUs (3P1D) 323.82 tok/s/GPU at 63.13 user tok/s (c128), 15% agentic trace subset. Note disagg TTFT P50 is higher (1280 vs 229 ms).
Kimi-K2.6 (vLLM aggregated): 4×B200 166.5 tok/s/GPU at 55.50 tok/s/user (c64) vs 8×H200 66.5 tok/s/GPU at 56.06 tok/s/user (c48).
DeepSeek-V4-Flash on B200 (vLLM): aggregated 362.1 system tok/s/GPU at 50.6 user tok/s vs disaggregated 2P1D on 12 GPUs 409.9 at 50.35.
Qwen3.6-35B-A3B (Dynamo + SGLang aggregated, 1 GPU): H200 1332.03, B200 2988.50, GB200 3035.98 output tok/s/GPU at concurrency 32 on a 3,541-request agentic trace.

Optimize · method

Measure, change one thing, measure again

pip install aiperf
aiperf profile --model Qwen/Qwen3-32B-FP8 \
  --url http://localhost:8000 --endpoint-type chat \
  --streaming --concurrency 10 --request-count 100 \
  --synthetic-input-tokens-mean 2000 \
  --output-tokens-mean 256
# results → artifacts/

Rules that keep results honest

Pick the SLA first: TTFT and ITL targets decide what "better" means.

One variable per run: agg vs disagg, KV vs round-robin, replicas, TP/PP.

Benchmark inside the cluster, not through a port-forward, then sweep concurrency.

Optimize · what to change

Start from the symptom, not the knob

Lever · KV-aware routing

Route each request to the worker that is cheapest to finish it

# frontend: turn it on (default round-robin)
--router-mode kv
# vLLM worker: publish KV events
--kv-events-config '{"publisher":"zmq",
  "endpoint":"tcp://*:20080",
  "enable_kv_cache_events":true}'
# tune through a policy file
--router-policy-config worker-selection.yaml

overlap_score_creditLower → spread load, lower ITL

prefill_load_scaleRaise → when long prompts saturate workers

Score by

cachedto computealready queued· an 8-block prompt arrives

overlap_score_credit1.0

Inspired by the router deck's "How it picks a worker". Each worker's cost = blocks it still has to compute + work it already has; lowest wins. Cache only piles onto the busy worker; load only throws away reuse; round-robin ignores both.
The real cost function (router configuration and tuning): prefill_load_scale × adjusted prefill blocks + decode blocks (+ optional active-request term). overlap_score_credit scales how much a cached block counts: lower it to spread load and reduce ITL; raise prefill_load_scale when long prompts saturate workers. router_temperature adds randomness.
Other flags: --no-router-kv-events (approximate routing), --router-predicted-ttl-secs 5 for fan-out or best-of-N bursts, --no-router-assume-kv-reuse for disagg, --router-queue-threshold and --router-queue-policy wspt|fcfs. The older --router-kv-overlap-score-credit and --router-prefill-load-scale flags are deprecated (removal in v1.7).
Worker numbers are a toy example. Deep dive: router.html.

Lever · disaggregation

Split prefill and decode, but only when your workload calls for it

✅ Use it for

Long prompts
High concurrency
Different TP per phase

⛔ Skip it for

Small models
Short prompts
No fast fabric

🔮 Not sure? Project it first

AISim compares agg vs disagg for your model, traffic and SLA in ~30 s. No GPUs.

pip install aisimulate
aisimulate recommend -c my-model.yaml  # agg vs disagg

AISim on GitHub →AISim deck →

🔌 Deployed? Test the transfer path

# from the dynamo-interconnect-check skill
check_interconnect.py env  <recipe>        # NIXL/UCX/NCCL
check_interconnect.py node --pod <worker>  # IB, RDMA, NVLink
check_interconnect.py nixl --pod <worker>  # transfer test

Prefill workers1

Decode workers2

★ best for this load: · live toy model, 6 req/s

TTFT p90–

SLA 1 s

ITL–

SLA 35 ms

Goodput / GPU–

When to use it (disaggregated serving overview): long prompts, high concurrency, or different TP needs per phase. When not to: small models, short prompts, or no fast fabric. Without RDMA, KV transfer falls back to TCP (the install guide quotes ~98 s TTFT vs ~200–500 ms with RDMA).
Live toy simulation: requests arrive (6/s), wait for a prefill worker (0.25 s each), hand their KV to the least-loaded decode worker, then stream 60 tokens; a decode step takes 20 ms + 2.5 ms per active stream on that worker. A request counts toward goodput if TTFT ≤ 1 s and its average ITL ≤ 35 ms. Start at 1P/2D: prefill can only do 4 req/s, so the queue grows and TTFT fails. Add prefill → TTFT recovers but decode streams pile up → ITL fails. Add decode → both pass. Keep adding → goodput per GPU drops. The ★ best split is found by a quick seeded sweep of all 36 splits when the slide loads (2P/3D with these settings). Real numbers: AISim or a DGDR profile.
Transfer-path check: the dynamo-interconnect-check agent skill ships scripts/check_interconnect.py with three read-only subcommands: env (NIXL/UCX/NCCL env vars on a recipe), node (InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod) and nixl (NIXL transfer-test readiness for a pod). Run it after deploying a disaggregated recipe.
Flags: --disaggregation-mode prefill|decode; sharedMemorySize: 16Gi in the DGD (64 MB default breaks NIXL); kubectl logs <prefill-pod> | grep NIXL should show UCX.
Project it first with AISim: aisimulate recommend searches parallelism, batching, replicas and agg vs disagg and returns the fewest GPUs that meet the SLA, in about 30 seconds without GPUs. The Kubernetes equivalent is a DGDR with searchStrategy: rapid.
Sizing once deployed: no fixed ratio; prefill queue grows → add prefill; decode KV near its limit → add decode; disagg worse than agg → check NIXL/RDMA first. vLLM flags: --kv-transfer-config with NixlConnector, kv_role kv_producer / kv_consumer.

Lever · memory and capacity

Keep KV instead of recomputing it. Let the Planner follow traffic.

KV offloading · pick one backend

LMCache--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both",…}'

SGLang HiCache--enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through

KVCRcross-node KV sharing · see the KVCR deck

FlexKVDYNAMO_USE_FLEXKV=1 · FLEXKV_CPU_CACHE_GB

Size the pod memory limit to hold the host cache.

Planner · in the DGDR

features:
  planner:
    mode: disagg
    optimization_target: sla  # default: throughput
    ttft_ms: 500.0
    itl_ms: 50.0
    enable_throughput_scaling: true  # every 180 s
    enable_load_scaling: true        # every 5 s
    advisory: true   # recommend only

step 1advisorywatch its recommendations

step 2throughputlet it scale

step 3sla+ Prometheus, both scalers

Agent skills

Hand the loop to a coding agent, with guardrails

# just state the outcome; skills are picked by intent
"Optimize this deployment for output tokens/s per user
 under a 200 ms TTFT SLO. Budget 8 GPU-hours and stop
 after three failed deployments."

Cheat sheet

Everything on one page

💻 Local

python3 -m dynamo.frontend
python3 -m dynamo.vllm --model …

Local Guide → ☸️ Platform

helm upgrade --install \
  dynamo-platform …

Kubernetes Guide → 🎯 Auto-size

kind: DynamoGraphDeploymentRequest
searchStrategy: rapid

DGDR + Profiler → 📖 Recipe

kubectl apply -f \
  <model>/<backend>/<mode>/deploy.yaml

recipes/ → 📊 Benchmark

aiperf profile --url … \
  --concurrency 10 --streaming

AIPerf → 🤖 Agents

"Optimize for tok/s/user
 under a 200 ms TTFT SLO"

.agents/skills →