NVIDIA Dynamo · Many-model serving
GPU sharing
Small models rarely need a whole GPU. Pack several onto one, each with its own memory and compute share.
KAI-SchedulerHAMiMPS v3Dynamo v1.6
The benefit · 16 models, mostly quiet
Most GPUs sit idle. Pack the quiet models together.
Each box is one team's fine-tuned Qwen3-4B: a few busy, most quiet.
quietsteadybusyfill = how busy the GPU is
Three configurations
Same GPU, three ways to fill it
Under the hood · GPU fractions
One annotation becomes a hard SM and memory cap
1Pod
→
gpu-fraction: "0.5"non-root · container "main"2KAI-Schedulerpicks a GPU with room for
SMs and memory
→
SMs and memory
gpu-compute.portion3Node agents
→
fractiondinjects the caps (NRI)
mpsdone MPS namespace per pod
4GPU enforces
54 / 108 SMsMPS server
~20 GBdriver cap
Measured · Dynamo docs benchmark
Same 8 GPUs: twice the models, no loss in output
Dedicated1 model per GPU
8 modelsbaseline
KAI + HAMi2 models per GPU
16 models80–91% of output
KAI + GPU fractions2 models per GPU
16 models99–130% of output
Total output from the 8 GPUssame traffic per model in all three
Recommendation
✓
Low-traffic models → GPU fractions2× models, ≥ same output
■
Busy models → dedicatedgain gone at ~32 requests
!
HAMi → only if you can't build10–20% less output
Sharing makes each response a bit slower.
Reading the numbers · same traffic per model
Why fractions win at light load, and when the gain fades
Output vs dedicated16 shared models vs 8 dedicated, same 8 GPUs
Rule: share quiet models that can take a bit more latency; keep busy ones dedicated.
↗
Idle time gets usedA quiet GPU mostly waits; a second model fills it: +30%
⇅
Busy GPUs have no slackAlready full, plus two sets of weights: 99%
⏱
Split SMs, don't take turnsHAMi makes one model wait: 80–91%
Deploy
Same DGD, a different annotation
1 · Worker specDGD podTemplate · YAML
# worker podTemplate: a whole GPU resources: limits: {nvidia.com/gpu: "1"} args: [--gpu-memory-utilization, "0.85", --max-model-len, "4096", --max-num-seqs, "32"]
# worker podTemplate: half a GPU, memory capped labels: {kai.scheduler/queue: default-queue} annotations: {gpu-fraction: "0.5"} spec: schedulerName: kai-scheduler # no nvidia.com/gpu resource args: [--gpu-memory-utilization, "0.85"] # of ~20 GB
# worker podTemplate: half the SMs + memory labels: {kai.scheduler/queue: default-queue} annotations: gpu-fraction: "0.5" nvidia.com/container.main.gpu-compute.mode: sm-sharing spec: schedulerName: kai-scheduler runtimeClassName: nvidia securityContext: {runAsNonRoot: true, runAsUser: 1000} args: [--gpu-memory-utilization, "0.40"] # of 40 GB
2 · Install and deployshell · from examples/gpu-sharing
cd examples/gpu-sharing
common/install-dynamo-platform.sh
NODE_NAME=$GPU_NODE common/download-model.sh
NODE_NAME=$GPU_NODE baseline/gen-dgds.sh \
| kubectl apply -f - # 8 DGDs
common/install-dynamo-platform.sh kai-hami/install-kai-hami.sh # KAI v0.17.0 + HAMi isolator 1.1.0 NODE_NAME=$GPU_NODE kai-hami/gen-dgds.sh \ | kubectl apply -f - # 16 DGDs # check: CUDA_DEVICE_MEMORY_LIMIT=20070m
# build both forks at the pinned SHAs (19 images) helm install kai-scheduler ... \ --set global.gpuSharingMode=NvFractions helm install gpu-fractioning ./operator/charts NODE_NAME=$GPU_NODE kai-gpu-fractions/gen-dgds.sh \ | kubectl apply -f - # 16 DGDs # check: 16 kai_* MPS namespaces at 50.00
Beyond pairs · MPS per-GPU limits
Cap a large model, pack small ones into what's left
Large model's SM share70%
Large model70% of SMs · 56 GB
Memory bandwidth is shared by everyone
MPS configTOML
[servers.shared.namespaces.large]
pinned_memory_limit = "56G"
active_thread_percentage = "70"
Silent failures to watch
A ceiling, not a reservationNeighbors still contend for bandwidth.
Wrong pipe directoryClient gets the whole GPU, no error.
Unlisted namespaceLimits only apply to
default.Env-var memory capsTenants can drop them; use cgroup
dmem.Status · Nov 2026
Reproduce it, then measure your own workload
Where each setup stands
readyBaseline and KAI + HAMistock charts, published scripts
previewGPU fractionsforks pinned to open PRs
manualMPS per-GPU limitssingle-GPU verified, no Kubernetes
Illustrative: one model, one GPU type, no KV reuse. Percentages transfer; absolute numbers don't.
Run the benchmark
Sweepshell · from examples/gpu-sharing
# drive every model at once, sweep load ENDPOINT_MODE=dns NUM_MODELS=16 \ common/run-sweep.sh fractions # 1 2 4 8 16 32 requests per model