NVIDIA Dynamo · Many-model serving

GPU sharing

Small models rarely need a whole GPU. Pack several onto one, each with its own memory and compute share.

KAI-SchedulerHAMiMPS v3Dynamo v1.6
The benefit · 16 models, mostly quiet

Most GPUs sit idle. Pack the quiet models together.

Each box is one team's fine-tuned Qwen3-4B: a few busy, most quiet.

quietsteadybusyfill = how busy the GPU is
Three configurations

Same GPU, three ways to fill it

Under the hood · GPU fractions

One annotation becomes a hard SM and memory cap

1Podgpu-fraction: "0.5"non-root · container "main"
→
2KAI-Schedulerpicks a GPU with room for
SMs and memory
gpu-compute.portion
→
3Node agents
fractiondinjects the caps (NRI)
mpsdone MPS namespace per pod
→
4GPU enforces
54 / 108 SMsMPS server
~20 GBdriver cap
Measured · Dynamo docs benchmark

Same 8 GPUs: twice the models, no loss in output

Dedicated1 model per GPU
8 modelsbaseline
KAI + HAMi2 models per GPU
16 models80–91% of output
KAI + GPU fractions2 models per GPU
16 models99–130% of output
Total output from the 8 GPUssame traffic per model in all three
Recommendation
✓
Low-traffic models → GPU fractions2× models, ≥ same output
■
Busy models → dedicatedgain gone at ~32 requests
!
HAMi → only if you can't build10–20% less output

Sharing makes each response a bit slower.

Reading the numbers · same traffic per model

Why fractions win at light load, and when the gain fades

Output vs dedicated16 shared models vs 8 dedicated, same 8 GPUs

Rule: share quiet models that can take a bit more latency; keep busy ones dedicated.

↗
Idle time gets usedA quiet GPU mostly waits; a second model fills it: +30%
⇅
Busy GPUs have no slackAlready full, plus two sets of weights: 99%
⏱
Split SMs, don't take turnsHAMi makes one model wait: 80–91%
Deploy

Same DGD, a different annotation

1 · Worker specDGD podTemplate · YAML
# worker podTemplate: a whole GPU
resources:
  limits: {nvidia.com/gpu: "1"}
args: [--gpu-memory-utilization, "0.85",
       --max-model-len, "4096", --max-num-seqs, "32"]
# worker podTemplate: half a GPU, memory capped
labels:      {kai.scheduler/queue: default-queue}
annotations: {gpu-fraction: "0.5"}
spec:
  schedulerName: kai-scheduler
  # no nvidia.com/gpu resource
args: [--gpu-memory-utilization, "0.85"]  # of ~20 GB
# worker podTemplate: half the SMs + memory
labels:      {kai.scheduler/queue: default-queue}
annotations:
  gpu-fraction: "0.5"
  nvidia.com/container.main.gpu-compute.mode: sm-sharing
spec:
  schedulerName: kai-scheduler
  runtimeClassName: nvidia
  securityContext: {runAsNonRoot: true, runAsUser: 1000}
args: [--gpu-memory-utilization, "0.40"]  # of 40 GB
2 · Install and deployshell · from examples/gpu-sharing
cd examples/gpu-sharing
common/install-dynamo-platform.sh
NODE_NAME=$GPU_NODE common/download-model.sh
NODE_NAME=$GPU_NODE baseline/gen-dgds.sh \
  | kubectl apply -f -          # 8 DGDs
common/install-dynamo-platform.sh
kai-hami/install-kai-hami.sh
  # KAI v0.17.0 + HAMi isolator 1.1.0
NODE_NAME=$GPU_NODE kai-hami/gen-dgds.sh \
  | kubectl apply -f -          # 16 DGDs
# check: CUDA_DEVICE_MEMORY_LIMIT=20070m
# build both forks at the pinned SHAs (19 images)
helm install kai-scheduler ... \
  --set global.gpuSharingMode=NvFractions
helm install gpu-fractioning ./operator/charts
NODE_NAME=$GPU_NODE kai-gpu-fractions/gen-dgds.sh \
  | kubectl apply -f -          # 16 DGDs
# check: 16 kai_* MPS namespaces at 50.00
Beyond pairs · MPS per-GPU limits

Cap a large model, pack small ones into what's left

Large model's SM share70%
Large model70% of SMs · 56 GB
Memory bandwidth is shared by everyone
MPS configTOML
[servers.shared.namespaces.large]
pinned_memory_limit      = "56G"
active_thread_percentage = "70"
Silent failures to watch
A ceiling, not a reservationNeighbors still contend for bandwidth.
Wrong pipe directoryClient gets the whole GPU, no error.
Unlisted namespaceLimits only apply to default.
Env-var memory capsTenants can drop them; use cgroup dmem.
Status · Nov 2026

Reproduce it, then measure your own workload

Where each setup stands
readyBaseline and KAI + HAMistock charts, published scripts
previewGPU fractionsforks pinned to open PRs
manualMPS per-GPU limitssingle-GPU verified, no Kubernetes

Illustrative: one model, one GPU type, no KV reuse. Percentages transfer; absolute numbers don't.

Run the benchmark
Sweepshell · from examples/gpu-sharing
# drive every model at once, sweep load
ENDPOINT_MODE=dns NUM_MODELS=16 \
  common/run-sweep.sh fractions
# 1 2 4 8 16 32 requests per model

Speaker notes

← → step · N notes · F fullscreen