NVIDIA Dynamo · Omni serving

Serving every modality
with Dynamo

Vision-language models, speech and diffusion on one OpenAI-compatible serving stack: architecture, the features that make each modality fast, and live demos of how they work.

01Image & video

Parallel media decoding, embedding cache, multimodal KV routing, encoder disaggregation.

02Speech

Streaming ASR → LLM → TTS on /v1/realtime and /v1/audio/speech; audio-in/audio-out with Omni models.

03Diffusion

Text/image → image, video, audio and text across vLLM-Omni, SGLang, TensorRT-LLM and FastVideo.

← → navigate · N speaker notes · F full screen · T light/dark

Architecture

One serving stack, every modality

Highlight a path:

Where things stand · Dynamo 1.5

Omni at a glance

Image & video

Today

Parallel decodingMM KV routingEmbedding cacheEncoder disaggShared image cacheImage / video-aware routing

Backends

vLLM · SGLang · TRT-LLM

Speech

Today

Streaming ASRStreaming TTSASR → LLM → TTSAudio in / outIncremental input

3 OpenAI-compatible endpoints

/v1/realtime

/v1/chat/completions

/v1/audio/speech

Diffusion

Today

ImageVideoAudioText (dLLM)

4 backends

vLLM-Omni · SGLang

TRT-LLM · FastVideo

Offline batch

Today

Batch inference APIVLM captioning

Runs on

Dynamo workers

Triton worker for non-LLM models

01

Vision-language models

Four features that cut preprocessing, re-encoding and re-prefill for image and video requests. Use them separately or together, depending on the workload.

① Parallel media decoding② Multimodal KV routing③ Embedding cache④ Encoder disaggregation

Image & video · architecture

Where each feature acts on a VLM request

Image & video · when to use what

Pick the feature that matches the bottleneck

If your workload…

Turn on

Throughput ↑ 050100%

TTFT ↓ 050100%

spends a lot of time preprocessing images

① Parallel decoding

+20–30%

−10–30%

^*

repeats images across many workers

② MM KV routing

—

−15–30%

^**

re-downloads the same image URLs on many workers

Shared image cache

—

^†

repeats the same images

③ Embedding cache

+10–30%

−10–45%

^***

is bottlenecked on the vision encoder

④ Encoder disagg

—

−26–90%

^****

Features compose: frontend decoding gives content-stable image hashes for KV routing; the embedding cache can sit behind a disaggregated encoder.

* Qwen3.5-397B on 8×H100 and Qwen3.6-35B-A3B-FP8 on 1×H100 (vLLM / SGLang) · 5 high-res images per request

** Qwen3-VL-30B-A3B-FP8 on 8×B200 (vLLM / SGLang) · 1 image per request, 50–90% image reuse

*** Qwen3-VL-30B-A3B-FP8 on GB200 (vLLM) · 3 images per request, 20–80% image reuse

**** Qwen3.5-122B-A10B-NVFP4 on 4×GB200 + 2×RTX 6000 (heterogeneous) · e2e latency −2% to +85%

† No benchmark yet · saves repeat downloads of image URLs (vLLM, TRT-LLM); needs Redis Cluster or Dragonfly

Internal NVIDIA benchmarks, Dynamo 1.5.

① Parallel media decoding

Decode images in parallel, off the worker

Frontend fetches the URL or decodes the base64 data URL.
Images decode concurrently on a CPU pool (libjpeg-turbo for JPEG).
Pixel buffer is registered with NIXL; the worker reads it directly.
Worker still runs its processor + vision encoder — this moves decoding, it doesn't skip encoding.
Enable on the worker: --frontend-decoding (vLLM / TRT-LLM also --enable-multimodal). Images, aggregated topology.

Images / request

CPU cores

Resolution

② Multimodal KV routing · 1/2

The router must see images the way the KV cache does

Router needs how many KV blocks each image occupies…
…and which image each block belongs to.
Naive view: each image is one placeholder token — wrong block count, and every image hashes the same.
Dynamo expands images to their real visual-token count in a routing-only view and tags blocks with the image's multimodal hash.

Visual tokens per image KV block size

Swap 2nd image B → C

② Multimodal KV routing · 2/2

Cache credit vs load — and stable image identity

Router decision

cost = overlap_weight × prefill_blocks + active_blocks
Lowest cost wins.

Request:

Multimodal-aware

overlap_weight

Image identity: URL vs content

Same photo, two signed URLs. By default the router hashes the URL → miss. With --frontend-decoding it hashes the pixels → hit.

Frontend decoding

③ Embedding cache

Encode an image once, reuse the embedding

Stores post-encoder visual embeddings — the exact tensors fed to prefill.
Held in CPU RAM, LRU-evicted, sized with --multimodal-embedding-cache-capacity-gb.
Hit ⇒ skip the vision encoder; miss ⇒ encode, send over NIXL, save.
Not the KV cache: embedding cache skips encode, KV cache skips prefill.
Footprint: 2400×1080 on Qwen2.5-VL-7B ≈ 23 MB; 512² ≈ 2.2 MB.

Pick images (click order = prompt order), then send.

Cache capacity

④ Encoder disaggregation

Should you split out the vision encoder?

Your workload

Images per request

Output length

LLM

Chat traffic mixed in

Cheaper GPU tier

Where request time goes

vision encoderLLM prefill + decode

Time split is illustrative. Measured numbers: Dynamo EPD digest, Qwen3.5 on GB200.

Speaker notes

Rule of thumb from the Dynamo EPD digest: EPD pays off when vision encoding is a significant share of request time. It isolates encode work, which must be large enough to cover coordination and embedding-transfer overhead.
Image-heavy: Qwen3.5-122B-A10B NVFP4 on 4× GB200, 10 images/request (≤256 tokens each), OSL 1024 — mean TTFT −58% colocated, −50% heterogeneous; E2E gains modest. Across a 5–50 image sweep, TTFT improved 25–93% over aggregated while EPD stayed nearly flat as visual tokens grew.
Output length: 5 images, OSL 128 → 2,048 — heterogeneous E2E gain 20.3% → 5.2%; colocated went from +11.8% to a 2.5% regression. TTFT gains hold.
Model size (colocated goodput vs aggregated): 4B 2.62×, 9B 1.50×, 27B 0.65× (ViT share 7.2% / 4.7% / 1.7%). Precision: 1.78× BF16 vs 2.64× with NVFP4 LM weights.
Mixed 50:50 traffic, OSL 128, colocated: text TTFT 92.3 → 53.3 ms, image TTFT 289.9 → 200.6 ms.
Skip it: light media, long decodes, large dense models. 43% of TTFT was spent before the ViT in one breakdown — EPD doesn't help there; use frontend decoding and the embedding cache.
Topology: colocated encoders generally win on homogeneous clusters (homogeneous disaggregated always underperformed colocated). A separate lower-cost GPU tier makes disaggregated encoders attractive: 70% more goodput at the same SLO (ITL < 100 ms).
Headline: up to 5× faster TTFT and 7× faster end-to-end in the digest's best cases. Embeddings moved over NIXL (UCX RC/TCP Ethernet, 20 Gbps peak).
Support: vLLM E/PD + E/P/D (separate encoder handles image_url; video_url stays on PD); SGLang both; TRT-LLM E/P/D.

Live demo · all four features

Same workload, features on and off

① Parallel decoding

② MM KV routing

③ Embedding cache

④ Encoder disagg (+1 GPU)

Images / request Image reuse Arrival rate Resolution

3 LLM GPUs, 80 requests. Illustrative timing model.

GPU / CPU timeline

decodeencodeprefillNIXL

Feature ladder — stacking features on this workload

Measured · Qwen3.6-35B-A3B-FP8 · 8K tokens + 5 × 2400×1080 images · concurrency 30

Frontend decoding + embedding cache vs stock serving

vLLM · 1×H100 · Dynamo, adding each feature

config	RPS	TTFT avg (s)	p50 (s)	p90 (s)	p99 (s)
vllm-serve	0.719	18.17	18.83	26.99	48.59
+ frontend decoding	0.811	7.19	3.57	10.99	34.90
+ embedding cache	0.933	6.10	2.37	22.87	34.58
Δ vs vllm-serve	+29.8%	−66%	−87%	−15%	−29%

SGLang · H100 and B200 · vs sglang-serve

GPU	req/s	tok/s	TTFT avg	wall
H100	+43%	+42%	−41%	−30%
B200	+26%	+25%	−35%	−21%

240 requests, same workload on both engines; frontend decoding confirmed engaged in worker logs.

In production

Customer wins

Consumer maps service

2×

requests/s vs Triton + vLLM for a safeguard VLM

AI agent company

+15%

throughput

~10% from the embedding cache
~5% from Rust frontend decoding
~2M-request campaign

Qwen3-VL
on Dynamo

VLM serving with custom multimodal embeddings + MM routing

02

Speech & voice agents

Dynamo serves voice through OpenAI-compatible endpoints: separate ASR, LLM and TTS workers for a cascaded pipeline, or a vLLM-Omni model for audio in and audio out.

/v1/realtime · streaming ASR/v1/chat/completions · LLM/v1/audio/speech · TTS/v1/realtime · audio-in/audio-out

Speech · architecture

Cascaded voice agent on Dynamo

Speech · serving patterns

Cascaded pipeline or audio-native model

Live demo · one voice turn

Where the time goes between “user stops talking” and first audio

Utterance length Uncached LLM context at turn start LLM reply length

Sentence-chunked TTS

Incremental text input experimental

ASR revises final words

Illustrative timing: ASR final ≈250 ms after commit, prefill 0.08 ms/token, ~120 ms request overhead, 70 tok/s decode, TTS first byte ≈150 ms.

user speechASRspeculative prefillLLM prefillLLM decodeTTS → audio

Speaker notes

The metric that matters for voice is end-of-speech to first audio.
Sentence-chunked TTS is the recommended orchestration: send sentence-sized text to TTS as the LLM streams, instead of waiting for the full reply. Turn it off to see the penalty grow with reply length.
Incremental text input forwards provisional ASR text to the LLM over a realtime WebSocket so prefill overlaps transcription. Experimental: needs --realtime and --enable-prefix-caching on the worker and source-built images. Our running deck cites up to ~40% lower TTFT; the docs list it as planned work.
Turn on “ASR revises final words”: the speculative text is cleared and the final transcript resent — the benefit shrinks. Set uncached context to 0: when the history is already prefix-cached, only the short transcript overlaps, so gains are small — matching the README caveat.

Speech · wire protocol

A voice turn, event by event

Incremental text input

Speech · text-to-speech

One endpoint, several TTS backends

Qwen3-TTS on vLLM-Omni — --output-modalities audio. CustomVoice (predefined speakers + style) or VoiceDesign (describe a voice).
Streaming TTS (e.g. Magpie TTS) — 24 kHz PCM chunks for low time-to-first-audio.
Formats: wav, mp3, pcm, flac, aac, opus · speed 0.25–4.0.
Voices and languages load from the model config at startup.
Not yet vLLM-Omni audio streaming (stream: true) and voice cloning (Base task).

Backend / model Voice Input text Instructions (style) Speed Format

Preview uses the browser's built-in voice, not Dynamo.

Speech · status & roadmap

What ships, what's next

Available

Streaming ASR on /v1/realtime
TTS on /v1/audio/speech (streaming PCM)
Cascaded ASR → LLM → TTS example on Kubernetes + smoke test
Qwen3-ASR / Qwen3-TTS launchers
Qwen3-Omni audio-in/audio-out

Experimental

Incremental text input: overlap ASR with LLM prefill (input_text.append)
Realtime text path for the LLM (--realtime)
Needs source-built images today

Diffusion & Omni generation

Image, video, audio and text generation as first-class workloads: same frontend, routing, scaling and observability as LLM inference, with OpenAI-style endpoints.

/v1/images/generations/v1/videos/v1/audio/speech/v1/chat/completions

Diffusion · what runs where

Pick a workflow, see backends and how to call it

Click a row. ✓ supported · ⚗ experimental · — not supported

Diffusion · choosing a backend

Four backends, four sweet spots

vLLM-Omni

Broadest coverage

✓Text → Image

✓Text → Video

✓Image → Video

✓Text → Audio

–Text → Text

★Mixed media & multi-stage pipelines

⚠One output modality per worker

SGLang

LLM diffusion

✓Text → Image

✓Text → Video

✓Image → Video

–Text → Audio

✓Text → Text

★Alongside an SGLang fleet; diffusion LMs

⚠No text-to-audio

TensorRT-LLM

Experimental

⚗Text → Image

⚗Text → Video

–Image → Video

–Text → Audio

–Text → Text

★NVIDIA-optimized image & video

⚠Not for production yet

FastVideo

Kubernetes

–Text → Image

✓Text → Video

–Image → Video

–Text → Audio

–Text → Text

★Fastest text-to-video in production

⚠1 request per worker → add replicas

Live demo · step distillation

Fewer denoising steps, same picture

Diffusion starts from noise and denoises step by step; cost ≈ steps × transformer pass.
NVIDIA FastGen distills 50-step models to ~4 steps (DMD2 and other techniques).
vLLM-Omni runs the pipelines; Dynamo serves them with the same routing, scaling and API as LLMs.
Steps are a request knob: nvext.num_inference_steps.

Measured end-to-end speedup · DMD2 4-step vs 50-step

Animation time scaled; per-step cost is illustrative. Picture is a procedural scene, not model output.

Diffusion · disaggregated multi-stage serving

Split the pipeline into stages, scale the bottleneck

Each stage (e.g. AR → DiT for GLM-Image) runs as its own process on its own GPU.
The router is a pure message broker: it passes lightweight connector references, never the bulk data.
Scale a stage by launching more workers with the same --stage-id; the router load-balances across replicas.
Experimental tested with GLM-Image; inter-stage streaming not yet supported.

Throughput rate matching · Qwen3-Omni Thinker → Talker → Code2Wav

Thinker

Talker

Code2Wav

Measured: +13% tokens/s per GPU from rate matching · Qwen3-Omni on Dynamo + vLLM-Omni. Stage rates above are illustrative.

Diffusion · FastVideo × Dynamo · with Hao AI Lab (UC San Diego)

Real-time 1080p video on a single GPU

Before · 8× B200

~1 min

multi-step, dense attention

→

12× faster

⅛ the GPUs

After · 1× B200

< 5 s

5 s of 1080p video

◫Video sparse attention

»Step distillation

⚡NVFP4 + kernel fusion

⬡Dynamo: scale on Kubernetes

Race: make a 5-second clip

Playback at 10× real time

Faster than playback ⇒ video becomes an interface: steer the next clip in seconds.

What's next

Closing the real-time loop

Streaming inputs

Push audio, image and video frames as they arrive.

Workers consume input incrementally over WebSockets.

Live voice · camera · agents

Real-time video API

Add video to /v1/realtime (text + audio today).

Steer with streamed input, stream video out while generating.

e.g. Krea-Realtime-Video, LTX-2

Cost & SLA routing

Send each request to the cheapest GPU tier that meets its SLA.

Priority traffic escalates to the fastest tier.

Demo in the backup slide

Real-time input → real-time output → real-time interaction

Takeaways

One stack for every modality

Image & video

Four composable features; up to +30% throughput and −87% p50 TTFT measured; in production with customers including Pinterest.

Speech

Streaming ASR, LLM and TTS as independently scaled stages behind OpenAI realtime/speech APIs; incremental input next.

Diffusion

Image, video, audio and text generation across four backends; ~10× from step distillation; real-time 1080p with FastVideo.

docs.nvidia.com/dynamo · multimodal/overview · multimodal/voice-pipelines · diffusion/overview · github.com/ai-dynamo/dynamo

Backup · architecture roadmap

Cost & SLA-aware pipeline orchestration

Omni pipelines chain stages with different needs — prompt-enhance LLM, encode, denoise, VAE decode, upscale.
Planner publishes per-hardware capacity, latency and $/request for each stage.
Router sends each request to the cheapest tier that still meets its SLA, aware of current KV/load.
High-priority requests escalate to the fastest GPUs.
Same idea already ships for VLMs: run a light vision encoder on lower-cost hardware (see encoder disaggregation).

Future work — numbers in the demo are illustrative.

Latency SLA

Cluster load

High-priority request

Serving every modalitywith Dynamo

One serving stack, every modality

Omni at a glance

Image & video

Speech

Diffusion

Offline batch

Vision-language models

Where each feature acts on a VLM request

Pick the feature that matches the bottleneck

Decode images in parallel, off the worker

The router must see images the way the KV cache does

Cache credit vs load — and stable image identity

Router decision

Image identity: URL vs content

Encode an image once, reuse the embedding

Should you split out the vision encoder?

Same workload, features on and off

Frontend decoding + embedding cache vs stock serving

vLLM · 1×H100 · Dynamo, adding each feature

SGLang · H100 and B200 · vs sglang-serve

Customer wins

Speech & voice agents

Cascaded voice agent on Dynamo

Cascaded pipeline or audio-native model

Where the time goes between “user stops talking” and first audio

A voice turn, event by event

One endpoint, several TTS backends

What ships, what's next

Available

Experimental

Next

Diffusion & Omni generation

Pick a workflow, see backends and how to call it

Four backends, four sweet spots

vLLM-Omni

SGLang

TensorRT-LLM

FastVideo

Fewer denoising steps, same picture

Measured end-to-end speedup · DMD2 4-step vs 50-step

Split the pipeline into stages, scale the bottleneck

Throughput rate matching · Qwen3-Omni Thinker → Talker → Code2Wav

Real-time 1080p video on a single GPU

Race: make a 5-second clip

Closing the real-time loop

Streaming inputs

Real-time video API

Cost & SLA routing

One stack for every modality

Image & video

Speech

Diffusion

Cost & SLA-aware pipeline orchestration

Serving every modality
with Dynamo