NVIDIA Dynamo · experimental · v1.5 → 1.6

Dynamo Sidecar

Take Dynamo out of the engine process. Keep everything Dynamo is good at.

Before · integrated backend

Today, Dynamo lives inside the engine

The idea

Split the worker in two

python -m dynamo.vllm: everything in one address spacetwo processes, two containers, one pod, connected over the engine's native gRPC

Dynamo side

runtime: Dynamo sidecar CPU · Rust

Register the engine in discovery Forward KV-cache events Watch health, drive lifecycle Relay requests (today) System metrics on :9090

native gRPC
loopback / UDS

Engine side

main: stock engine GPU · upstream

Native serve CLI and flags Scheduling and batching Sampling and token generation KV cache and NIXL transfer GPU execution

Blast radius

Only one tile in the system changes

Request path

Three generations of the hop to the engine

1 hoprouter → worker, then Python calls in-process

Rust ↔ PythonPyO3 crossing on the hot path, shared GIL

Custom imageengine pinned to Dynamo's release

2 hopsrequest plane, then loopback gRPC

Process boundaryno shared interpreter or GIL

Stock imagevllm/vllm-openai, including nightly

1 hoprouter calls the engine's gRPC directly

Sidecar off the request pathstill the frontend’s link for discovery and health

In progressPR #13292 · KV-aware routing on this path not yet done

Integrated: the router sends token IDs to the worker over the request plane. From there it's Python function calls into the engine, so the worker's Python handlers and the engine share one GIL.
Sidecar today (dev docs, 1.6): the router sends to the sidecar, and the sidecar converts the request to the engine's native gRPC over loopback. That's one extra local hop, but no shared interpreter.
Target (v1.5.1 overview and PR #13292): the router calls the engine's gRPC directly. The sidecar becomes a thin registrar: requests go straight to the engine, but the frontend still talks to the sidecar for discovery (registration, model metadata) and health. Router features like fault detection and migration are kept through the StreamingDispatch seam.
Known gap called out in PR #13292: KV-aware routing is off on the direct path because stock engines don't yet stream KV events the frontend can subscribe to.

Kubernetes

Inside the worker pod

initContainers:
- name: runtime        # Dynamo
  image: dynamo-sidecar:1.6.0
  command: [dynamo-vllm-sidecar]
  args: [--grpc-endpoint, 127.0.0.1:30001]
  restartPolicy: Always
containers:
- name: main           # engine
  image: vllm/vllm-openai:<ver>

Lifecycle · DEP #14657 + #14897

From pod start to routable, and back

1 · start

2 · load

3 · ready

4 · prepare

5 · register

6 · shutdown

Kubelet

start runtime first

start main

SIGTERM main first

Sidecar

/live ✓ before engine

Health.Watch · waiting

sees SERVING

read engine ID · build KV index

register model + endpoint

deregister · exit last

Engine

load weights · NOT_SERVING

SERVING

KV event sources

NOT_SERVING · drain

Router

worker routable ✓

worker removed

Press → to walk through the lifecycle. The sidecar's HTTP server comes up before it waits for the engine, which avoids a startup deadlock. The engine loads weights. The sidecar subscribes to the standard grpc.health.v1 Watch stream. The engine reports SERVING. Kubernetes probes the same state with Health.Check. The engine ID tells the sidecar whether its KV index belongs to this engine or a replaced one. The sidecar registers only after SERVING and KV prep. From here the router can pick this worker. The engine owns draining. The sidecar deregisters and exits without adding any grace period of its own.

Fault isolation · try it

What happens when something breaks?

Sidecar watches engine health
Worker is routable
Separate logs and probes
In-process: one crash takes down both

Deregistered immediately
Only the engine restarts
New engine ID: KV index reset
Re-registers once SERVING

Engine, weights and KV cache survive
Unroutable until the sidecar is back
In-flight requests fail or migrate
KV index is lost (recovery: WIP)

Engine reports NOT_SERVING (e.g. RL weight update)
Deregistered while paused
KV index kept
Re-registers once SERVING

Click the scenarios or press →. They come from DEP #14897 (sidecar behavior table) and the E2E tests T3/T4 in DEP #14657.
Engine crash: only main restarts. The new engine ID tells the sidecar the cached KV view is stale, so it throws it away.
Sidecar crash: the GPU process survives, so there's no weight reload, which is the expensive part. While the sidecar relays requests (today), in-flight requests fail or migrate. With direct dispatch they would keep streaming.
Engine crash: the watch stream drops, so the sidecar deregisters immediately; Kubernetes restarts only main while the sidecar stays up, and the worker re-registers after SERVING and KV prep.
Pause/sleep: used for RL weight updates. The worker leaves the routable set but keeps its cache index.
KV state after a sidecar crash: today the local KV indexer lives inside the sidecar, so if the sidecar goes down there is no way to recover the engine's KV state. Rebuilding the index on restart (for example by replaying the engine's events) is work in progress.

Router integration

KV-aware routing still works

Press → to follow a KV event and then a routed request. At startup the sidecar asks the engine where its KV event publishers are: vLLM's Control service, SGLang's GetServerInfo. Each DP rank publishes block stored/removed events over ZMQ. The sidecar forwards them to the event plane. The router's indexer matches the prompt's block hashes against what each worker and rank holds. The chosen DP rank travels as gRPC metadata, so the request lands on the rank that holds the cache.

Disaggregated serving

Prefill → decode, with engines doing the KV transfer

vLLM ✓

typed kv_transfer_params handoff

SGLang ✓

bootstrap connection handoff

TRT-LLM ✗

gRPC lacks disagg fields; Phase 1 blocker

Performance · mechanism

Why it can be faster: streaming stops being single-threaded

Concurrent streams96illustrative model, not a benchmark

–GPU idle, in-process

–GPU idle, sidecar

–sidecar throughput vs in-process

This is a toy model to explain the mechanism, not a benchmark.
Each step, the GPU runs a forward pass while the CPU works on the next step in parallel: the engine schedules the next batch, and responses from the last step are streamed out. Engine frameworks overlap scheduling with GPU execution, so the GPU only waits when CPU work takes longer than the forward pass.
Both kinds of CPU work grow with the number of concurrent streams.
In-process: one Python process streams every response for Dynamo and the engine. Multithreading doesn't help because of the GIL, so at high stream counts streaming takes longer than the forward pass and the GPU waits.
Sidecar: a multithreaded Rust sidecar talks to a multithreaded gRPC frontend, and both can scale up and out, so streaming keeps up.
Drag the slider: at low stream counts both lanes match. The gap only opens once streaming outgrows one Python process, which is why the benefit shows up mainly in high-throughput deployments.

Performance

Where the sidecar is faster

Engine output2.0×

Sidecar · multithreaded Rust + gRPC, scales outIn-process · one Python process, GIL-bound

Costs to watch

KV events: one more hop to the router
No forward-pass metrics for load-based planning

Minor in practiceloopback over localhostgRPC ser/de across cores

Before it becomes the default

Per-token overhead stays small
KV-aware routing works as well as today
Same features as in-process
Benchmarked against in-process

vLLM, SGLang and TRT-LLM switch together · DEP #10842

The key limit of the in-process path: throughput is capped by how fast one Python process on the Dynamo and engine side can process streaming responses. Multithreading doesn't help because of the GIL.
With the sidecar, a multithreaded Rust sidecar talks to a multithreaded gRPC frontend, which removes that bottleneck. Both can scale vertically and horizontally for higher throughput.
The benefit is limited to high-throughput deployments where the engine produces more than the single Python process can handle. The speedup depends on how far past that Python roofline the engine runs. Drag the slider: below the roofline both paths match; past it, the gap is the speedup.
Costs: the loopback hop (localhost) and gRPC serialization (spread across cores) are minor in practice. Watch KV events reaching the router one hop later, and the missing forward-pass metrics for load-based planning.
Direct dispatch is not a win over in-process: it only removes a relay hop the sidecar itself introduced.
Before it becomes the default (DEP #10842): the sidecar has to pass a set of checks — per-token RPC overhead within a budget, KV-aware routing as accurate as in-process, feature parity, and a benchmark of in-process vs sidecar vs direct dispatch. The decision applies to vLLM, SGLang and TRT-LLM together. No published numbers yet; that's what #10842 produces. The chart is illustrative.

The boundary · DEP #10838

How the sidecar talks to each engine

Engine-specificfast to ship · code per engine

OpenEngineone surface for every engine: GenerateGetModelInfoGetLoadHealthAbortDrainSubscribeKvEvents

Where it stands · Oct 2026

Readiness and roadmap

	Agg	Disagg	KV routing	K8s
vLLM	✓	✓	✓	✓
SGLang	✓	✓	✓	✓
TRT-LLM	✓	✗	~	agg

validated launch topologies, not full parity

Open gaps

Metrics & tracing

forward-pass metricsKV load gaugesengine metric re-exporttrace propagation

Not supported yet

multinode on K8sGMS · failover · checkpoint

Operator + CRDsingle-node agg / disagg · in progress

Observabilitymetrics, traces

MultinodeTP / PP

ResilienceGMS, failover, checkpoint

Try it

Run a sidecar in two minutes

# local, single GPU: no etcd or NATS
export DYN_DISCOVERY_BACKEND=file
./lib/sidecar/vllm/launch/agg.sh

# kubernetes
kubectl apply -f lib/sidecar/vllm/deploy/agg.yaml

# image (1.6.0+)
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-sidecar:1.6.0