NVIDIA Dynamo · KVCR

Distributed KV Cache Management with Dynamo Router and KVCR

Routerplans KV moves Worker A Worker B KVCR KVCR
The problem

Four requirements for KV cache management

1

Avoid recompute

↓ store↑ retrieve HBM DRAM Remote storage

Store KV in cheaper tiers and bring it back instead of recomputing.

2

Avoid hot spots

same prefix, many requests busyidleidle

One popular prefix shouldn't pin all its traffic to one worker.

3

Recover failures

worker dies other worker cache lostrecompute

Losing a worker shouldn't mean recomputing its whole cache.

4

Signal aware

Agent harness sub-agent done context compacted KV cache safe to evict

The workload knows when KV is done with — use that signal.

Dynamo’s approach

Router plans. KVCR acts.

Client Router load and KV awareload and KV aware · plans KV moves WorkerEngine WorkerEngine KV cache managerdecides where KV goes, on its own signals 1hint-aware router kv hint 2hints ride on requests KVCR KVCR KV moves between workers 3KVCR library
Planning

Plan early, with the full picture

Router routes this request, then the nextbuilds a forward plan signals cache load now Worker A Worker B t1 t2 busyidle t2 copy t1 KV t3 t4 balanced time
The router sees one request at a time, so t2 queues behind t1 on busy Worker A while B sits idle.With signals, the router plans ahead: KVCR copies t1’s KV to B early, so t2–t4 run there.
Memory tiers

Four tiers, two owners

Tier
Capacity
Owner
Shared with
G1 · HBMGPU memory
Engine
this engine only
G2 · DRAMhost memory
KVCR
other workers, via KVCR
G3 · NVMelocal diskfuture
KVCR
other workers, via KVCR
G4 · Remoteshared storagefuture
KVCR
other workers, via KVCR
KVCR in action

Sharing G2 across workers

Router t2 blocks: Worker B, G2 Worker AEngineG1 HBMG2 DRAMKVCRWorker BEngineG1 HBMG2 DRAMKVCR 1 kv hint: fetch t2 from B 2 3 4
The router finds the KV. KVCR brings it to the worker that needs it. Press → to step through. 1 · The router knows where the KV is: t2’s blocks sit in Worker B’s G2. 2 · The plan rides on the request as a hint: fetch t2 from B. 3 · A’s KVCR pulls the blocks from B’s G2, peer to peer. 4 · A’s engine loads them into G1 and runs t2 without recomputing.
KVCR-Guard · try it

Recovering from failures

Router t2 goes to Worker BB’s engine is downt2 goes to A, KV from B’s poolS1 goes to Worker B again hint: fetch t2 from B’s pool Worker A Engine G1 HBMG2 DRAM KVCR Worker B Engine down G1 HBM lost KVCR KVCR-Guard G2 DRAM pool Backup KVCR standbycheckingactive
The DRAM pool sits in its own container, KVCR-Guard, so it can outlive the engine. The engine fails and G1 is lost. The pool survives. The guard confirms the engine is dead — a timeout alone isn’t enough to take over. The backup serves the pool. Worker A copies the KV from B’s pool, peer to peer, instead of recomputing it. The engine comes back and refills G1 from the pool — a copy, not a recompute — then a fenced handoff returns control.
Try it

Live simulation: KVBM vs KVCR

Results so far

KVCR results so far

Early results from benchmarks and production.

Benchmark
DeepSeek V4 Pro · B200 · sticky routing
Mean TTFT1.95×faster
p99 TTFT on burst6.0×faster
Output throughput↑21.4%
Production, in-house
DeepSeek V4 Pro · 22 B200 nodes · agentic
TTFT p501.28×faster
TPOT1.37×faster
Benchmark
AgentX · Qwen3.8-27B · H200 · concurrency 128
Input tokens / s at high load↑25–50%
TTFT p901.6–3.4×faster
Production, external
KVCR 0.1.0 in production
KV cache reuse↑14%
KVCR-Guard

KVCR-Guard cuts failover tail latency

Qwen3-235B · B200 TP4 × 2 workers · 512 GB/worker · C128 · ISL 50K–200K (P50 82K) · internal multi-turn trace

TTFT during failover · seconds, lower is better

Before the fault: TTFT P50 / P90 / P95 = 0.69 / 1.71 / 2.21 s
Output throughput+6.5%458 vs 430 tok/s
Successful requests268 vs 248
Visible at the frontend210.4 s sooner217.6 s vs 428.0 s
Mechanism evidence85.94 GiB served53 transfers · promotion in 0.81 s
Primitives

Programmatic KV Cache Control

Primitives · Copy

Copy: fork KV for a sub-agent

Agent harnessspawns a sub-agentHARNESS METADATAsessionS1intentsub-agent forkISL / OSL48k / 2kstartsnowruns~350 sRouterA busy: load balance to BB has plenty of free KVpicks: copyWorker AbusyG1 HBMG2 DRAMKVCRreads from G2Worker Bplenty of free KVG1 HBMS1 loaded for the requestG2 DRAMcopy lands hereKVCRwrites into G2hint1request + fetch hint2copy3
The harness spawns a sub-agent that forks session S1. Press → to step.1 · The harness sends a hint: S1 forks into a sub-agent, 48k in, ~350 s of work.2 · A is busy, so the router load-balances to B and picks copy.3 · KVCR copies S1 from A’s G2 into B’s G2. B has free KV space and the copy is likely to be reused.
S1 in HBMS1 in DRAMtemporaryfreedother sessions
Primitives · Invalidate

Invalidate: drop context that isn’t needed

Agent harnesscompacts the contextHARNESS METADATAsessionS1eventcompactionISL96k → 8kkeepssummary + tailnext~80 sRouterOld blocks are deadCopies on A and Bpicks: invalidateWorker AbusyG1 HBMsummary keptG2 DRAMold context freedKVCRfrees blocksWorker Bholds a stale copyG1 HBMG2 DRAMstale copy freedKVCRfrees blockshint1invalidateinvalidate2
The harness compacts S1’s context. Press → to step.1 · The harness reports compaction: 96k becomes 8k, keeping the summary and the tail.2 · The old blocks are dead on both workers. The router frees every dead copy.
S1 in HBMS1 in DRAMtemporaryfreedother sessions
Primitives · No retain

No retain: use it once, leave nothing behind

Agent harnessone-off turnHARNESS METADATAsessionS1intentone-off turnISL / OSL30k / 500startsnownextnone soonRouterA: load is transientB: KV is contendedpicks: no copy, HBM onlyWorker A · rank 0load spikeG1 HBMG2 DRAMKVCRreads from G2Worker B · rank 1KV 92% fullG1 HBMS1 held for this requestG2 DRAMno S1, not demotedKVCRpulls to HBMhint1request + one-time fetch2pull once3
A one-off turn on S1. Press → to step. A and B can be workers or ranks in one server.1 · The harness says this is a one-off turn: nothing follows soon.2 · A’s load is only a spike and B’s KV is 92% full, so the router picks no copy, HBM only.3 · B holds S1 in HBM for one request, then drops it. No demotion, no pressure on B’s DRAM.
S1 in HBMS1 in DRAMtemporaryfreedother sessions
Primitives · Move

Move: copy and invalidate

Agent harnesssession wakes back upHARNESS METADATAsessionS1eventwake-upISL / OSL120k / 1kidle~40 minresumesnowRouterA busy, B idleSession needs one homepicks: moveWorker AbusyG1 HBMG2 DRAMKVCRreads from G2Worker BidleG1 HBMS1 loadedG2 DRAMS1 now hereKVCRwrites into G2freed after the copyhint1request + fetch hint2copy34
S1 wakes up after ~40 minutes idle. Press → to step.1 · The harness reports the wake-up: 120k of context, resuming now.2 · A is busy and B is idle, and the session needs one home: the router picks move.3 · KVCR copies S1 from A’s G2 to B’s G2, and B loads it into HBM.4 · A invalidates its G1 and G2 copies. The harness gave the event; the router inferred the move.
S1 in HBMS1 in DRAMtemporaryfreedother sessions
Primitives · Prefetch

Prefetch: copy ahead to a worker with room

Agent harnessnext turn is comingHARNESS METADATAsessionS1expectnext turnISL / OSL64k / 1kin~2 sRouterA busy, B has roomLow KV pressure on Bpicks: prefetch copy to BWorker AbusyG1 HBMG2 DRAMS1 is hereKVCRreads from G2Worker Blow KV pressureG1 HBMloaded at request timeG2 DRAMprefetch lands hereKVCRwrites into G2hint1prefetch hint, ahead of request2prefetch3
The harness knows S1’s next turn is about 2 seconds away. Press → to step.1 · The harness signals the next turn: 64k of context, in ~2 s.2 · A is busy and B has room, so the router sends a prefetch hint before the request arrives.3 · KVCR copies S1 into B’s DRAM ahead of time. B loads it into HBM when the request lands.
S1 in HBMS1 in DRAMtemporaryfreedother sessions
Primitives · Topology-aware

Topology-aware: KVCR inside an RDMA domain

Agent harnessspawns a sub-agentHARNESS METADATAsessionS1intentsub-agent forkISL / OSL48k / 2kstartsnowruns~30 sRouterW1 holds S1, busyW2: same RDMA domain, freepicks: copy to W2Server 1 · H1rdma_domain: D1W1busy, has S1W2idle · copy hereServer 2 · H2rdma_domain: D2W3idleW4idleKVCRfast path inside D1no RDMA path from D1hint12✕not picked: no fast pathcopy inside D13
A sub-agent forks S1, which lives on busy W1. Press → to step.1 · The harness spawns a sub-agent from S1.2 · W1 is busy. W2 shares its RDMA domain; W3 and W4 are idle but have no fast path. The router picks W2.3 · KVCR copies S1 to W2 inside domain D1.
S1 in HBMS1 in DRAMtemporaryfreedother sessions

Speaker notes

← → step · N notes · F fullscreen