NVIDIA Dynamo · KubernetesDocs baseline: Dynamo v1.5.1

Planner

Grove

Planner decides how many engines to run. Grove decides what one engine instance is and where it lands.

Interactive: slides 7, 8, 9, 13, 15F full screen← → navigateN notes

What this deck covers. Two pieces of NVIDIA Dynamo's Kubernetes stack and how they work together:

Planner (dynamo.planner): the autoscaler. It decides how many prefill and decode engines a deployment should run.
Grove (ai-dynamo/grove): a Kubernetes API (PodCliqueSet, PodCliqueScalingGroup, PodClique, PodGang) that turns those counts into complete, gang‑scheduled model instances.

Sources. Behaviour is taken from the Dynamo v1.5.1 docs and checked against the Planner source on main (commit 7b0e63e) and the Grove API reference.

Flow. Slides 2–3 set up the problem, 4–9 explain Planner, 10–11 show the hand‑off to Kubernetes, 12–14 explain Grove, 15 ties it together in a simulator, 16–17 cover scale‑out and operations.

Keys. ← → navigate · F full screen · N toggles these notes.

The problem

Three problems a generic autoscaler can’t solve for LLM serving

Point of the slide: frame the three problems Dynamo has to solve. A CPU‑ or pod‑count autoscaler (e.g. a plain HPA) solves none of them. Each panel ends with what a solution needs; slide 3 shows which component provides it.

Two phases. Prefill processes the whole prompt at once and is compute‑bound; it determines time to first token (TTFT). Decode generates one token per step and is memory‑bandwidth‑bound; it determines inter‑token latency (ITL). In disaggregated serving they run on separate workers, so each needs its own replica count.
Demand is tokens. 10 req/s at ISL 8,000 is 16× the prefill work of ISL 500. Planner therefore forecasts ISL/OSL alongside request count, discounts ISL by the KV‑cache hit rate, and evaluates decode at context length ISL + OSL/2.
The unit of capacity can span nodes. A TP16 decode instance is two 8‑GPU pods. In a DGD this is multinode.nodeCount: 2 with gpu: 8 per node (total GPUs = nodeCount × gpu). Half an instance serves nothing, so capacity must be added in whole units.

Problems ① and ② are solved by Planner; problem ③ by Grove (next slide).

PlannerGrove

How Dynamo solves them: Planner + Grove

Maps the problems to components. Planner solves ① (separate prefill/decode counts sized on TTFT/ITL) and ② (forecasts request count, ISL and OSL, not just request rate). Grove solves ③ (adds and removes whole multi‑node instances, placed all‑or‑nothing).

Planner runs as a service inside the DynamoGraphDeployment (DGD), componentType: planner.

Inputs: Prometheus traffic metrics from the Frontend (dynamo_frontend_*) or a router, plus per‑iteration Forward Pass Metrics (FPM) that engines publish on the event plane.
Output: a replica count per role. The KubernetesConnector finds its parent DGD via DYN_PARENT_DGD_K8S_NAME, resolves services by subComponentType (prefill/decode) and patches spec.services[x].replicas.

Grove never receives a call from Planner. The Dynamo operator reconciles the DGD into Grove resources, and Grove plus KAI Scheduler place the pods.

So the DGD replica count is the contract. Anything that writes it (Planner, HPA/KEDA through a scaling adapter, a human) is handled the same way downstream.

Other connectors: virtual (non‑Kubernetes orchestrators poll decisions) and global-planner (slide 16).

PlannerHow it works today

Planner runs one pipeline every tick

Perf model from the first available:self‑benchmark→AI Configurator→profiling data→live FPM regression

Which proposers run depends onoptimization_target(next slide) ·advisory: truerecommends without scaling

The runtime is driven by OrchestratorEngineAdapter. Each tick runs a fixed pipeline:

Observe: worker counts, traffic metrics, FPM.
Predict: builtin_load_predict forecasts next‑interval request count, ISL and OSL.
Propose: builtin_throughput_propose (forecast → floor) and builtin_load_propose (live ±1). External gRPC plugins can propose too.
Reconcile / constrain: merge proposals, apply minimums and the GPU budget.
Execute: PlannerEffects.scale_to goes to the connector.

Cadence. The base tick (scheduling.scale_interval_seconds) defaults to the GCD of the enabled loop intervals, normally 5 s. Each plugin fires only when its own interval has elapsed.

Perf model. Capacity estimates come from a forward‑pass performance model (AISimulate). It is bootstrapped from the first available of: worker self‑benchmark, AI Configurator simulation (pre_deployment_sweeping_mode: rapid, default), real‑GPU profiling (thorough, 2–4 h), or live FPM regression. It keeps tuning itself from live FPM.

Advisory mode (advisory: true) runs everything but skips Execute; recommendations appear as dynamo_planner_* metrics, HTML reports and a live dashboard (port 8080).

PlannerConfig · optimization_target

Two ways to tell Planner when to scale

Key setting: optimization_target. It decides which loops run.

A · Scale on engine load (throughput default, latency, load): only the live loop runs (every 5 s). Code forces enable_load_scaling=True, enable_throughput_scaling=False, and ignores ttft_ms/itl_ms. Signals per engine:

Prefill: queued prefill tokens ÷ context length.
Decode: KV demand = (scheduled + queued KV tokens) ÷ max KV.
throughput: up at ≥ 1.0 / > 1.0, down below 0.1 / 0.6. latency: up at ≥ 0.1 / > 0.4, down when the queue is empty / KV < 0.1. load: prefill_scale_{up,down}_queue_tokens and decode_scale_{up,down}_kv_rate.
Scale‑up fires if any engine crosses the up threshold; scale‑down needs all engines under the down threshold plus a consolidation check.

B · Scale to latency targets (sla): you set ttft_ms (default 500) and itl_ms (50). The forecast loop is on by default; the live latency loop is opt‑in with enable_load_scaling: true. The rest of the deck assumes both are on.

Set these in a DGDR under spec.features.planner, or in the PlannerConfig JSON/YAML mounted into the Planner pod.

Plannersla target with load scaling on · both loops

How Planner makes the autoscaling decision

1 · every 180 s

Forecast a floor

floor = ⌈ forecast req/s
÷ req/s one engine
serves at the SLA ⌉

Capacity is ready before the demand arrives.

2 · every 5 s

Check live latency

▲ +1 every engine over SLA
▼ −1 load fits on N−1 engines
● hold otherwise

Estimated per engine from forward‑pass metrics.

3 · combine

Never below the floor

desired =
max(live decision, floor)

Short dips don't remove forecast capacity.

4 · guardrails

Then apply

no +1 while replicas start
≥ min_endpoint
GPU budget caps growth

Result is written to the DGD replica count.

The four steps Planner applies in sla mode with load scaling enabled.

Forecast floor (every 180 s, throughput_adjustment_interval_seconds). floor = ⌈forecast req/s ÷ req/s one engine serves within the SLA⌉. Each step moves at most ±8 replicas (max_throughput_scaling_replicas).
Live check (every 5 s, load_adjustment_interval_seconds). Per engine, estimate TTFT/ITL from FPM. +1 if every engine is over the SLA. −1 only if, after removing one engine (load × N/(N−1)), every engine still fits (slide 8). Otherwise hold.
Combine. desired = max(live decision, floor). The live loop can add above the floor but never cut below it, so a short lull doesn't remove capacity the forecast says is coming.
Guardrails. No scale‑up while replicas are still starting (scale‑down may cancel starting ones). Never below min_endpoint. At max_gpu_budget a scale‑up is held; forecast targets over budget are clamped proportionally (prefill floored first, decode gets the remainder).

The chart is a stylised illustration of steps 1–3 during a burst, not real data.

PlannerPredictive loop · optimization_target: sla

How the throughput loop picks a replica count

Predicted req/s

Avg ISL

Avg OSL

KV hit rate

TTFT SLA

ITL SLA

max_gpu_budget

Engines: prefill TP2 (2 GPUs), decode TP4 (4 GPUs). Curves are illustrative; real ones come from self‑benchmark, AI Configurator or profiling.

Worked example of step 1 (core/state_machine.py, core/throughput_scaling.py, core/perf_model/engine_query.py).

Forecast. Prometheus gives the last 180 s: request count, avg ISL, avg OSL (Frontend metrics), KV hit rate (router), spec‑decode accept length (engine). ARIMA (default; also constant, kalman, prophet) forecasts request count, ISL and OSL. KV hit and accept length reuse the last value. demand_rps = predicted num_req ÷ interval.
Capacity per engine. find_engine_capacity_rps queries the perf model. Prefill: ISL × (1 − KV hit, capped at 0.95); scans batch sizes, rps = batch ÷ TTFT, eligible if TTFT ≤ SLA. Decode: context = ISL + OSL/2; ITL = forward time ÷ accept length; rps = batch ÷ (OSL × ITL), eligible if ITL ≤ SLA. Highest eligible rps wins. If none meets the SLA it returns the best effort and logs a warning.
Divide. replicas = ⌈demand_rps ÷ engine_rps⌉, at least min_endpoint.
Constrain. Compare total GPUs with max_gpu_budget; clamp proportionally if over.

The sliders drive an illustrative curve (latency = base ÷ (1 − load)); real curves come from the perf model. If the model isn't ready, Planner holds the current count (model_not_ready).

PlannerReactive loop · sla mode

How the load loop decides +1, −1 or hold

Queued prefill tokens / engine

Imbalance across engines

New prompt ISL

TTFT SLA

Scale‑down sensitivity

Shown for prefill/TTFT. Decode uses scheduled + queued KV tokens → ITL (÷ accept length). ±1 per tick; no scale‑up while replicas are starting; never below the throughput floor.

Worked example of step 2 (core/load_scaling.py, _scale_decision).

Learn from forward passes. Engines emit FPM per iteration: wall_time, scheduled and queued prefill tokens, decode KV tokens, request counts. These tune the AISimulate perf model (up to max_num_fpm_samples = 64; ready after load_min_observations = 5). The scatter and straight‑line fit on the slide are a simplification.
Simulate the queue. TTFT for a new request = queued prefill tokens × (1 − KV hit) + its own ISL, split into chunks of max_num_batched_tokens, each chunk's forward time summed. ITL uses scheduled + queued decode KV, divided by accept length.
Decide. +1 if every engine's estimate > SLA. −1 only if, with load × N/(N−1), each engine's queue delay stays under (ttft_ms − own time) × load_scaling_down_sensitivity/100 (default 80). For decode: post‑consolidation KV must fit and ITL < itl_ms × sensitivity.

Hollow dots show each engine's estimate if one engine were removed. Try the imbalance slider: one hot engine is enough to block both scale‑up (not all over) and scale‑down.

PlannerInteractive · illustrative capacities

The decision rules, playing out over an hour

Cursor

① Forecast every 180 s sets the floor Target replicas · shaded = ready (~60 s start) ▲ ▼ live ±1 decisions Click the timeline → cards show the math at that moment

Interactive timeline. One simulated hour; the same rules from slides 6–8 applied tick by tick.

Top: observed req/s (green) and the 180 s forecasts (amber dashes).
Prefill / Decode: target replicas (solid), ready replicas (shaded; new replicas take ~60 s), forecast floor (amber dashes), ▲/▼ where the live loop changed the count.
Cards: click the timeline or drag the cursor. Left card shows the last forecast calculation. Right card shows three engines against the SLA (or thresholds) and the decision with its reason.

Things to point out: during a burst the live loop adds +1 steps before the next forecast raises the floor. "hold · starting" is the pending guard. Switch to throughput (default) to see threshold‑only behaviour with no floor.

Simplifications: prefill engine = 2 GPUs serving 9.1 req/s at SLA; decode engine = 4 GPUs serving 21 req/s; budget 40 GPUs; ARIMA approximated by a trend fit; threshold signals modelled from utilisation. Flapping in default mode under steady traffic is an artifact of that model.

PlannerGroveActuation path

One scale decision, end to end

The full actuation path for one decision (decode 2 → 3):

Planner calls set_component_replicas().
DGD spec.services.decode.replicas becomes 3. If the service has scalingAdapter enabled, a DynamoGraphDeploymentScalingAdapter (DGDSA) owns replicas via the Scale subresource, and a webhook blocks direct edits. Use one autoscaler per service (Planner or HPA/KEDA).
Dynamo operator reconciles the DGD into Grove resources.
Grove raises the PodCliqueScalingGroup to 3 and creates a new scaled PodGang.
KAI Scheduler binds the PodGang all‑or‑nothing, honouring topology constraints; if it can't fit, it stays Pending.
Pods start in the declared order (startsAfter), register with Dynamo discovery, and Planner sees them as ready capacity.

Inspect it: kubectl get dgd,dgdsa then kubectl get pcs,pcsg,pclq,pg,pod.

GroveResource mapping

What Planner’s number actually scales

replicas = complete instances · nodeCount = pods per instance · each instance = one PodGang

How the DGD you write becomes Grove objects (Grove is the default multinode orchestrator; set nvidia.com/enable-grove: "false" to use LWS + Volcano).

One DGD → one PodCliqueSet.
Single‑node service (e.g. prefill, 1 pod per engine) → one PodClique; replicas = number of pods.
Multinode service → one PodCliqueScalingGroup with a leader PodClique and a worker PodClique. replicas = complete instances; multinode.nodeCount = pods per instance (1 leader + nodeCount − 1 workers).
GPUs per instance = nodeCount × gpu; the engine's TP size must match (16 here).

Planner's replica number therefore lands on a PodClique for single‑node roles and on a PodCliqueScalingGroup for multinode roles. Grove also supports scaling the whole PodCliqueSet (service copies) and individual PodCliques.

Resource names on the slide are illustrative; the operator derives them from DGD and service names.

LeaderWorkerSetGroveSame service, two views

What does the scheduler treat as “one unit”?

Example service: 2 prefill + 1 decode instances, each instance = a leader pod + a worker pod.

LWSGrove

Leader + workers of one instance placed together✓✓

Prefill + decode placed together (a service that can actually answer)✗✓

Startup order across roleswithin a group✓

Scale one role without touching the other✓✓

Same service drawn twice: 2 prefill instances and 1 decode instance, each a leader + worker pod.

LeaderWorkerSet (kubernetes‑sigs/lws) models one multi‑pod role well: each group (leader + workers) can be gang‑scheduled by Volcano or KAI. A disaggregated service needs one LWS per role, and nothing links them, so the scheduler sees three independent units. Startup ordering (LeaderCreated / LeaderReady) applies only inside a group.

Grove describes the whole service as one PodCliqueSet and builds PodGangs from it:

Base PodGang (API role Anchor): all standalone PodCliques plus the first minAvailable replicas of each PodCliqueScalingGroup. Placed all‑or‑nothing: the smallest service that can answer a request.
Scaled PodGangs (role Tail): one per replica above minAvailable, each gang‑scheduled on its own.
minAvailable also drives gang termination: if a group falls below it for longer than TerminationDelay, the gang is torn down and rescheduled.
Cross‑role startup order: cliqueStartupType AnyOrder (default), InOrder, or Explicit with startsAfter.

LWS remains a good fit for single‑role multi‑node serving; the gap is cross‑role viability.

LeaderWorkerSetGroveInteractive · step through

Why it matters when GPUs are scarce

LeaderWorkerSet

Grove

Step through two scenarios with ◀ / ▶. Assumptions: 6 nodes; each prefill or decode instance needs 2 nodes (leader + worker); LWS groups are gang‑scheduled one at a time in arrival order; Grove schedules each base PodGang as one unit.

① Tight capacity (3 free nodes, service needs 4). LWS places prefill because that group fits, then decode waits: 16 GPUs are held and no request can be served (prefill has nowhere to send KV). Grove places nothing, so the 3 nodes stay available to other work, then places all 4 pods together when a node frees.

② Two services race (4 free nodes, A and B each need 4). LWS places A·prefill, then B·prefill, and both decodes wait forever: a resource deadlock with 32 GPUs held and both services down. Grove places A whole; B waits cleanly.

Scale‑out of an already running service behaves the same in both systems (one new gang per instance). The difference is only the cross‑role base gang.

GroveOn every scale event

Scale‑out adds a complete instance

Gang scheduling

Whole instance or nothing

Hierarchical gangs

Never prefill without decode

Startup ordering

Roles start in declared order

Topology‑aware

Set via topologyConstraint

Gang termination

Capacity count stays honest

Coherent updates

Prefill + decode roll together

Summary of what Grove adds to every scale event Planner triggers:

Gang scheduling: each instance is a PodGang; all pods bind or none do.
Hierarchical gangs: the base PodGang guarantees a viable service (≥ 1 prefill + ≥ 1 decode) before extras.
Startup ordering: cliqueStartupType / startsAfter, e.g. workers before leader (MPI) or router before workers.
Topology‑aware placement: pack an instance within a rack, block or NVLink domain; spread replicas for availability. Exposed on the DGD as topologyConstraint; a PCSG constraint must be equal to or stricter than the PodCliqueSet's.
Gang termination: below minAvailable (after TerminationDelay), the gang is removed as a unit, so Planner's ready count reflects real serving capacity.
Coherent updates: select the PodCliqueSet update strategy from the DGD with nvidia.com/grove-update-strategy (RollingRecreate or OnDelete).

Version pairing for Dynamo 1.5.x: KAI Scheduler ≥ v0.17.0, Grove ≥ v0.1.0‑alpha.13 when installed separately.

PlannerGroveInteractive · simplified model

Watch a scale decision reach the cluster

Try Burst: the load loop steps +1 before the floor rises. Budget 40 → clamping. Budget 80 + peak 70 + TP16 → the extra decode PodGang stays Pending, never half‑placed. 1 tick = 5 s; throughput loop every 60 s here. Numbers are illustrative.

Traffic

Peak req/s

ISL

OSL

TTFT SLA

ITL SLA

GPU budget

Decode shape

Speed

Throughput loop Load loop

Sim time

0:00

Traffic

0 req/s

GPUs / budget

–

Est. TTFT

–

Est. ITL

–

SLA met

–

Traffic vs forecastreq/s

observedPlanner forecast

Replicas: target vs readyengines

prefill targetreadydecode targetready

Latency as % of SLA100% = SLA

TTFTITLSLA

Cluster · 8 nodes × 8 GPUs–

prefilldecodestriped = startingoutlined = waiting

Grove resources (from DGD)PodGangs

Event lognewest first

End‑to‑end sandbox: Planner → DGD → Grove → KAI → pods on an 8‑node × 8‑GPU cluster (two racks).

Controls: traffic pattern, peak req/s, ISL/OSL, TTFT/ITL SLA, GPU budget, decode shape (TP16 two‑node PCSG vs TP4 single‑pod PodClique), loop toggles, speed.
Charts: traffic vs forecast, target vs ready replicas, latency as % of SLA.
Cluster: GPUs coloured by role; striped = starting, outlined = leader waiting on startsAfter.
Event log: each line is tagged PLANNER, OPERATOR, GROVE, KAI or POD so you can follow one decision through every layer.

Try: Burst (live loop reacts first), budget 40 (clamping), budget 80 with peak 70 and TP16 (a decode PodGang stays Pending instead of half‑placing).

Simplifications: 1 tick = 5 s; forecast every 60 s instead of 180 s; linear‑trend predictor; made‑up engine capacities and latency curves; rack packing is "preferred", not required.

PlannerBeyond one DGD

One Planner per DGD; GlobalPlanner coordinates many

A local Planner manages exactly one DGD. GlobalPlanner centralises scaling across several.

Pattern 1 · shared policy: independent DGDs (often different models) keep their own endpoints. Each local Planner sets environment: global-planner and global_planner_namespace; GlobalPlanner authorises callers (--managed-namespaces) and enforces a total budget (--max-total-gpus). No GlobalRouter needed.
Pattern 2 · one endpoint, many pools (shown): a control DGD (Frontend + GlobalRouter + GlobalPlanner) fronts private pool DGDs. GlobalRouter picks a prefill pool by ISL and TTFT target, and a decode pool by context length and ITL target. Pool Planners must use throughput_metrics_source: router so they forecast from their own LocalRouter.

Grove is unchanged: each pool DGD is its own PodCliqueSet with its own gangs.

Limitation today: one DGDR does not generate the multi‑pool topology; profile each pool separately and compose the DGDs by hand.

Operating notes

Things that bite in practice

Δt

Interval ≥ time‑to‑ready

Otherwise ticks just hold. Multinode gangs are slowest.

Averages hide tails

Bimodal ISL? Split into pools under GlobalPlanner.

One autoscaler per service

HPA on Frontend + Planner on workers. Never both on decode.

≥1

No scale‑to‑zero

Zero workers = model leaves /v1/models, no signal to return.

≠

Budget ≠ cluster

Planner caps the ask; KAI decides what fits. Misfits stay Pending.

1.5

Version pairing

KAI ≥ v0.17.0 · Grove ≥ v0.1.0‑alpha.13 when installed externally.

Interval ≥ time‑to‑ready. If an interval is shorter than scheduling + model load + registration, ticks see a transition in progress and hold. Multinode gangs are slowest; size throughput_adjustment_interval_seconds accordingly.
Averages hide tails. The forecast uses mean ISL/OSL. For bimodal traffic, split into pools behind GlobalPlanner.
One autoscaler per service. HPA/KEDA on the Frontend plus Planner on workers is fine; two autoscalers on decode will fight.
No scale‑to‑zero for workers. With zero workers the model disappears from /v1/models, so there is no demand signal to scale back up. Keep min 1.
Budget ≠ cluster capacity. max_gpu_budget (default 8) caps what Planner asks for; KAI decides what fits. A gang that doesn't fit stays Pending and Planner holds that role.
Version pairing. Dynamo 1.5.x with external installs: KAI ≥ v0.17.0, Grove ≥ v0.1.0‑alpha.13.

Useful signals: dynamo_planner_estimated_ttft_ms, dynamo_planner_estimated_itl_ms, dynamo_planner_predicted_num_prefill_replicas/_decode_replicas, and the load‑scaling decision enum.

SourcesChecked October 2026

References

Planner Design (v1.5.1) Planner Guide · PlannerConfig Global Planner Deployment Guide Autoscaling with DGDSA Grove in Dynamo docs Multinode Orchestration (versions) Grove API reference (PodGang, minAvailable) Dynamo‑Grove on AKS (Part 4)

Interactive slides are teaching models: documented control logic, made‑up capacities and latency curves.

Planner × Grove · NVIDIA Dynamo