AISim
Whole inference stack simulation

October 2026

AISim: We offer AIConfigurator today

Input · one command

bash

Fast:Simulate in 30 secs on a laptop
instead of days on actual GPUs.Static:One request shape, at steady load.

Use cases

Sizing GPU purchasesFewest GPUs that meet the SLA

Best config per modelParallelism, batching, agg vs disagg

SLA-based routingMatch requests to the right pool

Deployment tuningTest changes before production

Output

Qwen3-32B-FP8 on H200

Speaker notes

Starting point: AIConfigurator already ships. One command, about 30 seconds on a laptop, instead of days of benchmarking on GPUs.
Output is a Pareto frontier of configurations (agg vs disagg, parallelism, batch size) and the top configs. Toggle "Pareto plot / Top configs"; "Run again" replays the run.
Use cases: sizing GPU purchases, best config per model, SLA-based routing, tuning deployments.
The limitation, in the orange callout: it models one request shape at steady load. That sets up the next slide.

For developers

The terminal output is replayed from real aiconfigurator cli default runs (Qwen3-32B-FP8, 32× H200, TRT-LLM). The first tab is the README example: --isl 4000 --osl 500 --prefix 500 --ttft 300 --tpot 10, about 6.5 s on a laptop.
Method: op-level. Inference is broken into GEMM, attention, communication and element-wise ops; per-op times from a collected silicon database are composed by interpolation, then in-flight batching and disaggregated serving are modeled on top. --database-mode selects SILICON (default, reproducible), HYBRID, EMPIRICAL or SOL.
Modes: default, estimate, recommend (min GPUs for a load target), exp (YAML experiments), generate, support.
Links: AIConfigurator paper (arXiv 2601.06288) · Legacy CLI guide · Migration guide

AISim: But production traffic isn't static

AIConfigurator models

one ISL/OSL · fixed prefix · steady load

vs

Production traffic

bursts · mixed lengths · multi-turn and agentic reuse

AISim simulates the entire inference stack

Traffic:

AIConfiguratorassumes

AISimsimulates ● live

Traffic one ISL/OSL, steady

Traffic –

Queue not modeled

no waiting

Queue –

Scheduler one fixed batch

Scheduler –

KV cache fixed prefix

KV cache –

GPU kernelsmeasured timing · both use this–

Incoming load · prompt tokens arriving per second

Time to first token for each request, by arrival time

Speaker notes

Real traffic is bursty, mixes prompt lengths, and is increasingly multi-turn and agentic with prefix reuse.
AIConfigurator's steady-state assumption skips queueing, dynamic batching and KV cache pressure, which is exactly where tail latency comes from.
AISim simulates each layer request by request. Both tools share the same measured GPU kernel timing at the bottom.
Demo: click Steady, then Bursty, then Mixed prompt lengths. The time-to-first-token chart shows spikes a static model can't see.

For developers

The interactive chart is an illustrative toy model running in the page (simple per-op latency formulas for a 32-layer model). It is not AISim output; it only shows the qualitative effect of bursts and mixed lengths on queueing and TTFT.
Real equivalent: aisimulate predict -c prediction.yaml --capture-per-request. The traffic section sets source (synthetic lengths or a trace), load (concurrency, request rate, sessions per second) and stop. Per-request records go to requests.jsonl.
Links: Quickstart · Configuration reference · Understanding results

AISim: Forward-pass metrics (FPM)

FPM measures each whole model step for every batch shape. AIConfigurator adds up kernel estimates instead.

Then simulate any traffic by look-up

TrafficConcurrencyReplicasRoutingSLA

Self-service: weeks → ~1 hour 0.13 · 10/21

1 model1 GPU1 engine1 layoutStored locally

Speaker notes

Simple version: a model serves requests in quick steps. Instead of estimating how long a step takes, we time real steps for every batch size and shape once, and save the times in a table. The simulator then plays out any traffic and looks up each step's time in that table. Like timing a real kitchen cooking batches of 1, 5 or 20 orders, then using those times to plan any dinner rush, instead of adding up recipe times from a cookbook.

Top line: FPM measures each whole model step for every batch shape. AIConfigurator adds up per-kernel estimates (GEMM, attention, communication), which misses CUDA graphs, launch gaps and waiting on the slowest GPU. FPM's measured time includes all of that, which is why AISim is more accurate.
1 · One step: each bar is a request in the batch. Grey is tokens already in the KV cache; green is new tokens this step. Prefill requests bring many new tokens; decode requests add one token on top of a long context. The GPU runs model.forward() once over the whole batch, and we time it. The batch's shape (requests, new tokens, cached tokens) decides the time.
2 · The FPM table: decode is a 2D table (requests × KV context), because each decode request adds exactly one token. Prefill is 3D (requests × new tokens × cached tokens), because new tokens cost full compute while cached tokens only add attention reads; each stacked layer is more cached tokens. A mixed step is prefill time plus the extra decode cost. Between grid points, AISim interpolates.
Bottom left: the table isn't tied to a trace. Collection sweeps batch shapes, so the same table serves any traffic, concurrency, replica count, routing policy or SLA.
Bottom right, self-service (0.13, 10/21): onboarding a new model used to take weeks of NVIDIA engineering. With aisimulate onboard, customers collect FPM in about an hour on their own GPUs. It's quick because it measures one model, one GPU, one engine version and one parallelism layout, with no config sweep (the simulator does that later, offline). Each data point is a single forward pass, and the model is treated as a black box, so no engineering is needed.
Caveats if asked: the hour is GPU benchmarking time; cluster setup and validation are extra, and each additional parallelism layout is roughly another hour. A new model, GPU, engine version, precision or parallelism needs a new collection. Data stays local (Parquet plus metadata in the plan's systems/ folder). vLLM today, SGLang in 0.14.

For developers

FPM is a black-box model of the wall-clock time of one scheduler iteration (model.forward(batch)). Prefill is keyed by (requests, new tokens, cached KV tokens); decode by (requests, total KV context); mixed steps are prefill latency plus marginal decode cost. It uses the slowest rank's time, which captures uneven DP/EP load.
Selection: an FPM estimator needs a matching collected cell (model and revision, hardware, backend and version label, precision, exact topology). Otherwise prediction uses op-level timing. The executed estimator is reported per role in prediction_provenance (estimator, timing_source, data_version).
Onboarding flow: onboard init --model-config → onboard plan → onboard collect-fpm --execute --smoke → onboard collect-fpm --execute → onboard validate-collection → onboard finalize (if memory pending) → onboard validate-fpm / validate-serving. onboard resume restores a checkpointed session. An agent-guided procedure is documented.
Output: the plan directory holds request.yaml, support-plan.json, commands.json, predict/pilot.yaml, recommend/pilot.yaml and a local systems/ directory where the Parquet plus metadata pair is published; the generated configs point to it.
Limits: collection is vLLM only (SGLang, TRT-LLM not yet); text generation only for multimodal models; standard per-token KV caches (Llama, Qwen3, MiniMax-M2). Runs on Kubernetes or Slurm/Pyxis; it does not provision the cluster. FPM accuracy is not end-to-end accuracy; validate separately.
Links: FPM self-service guide · Implementation details · DEP #52 (FPM design) · FPM accuracy dashboard · FPE support matrix

AISim: Predictions match real GPUs

Qwen3.5-397B on B300, SGLang, 1k in / 1k out: output tokens/s per GPU. Measured by SemiAnalysis (InferenceX)

MeasuredAISim prediction TP2 + EP2TP4

1.5x to 1.6x

per-GPU throughput of TP2 + EP2 over TP4, measured. AISim predicted 1.4x to 1.6x.

Within 12%

measured vs predicted, every config and load.

2.4x to 3.1x

Our own B200 check (gpt-oss-120b): AISim's pick vs naive TP8, per GPU. Predicted 2.4x to 2.9x.

Production deployment within 5 to 10% of AISim's prediction.

Speaker notes

Evidence that the predictions hold up on hardware.
Qwen3.5-397B on B300 with SGLang, measured independently by SemiAnalysis (InferenceX). Solid lines are measured, dashed are AISim.
AISim predicted TP2 + EP2 beats TP4 by 1.4x to 1.6x per GPU; measured was 1.5x to 1.6x. Every config and load was within 12%.
Our own B200 check (gpt-oss-120b): AISim's pick delivered 2.4x to 3.1x the per-GPU throughput of naive TP8, vs 2.4x to 2.9x predicted.
A production deployment landed within 5 to 10% of AISim's prediction.

For developers

Chart data: Qwen3.5-397B on B300, SGLang, 1k in / 1k out, measured by SemiAnalysis InferenceX; dashed lines are AISim predictions for the same configs.
The gpt-oss-120b B200 check and the production deployment (5 to 10%) are the team's own runs; they are not published in the repo.
Public evidence: replay accuracy in the repo (e.g. MiniMax-M2.5 on 4× B200, 5,000-request Mooncake trace: TPOT error 2.8%, TTFT 13.5%, single run). The E2E dashboard aggregates many configs; aggregate TTFT error is still high (e.g. DeepSeek-R1 TTFT MAPE ~58%), so quote per-case results, not the aggregate.
Links: Replay accuracy evidence · E2E accuracy dashboard

AISim: one simulator, two layers, two modes

Two layers

Two modes one config file for both

Simulateaisimulate predict

What-if compareReplay a traceValidate sizing

Recommendaisimulate recommend

ParallelismBatchingReplicasAgg vs disagg

Speaker notes

How it's built. Two layers: an optional Dynamo layer (KV-aware router, Planner adding or removing workers) over the engine layer (vLLM, SGLang or TRT-LLM batching, KV cache with prefix reuse and host offload, TP/EP/DP, aggregated or disaggregated).
Everything is driven by measured GPU timing, A100 through GB300.
Two modes, one config file: predict replays a trace and returns every request's latency, for what-if comparisons and validating a sizing. recommend searches parallelism, batching, replicas and agg vs disagg, and returns the fewest GPUs that meet the SLA.

For developers

Stacks: --stack engine (default, built-in offline runner, no Dynamo dependency) or --stack dynamo (Dynamo-owned router and Planner via adapters). YAML router / planner sections require the Dynamo integration installed in the same environment; the validated pair is ai-dynamo 1.6.0.dev20260930 with aisimulate 0.13.0.dev.
predict writes prediction.json (plus requests.jsonl with --capture-per-request); --detail summary,memory,time,energy,source adds breakdowns. predict --online paces against wall clock where the stack supports it (engine stack is offline only).
recommend writes recommendation.json, a CSV and concrete recommendations/NNNN.yaml files you can replay with predict. Overrides use --set path=value; YAML is the only semantic surface.
Hardware coverage is per model/backend/version; check the FPE support matrix (refreshed nightly). B300 appears in the E2E accuracy data.
Links: Dynamo stack · Installation · Recommendation results · Adapter contracts

AISim: What you can ask

config.yaml

bash

Real runs from this week. Install today with pip install aisimulate.

Speaker notes

Three real questions, answered with real runs from this week. Click each tab.
Which config gets the most from my GPUs? A recommend sweep on B200: TP2 workers, about 1,200 output tokens/s per GPU within the latency target. We deployed the top picks on B200s to check them.
How many GPUs for 2,000 users? Predict says 400 H200s, 47 ms per token, 92% of requests meeting both targets.
Will it hold up under my real traffic? Replaying a trace on the Dynamo stack: 7 replicas carry it at 99% within SLA. KV-aware routing improves p90 time to first token but doesn't change the count.
Anyone can try it today: pip install aisimulate.

For developers

The three tabs are recorded outputs from the team's runs: aisimulate recommend -c gpt-oss-b200.yaml; aisimulate predict -c glm52-2000-users.yaml --set traffic.stop.requests=20000; aisimulate predict --stack dynamo -c qwen32b-trace.yaml --set engine.workers.aggregated.parallelism.replicas=$N --set router.policy=$ROUTER. The YAML files are internal, not in the repo.
Exit codes: 0 success, 1 execution failure or no feasible config, 2 config/validation error, 3 resource refusal.
Links: CLI reference · Quickstart

AISim: What else is newOn main today · ships in 0.13 (10/21)

Agentic trace replay

Replays multi-turn agent sessions with their dependencies and KV reuse.

2.8%TPOT error

0.3%KV reuse error

Mooncake trace · 5,000 requests · MiniMax-M2.5 on 4× B200

KV cache offload tiers

Simulates KV spilling to host memory and storage, with eviction. Sizes the CPU and SSD tier, not just the GPU count.

vLLM todaySGLang HiCache proposed

Deploy what it picks

A recommendation becomes ready-to-run manifests, plus an AIPerf job to benchmark it on real GPUs.

Dynamollm-d (Helm / Kustomize)Slurm

New serving shapes

Analytical what-ifs for encode / prefill / decode splits (multimodal) and attention-FFN disaggregation, before anyone builds them.

EPDAFDanalytical, fixed lengths

Also in predictions: modeled GPU power and energy (--detail energy), memory and KV capacity breakdown, and per-request records.

Speaker notes

Recent capabilities beyond the core simulator. All are working on the main branch today and ship in the 0.13 release on 10/21; the current stable release is 0.12.1.
Agentic trace replay: replays multi-turn agent sessions with their dependencies and KV reuse. On a 5,000-request Mooncake trace (MiniMax-M2.5, 4× B200), TPOT error was 2.8% and KV reuse error 0.3%.
KV cache offload tiers: simulates KV spilling to host memory and disk, so it sizes the CPU and SSD tier, not just the GPU count. vLLM today; SGLang HiCache is proposed.
Deploy what it picks: a recommendation becomes ready-to-run Dynamo, llm-d or Slurm manifests, plus an AIPerf job to benchmark it on real GPUs. Today this is a second step through the generator (legacy aiconfigurator CLI or SDK), not part of aisimulate recommend itself.
New serving shapes: analytical what-ifs for encode/prefill/decode splits (multimodal) and attention-FFN disaggregation. These use fixed request lengths, not traces, for now.
Predictions can also report modeled GPU power and energy, memory and KV capacity, and per-request records. Caveat if asked: power and energy come from per-kernel timing, so they aren't available yet for FPM-based predictions or for EPD/AFD.

For developers

Agentic replay: session dependency graphs, seeded snapshots and warmup, Weka and Mooncake traces, vLLM and SGLang, agg or P/D. Streaming observations and live warmup are in open PRs (#462 to #465). Replay features
KV offload: G2 host offload, agentic G2, G3 and Belady eviction for vLLM. SGLang HiCache G2 is proposed (#430).
Deployment generation: targets dynamo-j2, dynamo-python, llm-d-helm, llm-d-kustomize, fpm, plus Slurm bundles; produces k8s_deploy.yaml, bench_run.sh and k8s_bench.yaml (AIPerf). Today this goes through the legacy CLI or generator SDK. Deployment generation
EPD / AFD: analytical overlays only: fixed synthetic lengths, no traces, no deployment rendering. AFD writes afd-replay-spec.json and afd-qualification.json.
Power: power_w and power_coverage follow the modeled-power contract; numeric only at ≥90% coverage. Unavailable for FPM, fixed/polynomial, AFD and analytical EPD timing. Power contract

AISim: Q4 roadmap

October0.13 · 10/21

Onboard a new model yourself from measured forward passes (~1 hr on your GPUs)
Agentic trace replay with KV reuseNEW

November0.14 · 11/18

Measured-forward-pass modeling for SGLang
Multimodality support
Fractional GPU support (MIG, MPS)
Legacy aiconfigurator CLI retired: one aisimulate CLINEW

Nextproposed

Shared performance data on Hugging Face
Cost-aware search with your $/GPU-hour
Search across GPU types in one run
Custom SLOs and agent-assisted search

Speaker notes

October, 0.13 (10/21): forward-pass-metric self-service onboarding (about an hour on the customer's GPUs) and agentic trace replay.
November, 0.14 (11/18): forward-pass modeling for SGLang, multimodality, fractional GPUs (MIG, MPS), and retiring the legacy aiconfigurator CLI in favor of one aisimulate CLI.
Next (proposed, not committed): from open proposals in the GitHub repo. Shared performance data on Hugging Face, cost-aware search using your $/GPU-hour, searching across GPU types in one run, custom SLOs and agent-assisted search.

For developers

The 10/21 and 11/18 dates and the November items (SGLang FPM, multimodality, MIG/MPS) are internal plan; they are not tracked in public GitHub milestones. PR #413 bumps main to 0.14.0.
Legacy aiconfigurator removal in 0.14 is stated in the package README. The 0.13 wheel already removes the aiconfigurator / aiconfigurator_core Python import namespaces.
"Next" items come from open proposals: unified data on Hugging Face (DEP #435), cost optimization (#337), multi-GPU-type search (#338), custom SLOs (#339), agent-assisted search (#342). Long-range direction: Roadmap issue #1.

AISimWhole inference stack simulation

AISim: We offer AIConfigurator today

Use cases

AISim: But production traffic isn't static

AISim: Forward-pass metrics (FPM)

Then simulate any traffic by look-up

Self-service: weeks → ~1 hour 0.13 · 10/21

AISim: Predictions match real GPUs

AISim: one simulator, two layers, two modes

AISim: What you can ask

AISim: What else is newOn main today · ships in 0.13 (10/21)

Agentic trace replay

KV cache offload tiers

Deploy what it picks

New serving shapes

AISim: Q4 roadmap

AISim
Whole inference stack simulation