AISimulate

Feature Support Matrix

Serving techniques AISimulate simulates for each inference engine. Supported means the technique is replayed within the limits on its linked page; Limited means common configurations are rejected or only an analytical estimate exists, as the row's note says. Techniques supported on their own do not automatically combine: Replay feature support lists the composition boundaries, and the AgentX section covers agentic traffic.

Branch main · commit 00e3b8f35 · source features.toml

SupportedLimitedNot supportedN/A upstream

Topology and parallelism

TechniquevLLMSGLangTensorRT-LLMNotes
Aggregated servingSupportedSupportedSupportedEngine-specific scheduler and cache admission.
Prefill/decode disaggregationSupportedSupportedSupportedIndependently sized prefill and decode pools.
Tensor and expert parallelismSupportedSupportedSupportedPriced by the performance model; see the FPE matrix per model and GPU.
Attention data parallelismSupportedSupportedSupportedRank-local scheduling; a grouped pass ends at the slowest rank.
Context parallelismLimitedLimitedNot supportedPrediction only; decode CP depends on the model family.
Attention–FFN disaggregationLimitedanalyticalLimitedanalyticalLimitedanalyticalAnalytical estimate only, for fixed-length synthetic requests; no trace replay.
Encoder disaggregation (multimodal)LimitedanalyticalLimitedLimitedanalyticalSGLang replays static Qwen3-VL encoder pools event by event; the analytical estimate adds mean encoder latency for Qwen3-VL and Kimi K3 towers.

Scheduling and batching

TechniquevLLMSGLangTensorRT-LLMNotes
Continuous batching, chunked prefillSupportedSupportedSupportedEngine-specific admission and token budgets; SGLang --enable-mixed-chunk (off upstream by default) is not modeled.
Prefix caching (G1 reuse)SupportedSupportedSupportedBackend block and admission rules apply.
Sub-block prefix reuseLimitedhybrid KVSupportedNot supportedvLLM reuses whole blocks, except hybrid KV models at prefix_match_unit; SGLang matches per token at page size 1; TensorRT-LLM's default partial block reuse is not modeled.
Hybrid KV cache (per-request state)LimitedNot supportedNot supportedKimi K3 and DeepSeek V4 state in G1, aggregated and P/D; prefix reuse resumes from state snapshots. No G2/G3 or recommend.
Frontend tokenization costSupportedSupportedSupportedEach worker's tokenizer servers add a fixed plus per-input-token delay.
Response detokenization costSupportedSupportedSupportedEach emitting worker's detokenizer servers add a fixed plus per-output-token delay before the client sees tokens.

Speculative decoding

TechniquevLLMSGLangTensorRT-LLMNotes
MTPLimitedLimitedLimited1–5 drafts with a supplied acceptance rate; TensorRT-LLM keeps its legacy scheduling approximation.
EAGLE-3, DSPARKLimitedLimitedNot supportedLinear 1–5 token drafts with a supplied acceptance rate; no tree drafting; model-family gates apply.
DFlash, draft modelLimitedNot supportedNot supportedLinear 1–5 token drafts from a draft checkpoint; DFlash covers Llama and DeepSeek V4.
N-gram prompt lookupLimitedNot supportedNot supportedFixed draft length and acceptance assumption.

KV cache hierarchy

With KVCR, a worker fetches a cached prefix from a peer worker's G2.

TiervLLMSGLang / HiCacheTensorRT-LLMKVCR vLLMKVCR SGLangKVCR TensorRT-LLMNotes
G1Device memorySupportedSupportedSupportedN/A upstreamN/A upstreamN/A upstream
G2 localHost memory per DP rankLimitedNot supportedNot supportedN/A upstreamN/A upstreamN/A upstreamRequires prefix caching; generic speculation schemes are rejected.
G2 sharedHost memory pool across workersLimitedNot supportedNot supportedLimitedpeer fetchN/A upstreamN/A upstreamA shared pool needs matching geometry on every role. KVCR needs router hints (idle on CLI traffic), aggregated or P/D prefill pools, no G3.
G3 localStorage per workerLimitedNot supportedNot supportedNot supportedN/A upstreamN/A upstreamAggregated worker with attention DP 1 only; no P/D, agentic traffic, speculation or recommend.
G3 sharedStorage shared by workersLimitedNot supportedNot supportedN/A upstreamN/A upstreamN/A upstreamSame limits as G3 local.

KV transfer and routing

TechniquevLLMSGLangTensorRT-LLMNotes
P/D KV transfer timingSupportedSupportedSupportedConcurrent transfers share per-rank links max-min fairly. vLLM and TensorRT-LLM reserve source-first; SGLang destination-first.
KV-aware routing (Dynamo Router)SupportedSupportedSupportedRequires the optional Dynamo stack.
Autoscaling (Dynamo Planner)SupportedSupportedSupportedRequires the optional Dynamo stack.

Workloads

TechniquevLLMSGLangTensorRT-LLMNotes
Synthetic and trace replaySupportedSupportedSupportedMooncake and compatible trace formats.
Multi-turn sessionsSupportedSupportedSupportedThink time between turns.
Vision-language on the language workerNot supportedLimitedNot supportedFixed synthetic images, PP 1 and attention DP 1; frontend costs from a host cost table.

AgentX (agentic replay)

Offline engine stack only. Results are qualified functional_only, not hardware accuracy. Rows in other sections do not automatically compose with agentic traffic; these rows do.

TechniquevLLMSGLangTensorRT-LLMNotes
Weka, Agentic Mooncake and agentic Dynamo tracesSupportedfunctionalSupportedfunctionalNot supportedDependency-graph replay in lanes; Weka is the AgentX source format.
Seeded snapshotsSupportedSupportedNot supportedCut sampled uniformly between 25% and 75% of each play; import AIPerf cuts for parity runs.
Warmup, continuous profileLimitedLimitedNot supportedSequential lane recycling only (no random or shuffle); AIPerf --cache-bust is not implemented; engine.detokenization is rejected with a profile.
G2 host offloadLimitedNot supportedNot supportedOne aggregated worker or 1P1D, attention DP 1 on every role, static pools, no speculation.
G3 offload, KVCRNot supportedNot supportedNot supportedG3 is rejected; KVCR stays idle without router hints.
Speculative decodingLimitedHBM onlyLimitedHBM onlyNot supportedMTP and EAGLE-family schemes with KV kept in GPU memory.
KV-aware routing, PlannerNot supportedNot supportedNot supportedThe engine stack places requests round-robin without conversation affinity; AgentX on the Dynamo stack needs downstream qualification; agentic formats require planner.policy: disabled.

Combining techniques

Supported techniques are not automatically supported together. Replay feature support lists each composition boundary.

Which models and GPUs? →

The FPE Support Matrix covers model, GPU, backend and version.

How accurate? →

E2E and FPM accuracy dashboards report error against GPU runs.