NVIDIA Dynamo · AIPerf

AIPerf

Benchmark any inference endpoint with production-shaped traffic, and get the metrics that decide your SLA.

pip install aiperfApache 2.0successor to GenAI-Perf

Why it's hard

A benchmark that doesn't look like production tells you the wrong answer

Synthetic loadMooncake trace

success rate

100%

96%

tokens / s

22.5K

4.7K

same server · same config

Same server, two answers. Real prompt lengths and bursts cut throughput ~5× and exposed context-window failures that uniform synthetic prompts hide.

🧵

The client is the bottleneckad-hoc scripts are GIL-bound; AIPerf has to saturate a real server without slowing itself down

🎯

Throughput isn't goodputhigh tokens/s can hide that most requests missed their SLA

🧊

Cold starts liefirst requests warm caches and compile; they must not count

🤖

Agents aren't chatsmulti-turn sessions, subagents and tool pauses change KV reuse and load

Architecture

Three planes, so measuring never gets in the way of sending

~10 services in separate processes, talking over ZMQ. Press → to follow one request.

① Credit. The Timing Manager sends a credit, permission to send exactly one request, to a worker.

② Data. The worker reads its prompt from memory-mapped dataset files. No copies, no waiting.

③ Request. The worker sends it to your server and timestamps every streamed chunk.

④ Return. The credit goes back to the Timing Manager. A slow server means fewer free credits: natural backpressure.

⑤ Records. Raw results go to Record Processors, which scale with load and compute per-request metrics.

⑥ Results. The Records Manager aggregates and exports, alongside GPU and server metrics.

ZMQ pub/sub + ROUTER/DEALERwarm-up discardedsingle node or Kubernetes

From docs/architecture.md. Control plane: SystemController, Timing Manager, Dataset Manager, Worker Manager ("decides what, when, and how many requests to send"). Data plane: Workers and the inference server. Analytic plane: Record Processors, Records Manager, GPU Telemetry Manager, Server Metrics Manager. A telemetry sidecar streams to OTel and MLflow.
Why ZMQ: decoupling orchestration from execution keeps measurement accurate; workers spend their time on I/O and timing. Credits dispatch over ROUTER/DEALER with sticky sessions or least-loaded selection; returns come back on a PUSH/PULL fan-in.
README says 10 services; the plugin registry now lists 12 including an optional API service, network-latency probe and a Kubernetes worker-group manager.
Lifecycle: initialization → warm-up (discarded) → profiling → collection → processing → aggregation.

Load control

Shape the traffic exactly: closed loop, open loop, or replay a trace

Mode

Arrivals

--arrival-smoothness0.4

Ramp--concurrency-ramp-durationWarm-up--warmup-request-countCap prefill--prefill-concurrencyUsers--user-centric-rate --num-usersCancel--request-cancellation-rate

Find the limit

Don't guess the operating point. Sweep it, or let AIPerf search for it.

Method

TTFT SLA (ms)600

Workloads

From synthetic prompts to real agent sessions, byte for byte

Synthetic

Shape itISL/OSL distributions, shared prefixes, cache busting--isl 4000 --osl 500

Public datasets

Grab oneShareGPT, AIMO, MMStar, VisionArena, SPEED-Bench, ASR sets--public-dataset sharegpt

Production traces

Replay itMooncake · Bailian · Baseten · BurstGPT · SageMaker Data Capture--custom-dataset-type mooncake_trace

Agentic

Agents, for realWeka Claude Code sessions · TraceLab · H CUA · DAG sessions with subagents and tool pauses--scenario inferencex-agentx-mvp

Multimodal

Beyond textvision, audio/ASR, image generation and edit, video generation--endpoint-type image_generation

Endpoints

19 typeschat, completions, responses, messages, embeddings, rankings, raw, template · HTTP or SageMaker--endpoint-type chat --streaming

Metrics

Every token is timed. Goodput counts only the requests that met the SLA.

TTFT SLO (ms)600ITL SLO (ms)30

throughput–all 60 requests

goodput–

👆 Click any dot to see that request above. Drag the SLOs to move the box.

Output

One run: a console table, exports, plots, and live telemetry

Every run exports artifacts/<run>/profile_export_aiperf.csv · .json · per-request .jsonl · parquet · timeslices

Percentile

real aiperf plot --theme dark example from the docs · numbers = concurrency · hover a point

--ui-type dashboard · default in interactive terminals · live while the run is going

collect --gpu-telemetry DCGM · --server-metrics on by defaultstream to --otel-url · --mlflow-tracking-uri · --wandb-project

Press → to step through the four outputs (or click one). Exports are written on every run. The Pareto plot is redrawn from the dark-theme example in docs/tutorials/plot.md (Qwen3-0.6B vs Llama-3-8B, concurrency 1–32, values read from the image); the GPU telemetry chart is redrawn after the docs' dark-theme "Output Token Throughput with GPU Utilization" example (synthetic data with the same shape); the live dashboard values and telemetry curves on this slide are illustrative; the console table is the README example.
Console table from the README example (CPU-only Ollama granite4:350m, concurrency 5, 10 requests; the README notes it is not an official benchmark). Real output has columns avg / min / max / p99 / p90 / p50 / std.
Exports: profile_export_aiperf.csv and .json plus per-request JSONL; --export-level summary, records (default) or raw. aiperf plot writes PNG charts including Pareto curves of throughput per GPU vs latency and vs interactivity (--auto-plot). --ui-type dashboard is the default in interactive terminals. GPU telemetry via DCGM (or pynvml / amdsmi); server metrics scraped from /metrics by default; OTel, MLflow and W&B extras.

At cluster scale

Run it in the cluster, next to the model, as a Kubernetes job

helm install aiperf-operator \
  oci://ghcr.io/ai-dynamo/charts/aiperf-operator \
  -n aiperf-system --create-namespace

# one benchmark, or an in-cluster search
aiperf kube profile ...
aiperf kube sweep ...
aiperf kube dashboard

Beta in v0.13.0. CRDs: AIPerfJob, AIPerfSweep.

Who uses it

The harness behind public benchmarks and cloud recommendations

SemiAnalysis · InferenceX

Runs the AgentX benchmarkthe vendor-neutral client that sends the benchmark traffic: real Claude Code sessions replayed as DAGs, same harness version for every submission

AWS · SageMaker AI

Powers inference recommendationspicked after evaluating tools, for detailed, consistent metrics; AWS contributed multi-run confidence and adaptive convergence upstream

Also contributingCoreWeaveW&B export, sweep reliabilityBasetentrace replayPinterestDAG benchmarking

1.75M+pip installs · last 6 months

152Kpip installs · last 30 days

818GitHub stars

v0.13latest release · Sep 2026

SemiAnalysis InferenceX glossary: AIPerf is "the vendor-neutral client that sends the benchmark traffic"; agent sessions are rebuilt as DAGs (main turns, parallel subagent branches, tool-use pauses); all submissions for a model use the same harness minor version.
AWS ML blog (Apr 22 2026): SageMaker AI inference recommendations use AIPerf to benchmark each optimized configuration on real GPUs; AWS contributed multi-run confidence reporting and adaptive convergence / early stopping.
CoreWeave, Baseten and Pinterest are credited contributors in the NVIDIA developer blog acknowledgements, not stated production adopters. Stars as of Oct 25 2026. pip installs from pypistats.org (PyPI downloads, mirrors excluded): 1,752,895 from Apr 8 to Oct 24 2026 (the 180 days pypistats keeps), 152,180 in the last 30 days. Downloads include CI and repeat installs, so they overstate unique users.

Get started

One install, one command

Quick start

pip install aiperf

aiperf profile \
  --model Qwen/Qwen3-0.6B \
  --url localhost:8000 \
  --endpoint-type chat --streaming \
  --concurrency 8,16,32,64 \
  --goodput "time_to_first_token:600" \
  --auto-plot

Next steps

📚

Comprehensive guidefive real-world benchmarking use cases

🧪

Trace replaybring a production trace and replay it

🐙

GitHubai-dynamo/aiperf · source, issues, releases

📖

Docsdocs.nvidia.com/aiperf · CLI, metrics, tutorials