Benchmarking

Use this chapter to measure the streaming response plane. It describes the four in-repo harnesses, the external serving rig, and the measurement rules that earlier mistakes made necessary. Response plane performance has the results. To compare two commits on one machine, see Review for performance.

Select the harness

The first two in-repo harnesses model the same serving shape. Anchor hosts stand in for frontends and own one response stream per request. Engines run continuous batching: each forward pass puts exactly one token on each active request's stream. Request dispatch is an in-process channel. The response path is real: anchors live on different workers from the engines, so every attach and every token crosses a loopback TCP socket.

Run batched_streaming

Run all commands from the examples/ directory:

cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 24 --max-batch 8 --tokens 40

The example has three anchor hosts. It fails the run if any request misses a token, sees an unexpected frame, or if the batcher's velo_streaming_mux_records_per_batch{direction="sent"} does not account for every token.

Keep --pass-delay-ms above zero for a per-stream comparison. With no gap, the engine runs ahead, several tokens queue on each stream, and per-stream coalescing packs them. That flatters the per-stream path and hides the effect under test.

To reproduce the flush-policy comparison in Batched streaming, run each configuration five times with --flush-policy auto and --flush-policy manual:

cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 96 --max-batch 32 --tokens 40 --flush-policy auto --initial-credit 256
cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 96 --max-batch 32 --tokens 40 --flush-policy manual --initial-credit 256
cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 24 --max-batch 8 --tokens 40 --pass-delay-ms 0 --flush-policy auto --initial-credit 256

CI runs batched_streaming at the defaults with both flush policies.

Run response_plane_bench

Run all commands from the examples/ directory:

cargo run --release --all-features --example response_plane_bench -- --anchor-hosts 2 --engines 128 --requests 2000 --credit-sweep-interval-ms 200

CAUTION: Set --credit-sweep-interval-ms 200 to match velo's default. The harness default of 2 ms reproduces an earlier default and exists for the sweep A/B.

The harness reports TTFT and ITL as HDR histograms (p50, p95, p99). It also reports requests per second, tokens per wire write and process CPU. CPU is split into user and system time from /proc/self/stat. The consumer asserts the position of every token, so a run fails if sharing a batch ever reorders a stream.

Sweep --engines, not --anchor-hosts. The per-peer costs on a frontend scale with its ingress peers, and an anchor host's ingress peers are the engines that stream to it. --anchor-hosts moves the smaller side of the same product.

The harness agrees with batched_streaming. At --anchor-hosts 3 --engines 2 --requests 96 --max-batch 32 --tokens 40 --initial-credit 256, it reports 5.41 tokens per write where batched_streaming reports 5.38, both at a credit window of 256.

Limits of the in-process harness

• One process. The reported CPU belongs to the whole topology, not to one frontend. An A/B that changes one frontend-side setting is valid. A comparison with another system's per-frontend CPU is not.
• Loopback. It removes wire time, which exaggerates the syscall term in favor of the mux.
• Size. It runs out of memory above about 256 nodes in one process on a 374 GiB host. A 512-worker shape needs the external rig.
• Queueing. With --max-batch well below --requests, most of TTFT is the wait for a batch slot. Read TTFT and requests per second only as comparisons between arms.
• Raw percentiles. The harness reports percentiles over every recorded request after --warmup-requests. It does not separate steady state from the opening burst. See Steady-state and raw percentiles.

Run stream_lifecycle

Run all commands from the examples/ directory. The harness runs two nodes in one process over loopback TCP, on the messenger mux, or on one TCP connection per stream with --path per-stream. The producer opens short streams, finalizes each one, and then does what --mode says:

Wall time moves with the load on the machine. Compare builds and modes by the instructions of the process, divided by --streams, and take the median of several runs. On macOS:

cargo build --release --example stream_lifecycle
/usr/bin/time -l target/release/examples/stream_lifecycle --mode plain

On Linux, use perf stat -e instructions instead of /usr/bin/time -l. Batched streaming has the results for the mux.

On the per-stream path (--path per-stream), keep --streams to a few thousand and leave time between runs: each stream opens a connection, and the ephemeral ports run out. Provisional, on the same laptop and under the same conditions as the mux table, with --streams 4000 --warmup 400, the two modes interleaved, median of five: plain 525,586 instructions per stream (520,910 to 526,380) and retired 527,360 (526,161 to 529,770). The spreads overlap, so awaiting retired() costs nothing this run can resolve; the pump reports the retirement in both modes, and most of each figure is the connection each stream opens. The producer holds a stream for about 130 µs p50 and 500 µs p99 after finalize while it waits for the write. To be confirmed on dedicated hardware.

Run anchor_setup_bench

Run all commands from the examples/ directory. The harness has no network and no mux, so its numbers show the cost of the anchor path only. It runs two phases, each on its own runtime and manager:

1. Create --held SPSC anchors and hold them unattached. The harness reports the create time of each anchor, and the tokio tasks and the RSS that they add.
2. Run --tasks concurrent tasks. Each task runs --streams lifecycles: create an anchor, attach a sender on the same worker, finalize, and read the terminal. The harness reports streams per second and the p50, p99 and p99.9 of one lifecycle.

cargo run --release --example anchor_setup_bench

To compare two builds, run each build three times or more, interleave the runs, and take the median. The tail of the lifecycle depends on the thread count: a lock that every create takes shows at 8 threads and not at 1 or 2. Response plane performance has the results for the unattached timeout.

The external serving rig

The external rig runs velo as the response plane inside Dynamo's serving stack. It uses Dynamo's own HTTP frontend, its mocker engine and the aiperf load generator. The same rig runs Dynamo's own response planes, so only the response plane changes between arms. The rig scripts and the Dynamo adapter live outside this repository. The table below is the shape of the main comparison; other measurements state their own shape.

Steady-state and raw percentiles

A closed-loop load generator at concurrency N starts its measured phase with N requests at once. aiperf 0.10.0 ramps request rate, not concurrency. On the external rig, each rep therefore opens with a burst of 8,192 simultaneous requests. The 8,192 warm-up requests warm caches, but the measured phase still opens with its own burst.

The burst sets the raw p99. In six reps on the external rig, every request at or above the raw TTFT p99 started within the first 0.2 s of the measured phase. That tail was 760 to 930 ms for both velo and the comparison plane. Its excess over the median came mostly from the request plane and the mocker processes absorbing 1,024 simultaneous requests each, not from the response plane.

The rig's summary therefore reports steady-state percentiles beside the raw ones. Steady state is the set of requests that started 10 s or later into the measured phase, about 200,000 of 250,000 per rep. Read verdicts from the steady-state columns. Report the raw columns too, so that a reader sees the burst.

Measurement rules

Each rule below exists because a measurement without it was wrong.

1. Run on an exclusive node. A CPU A/B ran on a shared login node with about 11 other users. Its spread was up to 5.5 µs per token, against an effect of about 2 µs per token. The result was retracted. On an exclusive 144-core node, the total spread over 42 runs was 2.43 µs per token.
2. Build both arms from one tree on the node, and interleave them. Run the arms alternately, rep by rep, so that drift hits both. Normalize CPU per token or per request over identical work.
3. Pin the load generator away from the system under test. With aiperf unpinned on the frontend node, node A ran at 95% utilization. Its load average was 159 on 144 cores. velo's TTFT p50 read 992 ms. With the frontend and aiperf on disjoint cores, the same configuration read 98 ms. Pin etcd and nats-server to the load generator's cores too. Unpinned, they took 12% to 38% of the frontend's cores in profiled reps.
4. Read timestamps in the right zone. aiperf 0.10.0 writes naive local timestamps. Reading them as UTC put the CPU window after the run, on the idle tail. Every frontend CPU figure measured that way was about three times too low.
5. Compare reps at a matched backlog draw. On the external rig, one or more mocker processes fall behind during the opening burst and keep that backlog. The number of processes that hold the backlog (the "holders") sets throughput, ITL and end-to-end latency for every arm. Compare two arms only at the same holder count.
6. Do not fix the arm order. A matrix that always ran the same arm second put that arm in a degraded band in every rep. Its request-level numbers measured run position, not the setting.
7. Change one setting per arm. A pair that differed in two settings attributed a result to the wrong one.
8. Assert the transport per request. Fail any request whose negotiated key is not messenger-mux-v2. A silent fallback measures the per-stream path under a mux label.
9. Record the build. Write the velo commit and its dirty state into each rep's metadata. If the installed build and the checkout differ, fail the rep.
10. Report latency beside throughput. A change that adds latency can raise throughput by letting the consumer catch up. Watch TTFT in particular, because any windowed flush policy can make it worse.
11. Prove a regression test before you trust it. Revert the fix and make sure that the test fails. A test that passes with the fix reverted proves nothing.
12. Check that every process was scraped. The rig gave worker process p the metrics port base + p. With 16 processes the range covered port 9100, which node_exporter holds on the compute nodes. Process 10 served traffic with no /metrics, and every summed worker metric missed a sixteenth of the fleet. An empty scrape file must fail the rep.
13. Time the first record at both ends before you blame a stage for TTFT. Dynamo's worker time_to_first_response stops when the prologue is sent, not at the first token, so it cannot separate engine queueing from the response path. The rig adds histograms for the engine's first item and for its send on the worker, and for the prologue and the first data item on the frontend. With them, a 200 ms TTFT gap turned out to be the response path (the first token left the worker 10 ms after the request arrived) and not the mocker. See The credit window and a saturated frontend.
14. Sort reps by the environment before you compare arms. On the 2-node rig, reps fall into two states that move every arm, the comparison plane included. In the bad state, frontend CPU per request rose from about 11 to 16–20 ms and irq and softirq time on node A rose from 5–9 to 13–16 cores. Network softirq runs on the frontend's own pinned cores: with the frontend on 24 cores, it took 5 to 7 of them, in both planes. Three reps that mix the states average two regimes and settle nothing.