Response plane performance

This chapter reports how velo performs as the response plane of an LLM serving stack. It gives the result, the changes that produced it, the changes that did not help, and the mechanisms found on the way. Benchmarking describes the rig and the measurement rules. Read it before you compare any number here with a new one.

The response plane is the path that carries generated tokens from a worker back to the frontend. In these measurements velo runs the messenger mux over TCP inside Dynamo's serving stack. The comparison plane is Dynamo's multiplexed TCP response plane (Dynamo PR 11918, ported onto the same Dynamo base so that both planes share one stack). Both planes carry many streams over a few shared connections and batch their frames.

Result

Measured on 2026-09-10 on the external rig that Benchmarking describes: 512 mocker workers, concurrency 8,192 and 250,000 requests per rep. The frontend ran on one tokio runtime with 72 workers. Each of two matrices ran three reps per plane.

The velo configuration: mux over TCP, zero-RTT stream setup, write on admission (FlushPolicy::Auto with on_admission), reply_linger 1 ms, initial_credit 256 (the default when these were measured; now 32, see The credit window and a saturated frontend), async_open_ack off.

A holder is a mocker process that keeps part of the 8,192-request backlog after the opening burst. The holder count sets throughput and the tails for every plane, so reps compare only at a matched holder count. Steady state is the requests that started 10 s or later into the measured phase. Where two reps share a draw, both values are shown.

• At 1 to 3 holders, velo's first-token p50 is 2 to 7 ms ahead. At 5 to 6 holders, the two are level (55.5 against 55.1 ms).
• Steady-state p90 and p99 are level or ahead.
• Throughput is equal within the draw.
• Every rep in the two matrices had zero errors.
• Frontend CPU per request is 1.2 to 2.9 ms higher on velo. This is recorded, not a requirement.

Against the shipping per-request plane

Dynamo's shipping plane opens one TCP connection per response. Measured on 2026-09-05 with core pinning, before zero-RTT setup. The velo arm here is the mux with an attach round trip per stream. CPU is the corrected figure.

The per-request plane's accept loop is bistable. In the jammed state about 3,900 sockets sit connected but not accepted, and TTFT is the backlog divided by the accept rate. Its raw p99 of 1.7 s comes from that loop.

Dynamo's QUIC response plane (PR 11996) did not complete a clean run at this scale on this cluster. Six reps, measured before core pinning, had 4,947 to 84,525 errors each: HTTP 500s and streams with no content. Its low TTFT medians in those reps were survivorship, because dropped requests do not queue. A likely factor is net.core.rmem_max at 212,992 bytes, which clamps QUIC's UDP sockets about 30 times below what quinn requests. This is a result for this cluster, not a general verdict on QUIC.

What produced the result

The changes are listed in order of effect. Each one landed with a test that fails without it.

1. One tokio runtime in the frontend. This is the largest lever, and it is outside velo. The Dynamo frontend ran two 72-worker runtimes on 72 cores. The Python bridge built its own runtime lazily, because its initialization code sat in a branch that never ran. The velo node ran on the first runtime. The HTTP handlers and the per-stream reader pumps ran on the second. Every response record crossed between them through the injection-queue mutex of the second runtime. That lock took 2.4% of the cores plus 6.8% in lock contention. A worker blocked on that mutex does not service the time driver. Linger timers then stopped for seconds, and credit stopped. The 8,192 streams fell into a 6 to 8 s limit cycle. The fix initializes the bridge with Dynamo's runtime, a 24-line change. Thread count fell from 242 to 154.
2. Zero-RTT stream setup. The worker sends without waiting for the attach round trip. The worker-ingress-to-first-token segment fell from 78 to 48 ms at p50.
3. Credit on the next batch for every drained slot. Each drain names the slot it drained, and the next inbound batch reconciles it. Before this change, each inbound batch walked every slot of its peer. At about 1,000 slots per peer and 11 slots touched per batch, flume::Shared::len alone took 2.1% of the frontend's 72 cores, about 0.5 ms per request.
4. One timer per stream in the reader pump. The pump used to build a tokio::time::timeout per record. Each one took the time driver's lock twice. The Sleep subtree was 7.3% of the frontend's cores, plus 0.8% for the cancellation future. With one pinned timer, the timer subtree fell to 0.15%. The mux has since dropped the reader pump; its stream watchdog wakes once per window and does no timer work per record. See The reader pump leaves the mux data path.
5. Reply linger. Credit replies wait up to 1 ms to share a batch. With zero-RTT setup, the frontend's egress had carried one batch per credit reply: 1,027,872 outbound batches against 97,042 without zero-RTT. The linger cut frontend batches four to six times and saved 0.4 ms of CPU per request.
6. Control maps bounded by allocation. A 4,096-entry control cap was sized for about 1,024 slots per peer. One mocker process held 4,000 to 6,700 live slots. The cap refused credit grants and closes, and with async_open_ack it refused the answer that lifts a slot's fence. A fenced slot then waited for the 15 s watchdog, and the client saw an HTTP 500.
7. Drain-driven credit return with a visit floor. Measured on 2026-09-01 on an exclusive 144-core node with the in-process harness at 256 ingress peers, 42 interleaved runs, at a credit window of 256 with streams of 64 tokens, so no stream ran out of credit:

The hook costs about 2%. It is what makes the 200 ms sweep safe, because the sweep was the only path that returned credit to a quiet peer. The net gain is 4.5%, and the case for the change is correctness. Without a floor on doorbell visits, a stream drained through an 8-record window rang the doorbell 1,000 times in about 270 ms. The 2 ms floor spread the same 1,000 walks over about 3 s.

The Dynamo adapter also polls the anchor inline from the connection task instead of through a consumer task with a 64-deep mailbox. This removed about 0.2 ms of adapter CPU per request.

What is not a lever

These changes were measured or traced and did not help. They are recorded so that nobody tries them again without a new reason.

Where velo's extra CPU goes

A per-subtree partition of a perf profile of both frontends, with one runtime (one rep each, measured on 2026-09-06). Each stack line goes to exactly one bucket, and each plane sums to 100%. The values are milliseconds per request.

velo's anchor and adapter consumer (1.46) cost about what the comparison plane's reader and receiver cost (1.24). The excess is task hops. The reader pump is a channel-to-channel relay with no counterpart, and the lane, dispatch and transport are three stages where the comparison plane has one task. Each record also reaches the HTTP connection task as its own wake.

After the inline receiver, a second profile put the velo-only buckets at 3.20 ms per request. The buckets were: reader pump 1.01, anchor 0.85, ingress 0.51, adapter 0.49, TCP 0.18, dispatch 0.14, batcher 0.01. The reader pump and the anchor channel are the remaining addressable surplus. The pump cannot merge into the anchor channel (see the table above). Each other hop-chain change is optional and needs a same-matrix tail check. These two profiles predate the direct feed, which takes the reader pump off the mux data path. See The reader pump leaves the mux data path.

Before the fixes above, these symbols appeared only in velo's profile, as a share of the 72 frontend cores:

Removed, measured on the rig rather than in a profile: each bind used to spawn one task that slept for the 60-second accept window, and a claim did not cancel it. A frontend at about 2,000 requests per second held 146,000 live tokio tasks against about 25,000 on the QUIC plane. The accept window is now a deadline in one queue that the credit sweep drains, so a bind owns no task and a window closes up to one sweep interval late. The test a_finished_zero_rtt_stream_leaves_no_task_behind holds the task count to its baseline after streams end.

Removed, measured with anchor_setup_bench: each SPSC anchor used to spawn one task for its unattached timeout. Attach cancelled the task, but each stream still paid one spawn and one exit, and each unattached anchor held one task. An unattached deadline is now an entry in one queue per manager, and one task sweeps that queue. The sweep wakes at the earliest entry, so a timeout fires at its deadline, as before. The host is one aarch64 machine with 20 cores (10 Cortex-X925, 10 Cortex-A725). The unattached rows are the median of three interleaved runs at 8 tokio threads. The lifecycle rows are the median of five interleaved runs of 16M lifecycles (3.84M at 2 threads), 64 tasks. Such a run lasts about 10 s, so the sweep pops entries for all but its first second. CPU is the user and system time of the process, divided by the lifecycles.

At 2 threads the p99.99 ranges overlap: 127 to 251 µs with a task per anchor, and 133 to 216 µs with the queue. A run of 1.28M lifecycles ends before any entry comes due, so it measures only the push; the 16M run includes the sweep.

The queue does not remove an entry when its anchor attaches or ends. The sweep drops the entry when it comes due. A new entry comes due within 1 s, and a requeued entry within 60 s, so the queue holds about stream rate × 1 s of stale entries, at 24 bytes each. An entry queued at the deadline of its anchor stayed for the full timeout instead: 120 s by default, and 30 years for Duration::MAX.

Rejected: one lock for the whole queue. Every create takes a lock on the queue. With one lock, the lifecycle p99.9 at 8 threads was 110 µs, against 22 µs with a task per anchor. No entry came due inside that run, so the sweep did not cause the tail: creates waited on each other for the one lock. With 1 or 2 threads, the single lock was faster than the task at every percentile, so the cost was contention. The queue now has 16 parts, each with its own lock, and the id of an anchor selects its part.

The tests unattached_anchors_share_one_sweep_task and ended_anchors_leave_the_queue_within_the_first_look hold the task count and the queue length. MPSC anchors use a second queue of the same kind, and mpsc_anchors_share_one_sweep_task holds its task count.

Removed, measured with anchor_setup_bench and stream_lifecycle: each SPSC and MPSC sender spawned one heartbeat task that lived as long as its stream. Now one task per AnchorManager beats every remote sender, from the same kind of queue. A sender on the anchor's own worker does not beat at all, because no watchdog reads its beats. Same host and settings as above, median of three interleaved runs, against the deadline queue alone:

The two rows measure two different parts of the change. A same-worker sender now does not beat at all, so the same-worker row shows the cost of a heartbeat task per stream, not the cost of the sweep. Only the mux row exercises the sweep. Both arms of each row have the deadline queue. The same-worker runs are 1.28M lifecycles, which end before the anchor sweep pops anything, in both arms.

On the mux path the difference is inside the spread of the runs. The test senders_share_one_heartbeat_task holds the task count, and a_silent_remote_stream_outlives_the_watchdog_window holds an idle remote stream open for 10 heartbeat intervals.

These costs are counted from source, not measured:

• The per-record copy. Ingress copies each record body into a Vec (one allocation and one memcpy, estimated at 35 to 55 ns). Removing it needs Bytes from the slot buffer through to the consumer, and it is not done (see Batched streaming design). On the microbenchmark below, memcpy was 0.5% of frontend CPU. A typed Bytes consumer copies each payload a second time, in serde. Byte items remove that second copy.
• Wakes. flume fires a waker only for a parked receiver, so k records for one slot in one batch already cost one wake. With the direct feed, a record makes one hop, from the slot buffer to the consumer.

The reader pump leaves the mux data path

Measured on 2026-09-26 with a two-process microbenchmark shaped like response_plane_bench, on one Grace node: frontend 32 threads, 2,048 streams, 918 records of 160 B, jemalloc. The figure is frontend CPU per record, median of 3 runs. Each pair of rows is its own A/B. The direct-feed baseline is a separate run of the dirty-set build, so the rows do not chain.

In the profiles before these changes, the reader pump was 28% of frontend CPU at 12 peers. In the dirty set, only the drain that newly lists a slot touches the peer's shared wake flag. The direct feed removes a task wake, a channel send and receive, and a hook allocation per record. These are microbenchmark numbers, not rig numbers.

First-token mechanisms

A per-request join splits client TTFT into three segments. The join matches aiperf's request id, the frontend's log and the mocker's log, with clock skew below 1 ms. The three segments sum to the client's TTFT.

• A: client send to the frontend's "request received" log line (HTTP ingress).
• B: the frontend to the worker handler's start (the request plane).
• C: worker ingress to the first token at the client (mostly the response plane).

TTFT equals the wait for the first HTTP byte to within 0.3 ms at every percentile, in every plane. No plane writes a byte before the first token.

These mechanisms were found:

• A saturated load-generator node looked like a slow plane. Before core pinning, velo's TTFT p50 read about 1.1 s. The queues that velo owns held a small share of it: 26 to 36 ms in the node-global inbound queue, about 110 ms in the ordered lanes and 79 ms in the worker's egress queue. The rest was the time between a wake and a run on a node at 95% utilization. Pinning moved velo's p50 to 98 ms at the same throughput.
• The attach round trip was the largest term. Before zero-RTT setup, the worker waited for _anchor_attach before it started generating. Under load before pinning, that round trip averaged 390 to 524 ms. With pinning it averaged 13 to 22 ms. Zero-RTT setup removes it.
• The comparison plane has nothing on the first-record path that grows with load. The frontend mints the stream id at request registration. The prologue goes on an urgent lane that the writer drains before ordered data. The only shared queue that a first frame crosses is bounded by a 256 KiB byte budget per connection.
• Zero-RTT setup first moved the cost into the request path. With two frontend runtimes, segments A and B each grew by about 8 to 9 ms under zero-RTT setup, while C fell. The frontend sent ten times more batches, one per credit reply. The reply linger removed the extra batches but not the A and B growth, so the batches were a symptom of frontend contention. After the one-runtime fix, velo's p50 moved ahead of the comparison plane, and segment B contributes nothing to the steady-state tail.
• TTFT p50 followed the backlog draw. Before the one-runtime fix, velo's p50 was 54 to 55 ms at one holder, 61 to 62 at two, 73 to 76 at three and 83 at five. The comparison plane stayed at 48 to 49 ms. After the fix, velo's p50 is 39 to 45 ms at one to three holders.
• The steady-state tail is the hot mocker process and HTTP ingress, in both planes. In steady state, segment B contributes nothing to p99. Segment C adds 125 to 209 ms, and 59% to 89% of the p90-to-p99 band sits on one mocker process with 3,000 to 6,400 requests in flight. Segment A adds 39 to 112 ms. The raw p99 is the opening burst, described in Benchmarking.
• End-to-end and ITL tails measure one mocker process. Arrivals are equal across the 8 mocker processes. In-flight counts are not: seven sit at 150 to 200 and one at 3,000 to 7,000 for the whole run, in every plane. By Little's law, a request on the hot process stays about 15 s against 0.43 s elsewhere. End-to-end p99 and ITL p99 therefore measure which process holds the backlog, not the plane.
• A handler wait cost a hello round trip per request. The Dynamo adapter calls Velo::wait_for_handler before each request to make sure the worker can reach the frontend's _stream_stop handler. wait_for_handler refreshed the peer's handler list with a full _hello round trip every time, and under a saturated frontend that round trip waited in the frontend's messenger queues. It now returns at once when the handler list already names the handler. The test test_wait_for_handler_reuses_cache_and_refreshes_missing_handlers counts one handshake for repeated waits on a known handler, and a new one for a missing handler or a new instance. On the rig its effect on TTFT was within the rep-to-rep spread. A Dynamo adapter change also prepares each peer once instead of per request.
• velo's tail discipline is conditional. Before pinning, velo appeared to keep a much shorter end-to-end tail. That was a starved frontend limiting how many streams ran at once. With the frontend on 48 cores and the load generator on 96, velo held 8.1 s end-to-end p99 at 3,008 req/s against the comparison plane's 25.2 s at 2,460. On a 72/72 split, both planes posted about 11.5 s.

The credit window and a saturated frontend

Measured on 2026-09-26 and 2026-09-27. When the frontend is the bottleneck, every stream runs at its credit window, and the windows fill the shared path between a worker and the frontend: the worker's egress queue, both socket buffers and the frontend's reader. A new stream's first record queues behind all of it. The comparison plane has no per-request credit, and its worker-side send blocks, so its backlog stays in the producer.

Setup: the Dynamo mocker rig with 16 processes of 32 workers, concurrency 8,192, input length 1,024, output length 900, speedup 10 and 250,000 requests per repetition. The frontend was pinned to 24 or 32 cores. Each allocation ran one window against a window of 256, three repetitions per arm with the arms interleaved; ranges cover all three. The last column is a stream's first record (its prologue), from the worker handing it to the plane to the frontend receiving it, as a mean over requests.

Rows are grouped by allocation. In the 64 allocation, two of the three 256 repetitions and one of the 64 repetitions fell into the degraded environment state that Measurement rules describes. In their good repetitions, 64 read 98 ms of TTFT against 156 ms at 256, and its first record took 31 ms, the top of the range of 32. The data does not separate 64 from 32 beyond that.

The default window is 32. At 24 cores it brings velo's TTFT to the comparison plane's, with throughput within noise of 256 in the same allocation and ITL p99 no higher; ITL p50 is 0.1 to 0.4 ms higher, because streams need more grants. At 32 cores, 32 and 256 read the same TTFT p50, throughput and ITL, and the first record still waits less at 32. A window of 16 had the lowest TTFT at 24 cores, but it doubles the per-record credit-return cost ((drain_visit_floor + reply_linger) / initial_credit, about 190 µs at 16 against 94 µs at 32), and it was not measured at 32 cores.

At 32 the surplus waits in each stream's own withheld queue on the producer rather than in the shared path. In the first repetition of the 24-core allocation that ran 32 against 256, summed over the 16 worker processes, sender credit exhaustion rose from about 240,000 to about 1.2 million events; the worker's mean egress queue wait per frame fell from 9.5 to 1.2 ms, and the frontend's mean lane wait per batch from 1.8 to 0.7 ms. The per-stream backlog itself does not go away while the frontend is the bottleneck; it moves to where it delays only its own stream.

UCX transport instability

Measured on 2026-09-04, before core pinning: the velo mux over velo's UCX transport on InfiniBand (2 mlx5 adapters per node), against the same mux over TCP on 200G Ethernet. Three reps each, 512 workers, concurrency 8,192.

UCX did not beat TCP, and it did not hold its tail. The failure chain:

1. The KV router put about 5,760 concurrent streams on one of the eight mocker processes. This imbalance occurs with every transport.
2. All 64 workers in a process share one velo node and one mux peer link. All 1,722 failures were in that one process, spread evenly over its 64 workers.
3. Over UCX, that one link fell behind. On the hot process, the delay from worker completion to client completion grew to a median of 18.87 s. Over TCP, at the same or higher concurrency, it peaked at 1.0 to 2.75 s.
4. Heartbeats share the per-peer queue with data, so streams behind more than 15 s of backlog went silent at the frontend.
5. The reader-pump watchdog (3 × 5 s) killed 1,722 streams. Every log line showed anchor_frame_tx_len=0 and transport_rx_len=0. closed_slot drops were 1,722 × 255 exactly, so no record had arrived before each kill.
6. The worker was healthy the whole time. It completed all 27,702 of its requests.

The measured revision had no UCX backpressure edge beyond its shared ring:

• ucp_am_send_nbx never refuses. An exhausted endpoint returns a request pointer, not an error.
• inflight_ops counts posted operations but does not gate admission.
• The per-peer admission gate queues without a bound.
• All peers share one 1,024-entry ring into the progress thread. TCP gives each connection a 256-entry channel and a blocking write.

Version 0.14 bounds queued and outstanding sends per peer. A send permit is released on UCX completion or failure, and peer retirement cancels pending admission. The progress thread can still process control and endpoint cleanup when a data peer is full. This change requires a new saturation measurement; it does not change the historical results below.

Because admission never blocks, the on-admission flush policy never parks, and batches collapse. UCX carried 6.97 to 8.65 records per batch against 18 to 29 over TCP with the same settings. That is 2.6 to 4.1 times the message rate for the same records.

These alternatives were checked and refuted:

• Send-backpressure counters do not follow the failure. Their totals ran opposite to the instability across reps 2 and 3.
• The peer byte budget (8 MiB over 5,760 slots) does not depend on the transport. The TCP control carried 5,326 streams on the same budget and held p99 at 2.5 s.
• Batch collapse amplifies the failure but does not start it. The clean rep had fully collapsed batches too.
• UCX scheduling is not unfair. The failed peer carried eight times the load of the others.
• Spinning progress threads do not starve the CPU. There were 9 progress threads on 288 cores.

No tuning setting reaches this mechanism. The mux's stream watchdog, which replaced the reader pump's watchdog on single-sender streams, exempts a sender that holds no data credit, but here the senders held credit and their records were stuck upstream, so it would still have fired. The fix has two parts. The UCX transport needs a backpressure edge (gate admission on in-flight operations or on a per-peer ring share). Liveness needs a heartbeat path that data cannot block, or a watchdog that can tell a starved stream from a dead one. The UCX inbound path records frames like every other transport. Version 0.14 adds the bounded admission edge. Saturation measurements must still check whether data backlog delays stream heartbeats beyond the watchdog. RDMA performance covers the UCX transport outside the response plane.

Instrumentation cost

These instruments are on the hot path. Their costs are accepted without an A/B measurement:

• Egress writer instruments (velo_transport_egress_queue_wait_seconds, velo_transport_frames_written_total, velo_transport_write_duration_seconds, TCP, UDS and QUIC only). Per frame: one Instant::now() at send, then one elapsed() and one histogram observe at dequeue. Per write: a second observe and up to five counter compare-and-swap loops. If this cost shows in a measurement, observe once per write on the oldest frame instead of once per frame.
• Ordered-lane metrics for _stream_batch. Per batch: four Arc clones, one elapsed() with one histogram observe, and one gauge increment and decrement. This cost is the measurement of the lane wait itself, so no A/B applies.
• Batcher counters (velo_streaming_mux_records_sent_total, velo_streaming_mux_batcher_wakes_total). The collectors are pre-bound into arrays. Per record: one u16 increment. Per wake: one AtomicF64 compare-and-swap. velo_streaming_mux_staged_records already paid one gauge compare-and-swap per staged record.

The rule for new instruments on these paths: no label lookup per record. Pre-bind the collectors, and observe per batch where possible.

The instruments did not perturb the rig. Three instrumented reps and one control rep stayed inside the historical band. One outlier rep was a hot mocker process with 1,887 live slots, not the instruments.

Findings outside velo

• axum's graceful-shutdown watch costs 1.4 to 2.2 ms per request in both planes. It is one process-wide lock, polled again on every connection wake. It tracks the backlog draw, not the plane.
• Two tokio runtimes in the frontend. See item 1 of What produced the result. The fix belongs in Dynamo's Python bindings and helps every response plane.
• aiperf 0.10.0 can wait forever when a record references a dataset entry that fails to decode. It then writes no export. The rig patches it to finish after 15 report intervals with no progress, once all credits are complete and at least 95% of records are processed. The patch fired on runs that missed 2 to 397 of 250,000 records.
• Mixed model cards stop a mocker fleet silently. Packed workers with DYN_SYSTEM_PORT set publish a mix of self-hosted and fallback model cards. Dynamo's discovery controller parks them in a Conflict state with no log line, and the fleet never serves. Set DYN_SELF_HOST_METADATA=0 and leave DYN_SYSTEM_PORT unset.
• The socket buffer clamp. By default, velo requests 2 MiB socket buffers (see Socket buffers). This cluster's net.core.rmem_max and wmem_max are 212,992 bytes and cannot be raised, which clamps each buffer to about 208 KiB and disables autotuning. At 0.5 ms round-trip time, one connection carries at most about 400 MB/s. The rig's worst-case mux traffic is about 200 MB/s in total, so the clamp does not bind with 4 or more worker processes per node.