QUIC performance

This chapter records what the QUIC transport costs against TCP, the settings that change it, and one quinn defect that the transport works around. For the design, see Transports.

Settings

The examples read VELO_QUIC_LANES, VELO_QUIC_MAX_MTU, VELO_QUIC_STREAM_WINDOW, and VELO_QUIC_SERVER_ENDPOINTS, so a sweep can change them without new flags.

Packets above 6550 bytes are lost

quinn gives up to 10 packets to one GSO send, and it does not limit the size of the batch. A UDP datagram holds at most 65507 bytes. When a packet is larger than 6550 bytes, a full batch is too large, and the kernel returns EMSGSIZE. quinn-udp 0.5 treats EMSGSIZE as success, because it expects that error only from MTU probes. So the whole batch is lost with no error, and each flight waits for a loss-probe timeout.

Measured on loopback with 64 KiB messages, one at a time:

The transport lowers max_mtu to 6550 and logs a warning. The test large_frames_round_trip_at_a_jumbo_mtu requests 8952 and fails if 200 round trips of 64 KiB take 2 s or more. Without the limit they take about 9 s.

Loopback measurements

Measured on 2026-09-23 on one aarch64 workstation (20 cores, shared, load average near 8), over loopback, with the throughput example and 5,000 messages for each cell. Two reps for each configuration. These numbers show direction only. For a network between two nodes, see Two nodes. For the response plane on a cluster, see Response plane on a cluster.

• Small messages cost about twice the latency of TCP. Each direction passes through quinn's connection and endpoint driver tasks, so one message takes more task hops than on TCP.
• Large messages reach about 600 to 800 MiB/s on one connection. One quinn task does the packet work and the encryption for a connection. TCP moves 64 KiB segments on loopback, and the kernel does that work.
• max_mtu 6550 gives 25% to 30% more for large messages when the path carries it. It does not help small messages.

Settings that did not help

• A larger initial congestion window. 1 MiB instead of quinn's 14,720 bytes made no difference, at the default MTU or at 6550.
• Larger flow-control windows. A 16 MiB stream window with a 64 MiB send window made no difference to one-at-a-time traffic, and was slower with 64 in flight (9,300 to 9,600 against 10,900 to 11,200 msg/s).
• Charging Tokio's task budget per batch of frames. Dynamo's QUIC plane needed this because it polled quinn once per frame header and once per payload. The velo reader uses FramedRead, which decodes all buffered frames before it polls quinn again, and routes frames through flume, which does not charge the budget.
• LSE atomics. Building with -C target-feature=+lse changed neither transport by more than 5%, on loopback on a 144-core Grace node.

Two nodes

Measured on 2026-09-29 with the throughput example in its two-host mode. The server ran on one node and the client on another. Each node has 144 Grace aarch64 cores, and the process had 16 of them. The nodes connect through 200G Ethernet at MTU 1500, with net.core.rmem_max at 212,992. Each cell sent 20,000 messages. Three reps, with TCP and QUIC interleaved.

• Small messages cost about 25 µs more per round trip than on TCP, as on loopback. The network does not change this cost: it comes from the task hops inside quinn. With many messages in flight the gap closes, and pipelined QUIC matches TCP.
• One QUIC connection tops out at about 0.8 GB/s, the same ceiling as on loopback on the same node type. So the limit is the CPU of the connection, not the network. A profile of the 64 KiB case put about 15% of the samples in AES-GCM. The rest was quinn's packet work, the kernel's UDP path, and task wakeups. TCP moves about 3.9 GB/s, because the kernel and the NIC offloads share that work.
• A peer that needs more than about 0.8 GB/s needs more than one connection; see Lanes. The Dynamo response plane on the cluster carries far less than that per connection (see below).

Lanes

With lanes(n), each peer gets up to n connections, one for each lane used, each dialed from its own UDP socket. Each lane has its own quinn endpoint driver, shared by every peer dialed on that lane, and each connection has its own connection driver. The work runs on up to n cores.

The messenger sends its own traffic on lane 0, so lanes(n) does not change the throughput of ordinary messages. The batched-streaming mux places streams over the lanes (see Lanes), so streams do get faster. A caller can also send on a lane with send_message_on_lane.

Measured on 2026-09-29 across the same two nodes as above, 64 KiB messages pipelined, 20,000 messages per cell, two reps. A prototype, which is not in the tree, spread the messages round robin over the lanes, with one connection per lane, each from its own UDP socket. The receiving node had 32 server sockets in one SO_REUSEPORT group.

The UDP receive-buffer error count stayed below 50 in each run, so the gain comes from more cores, not from more socket buffers. 64 B pipelined messages do not change with the lane count.

Lanes on one server socket share its endpoint driver. So each server socket has its own port, and a dialer sends lane k to socket (offset + k) % n, which spreads its lanes evenly. The prototype instead put all server sockets on one port in a SO_REUSEPORT group, where the kernel hashes each connection to a socket at random, so some sockets carried three or four lanes and others none. The table below shows what that cost with 8 lanes. It ran in a separate job on another pair of nodes, so its 32-socket row differs from the table above by run-to-run spread.

With one port for each socket, the lanes of one dialer are placed evenly, so fewer sockets do. Measured on 2026-09-29 across two nodes, 64 KiB messages pipelined, 20,000 messages per cell, two reps, with messages spread round robin over the lanes by a measurement build:

With 8 lanes, 4 sockets carry two lanes each and reach the same rate as 8 sockets. One lane is unchanged.

A fixed bind_addr port P binds ports P to P + n - 1, one for each server socket. Peers spread over all of them, ordinary traffic included, so open all of them in a firewall. Transports on one host need fixed ports at least n apart.

During a rolling upgrade, an old peer can dial a new node, and a new peer can dial an old node. A peer without per-socket ports dials only the first port of a new node, so until it upgrades it shares that socket with the other old peers.

A TCP prototype with the same layout, one connection per lane, scaled too: 2.5, 4.6, 8.1 and 13.1 GB/s (10^9 bytes per second) with 1, 2, 4 and 8 lanes. One TCP connection is limited by its receiver: one reader task does the recvmsg copy, and by default the transport's fixed socket buffers cap the TCP window (see Socket buffers). TcpTransportBuilder::lanes(n) builds the same layout. See TCP lanes. These TCP numbers came from the prototype, not from that code.

Streams over lanes

Measured on 2026-09-30 across two nodes of the same type as above, with the throughput example in its stream mode (--modes stream). The client creates the anchors, and the server attaches to them and streams to them, so the client is the consumer and places the streams on lanes. Both sides set VELO_QUIC_LANES to the lane count and keep the default of 4 server sockets. Each cell sends 200,000 items in total, split evenly over the streams, after a warm-up round. Two reps. The TCP row is one connection in the same job, for reference.

16 KiB items, MiB/s:

64 B items, items per second:

• Streams scale with lanes as messages do. One lane stays at the one-connection limit, 500 to 780 MiB/s. Each doubling of lanes nearly doubles the rate, up to about 5,300 MiB/s at 8 lanes, which is 2.5 times TCP on one connection.
• 16 streams do not fill 8 lanes. With two streams on each lane, 8 lanes reach about 3,700 MiB/s. Each stream waits for credit every 32 records, which limits the rate of one stream. This was not profiled.
• Small items lose a little at few streams. 16 streams of 64 B are about 9% slower at 8 lanes than at 1. The likely cause is that each lane's batch then carries the records of only two streams. This was not profiled. At 256 streams, 8 lanes move twice as many items as 1.
• The streams used the lanes. One lane carries at most about 790 MiB/s, so 5,300 MiB/s at 8 lanes needs data on at least 7 lanes. The lane tests check the placement directly.

A second job on another pair of nodes gave 533 to 735 MiB/s at 1 lane and 3,721 to 5,150 MiB/s at 8 lanes, so the spread between node pairs is larger than between reps.

Later runs on three more pairs of nodes gave 3,237 to 5,402 MiB/s at 8 lanes and 64 streams (median 5,198, 11 reps), and 3,655 to 5,326 MiB/s at 256 streams (median 4,948). The first rep of a job is often the low one. A diagnostic build counted the bytes that each lane's batch handler received: over 8 lanes, the busiest lane carried at most 1.09 times the mean, at 64 and at 256 streams.

The same stream mode over TCP lanes (VELO_TCP_LANES), on one pair of nodes, 2 reps, 16 KiB items, MiB/s:

With 64 B items at 256 streams, TCP moved 783,000 to 857,000 items/s at 1 lane and 2,120,000 to 2,219,000 at 8 lanes. At 16 streams, 8 lanes were slower than 4, for both item sizes.

Batched streaming over QUIC

Measured on the same workstation with response_plane_bench (2 anchor hosts, 64 engines, 2,000 requests, 200 warm-up requests, --flush-policy auto), three reps.

In the two quieter reps, QUIC matched TCP on throughput and first-token latency (11,300 and 11,000 against 11,600 and 11,000 requests/s). It used 15% to 30% more CPU for each request. The mux batches many token records into each frame, so the per-message cost of QUIC matters less here than in the one-at-a-time tests. max_mtu 6550 was slower in all three reps with these small frames. Use it only for traffic with large messages.

Response plane on a cluster

Measured on 2026-09-23 with Dynamo's response plane on two nodes. Each node has 144 Grace aarch64 cores. The nodes connect through 200G Ethernet at MTU 1500. One node runs the frontend, etcd, NATS, and the load generator. The frontend has cores 0-71 and the load generator has cores 72-143. The other node runs 512 mock workers in 8 processes. Each run sends 250,000 requests at concurrency 8,192, with 1,024 input tokens and 256 output tokens. Each configuration ran three times, and the runs of the four configurations alternated.

The hosts have net.core.rmem_max and net.core.wmem_max at 212,992. The transport requests 8 MiB and 4 MiB for each socket, and the kernel gives 425,984 bytes. The transport logs one receive-buffer and one send-buffer clamp warning for each socket. The frontend used 32 server endpoints. The workers used the default of 4.

The configurations:

• Velo, TCP. The messenger mux over the TCP transport. The frontend mints the stream terms at registration (zero-RTT attach), and batches write when the transport admits them.
• Velo, QUIC. The same configuration over the QUIC transport.
• Multiplexed TCP. A multiplexed TCP response plane that does not use velo, for reference.
• Dynamo QUIC. Dynamo's own QUIC response plane on quinn 0.11.12, with 32 server endpoints and no batch interval.

A mock-worker process that holds a share of the 8,192 open requests answers late. Throughput and first-token latency change with the number of these processes in a run. The table therefore compares runs with the same number, shown as "Holders". Steady state is the requests that started 10 s or more after the first request. ITL is the mean inter-token latency of each request. CPU is frontend CPU time for each request. Drops is the RcvbufErrors count of the frontend host for the run.

TTFT p50 and p99 are steady-state values. One run of Velo over TCP had 7 holders and is not in the table. No run had client errors.

• QUIC shows no cost for velo here. At the same number of holders, velo over QUIC and velo over TCP agree on throughput, latency, and CPU. The differences are 1% to 5%, and one or two runs for each cell cannot resolve them. On loopback, QUIC used 15% to 30% more CPU for each request. On the cluster, most of the frontend CPU is outside the transport, and the mux puts many records in each frame.
• The clamped buffer loses about 0.1% of datagrams. With 32 endpoints, the velo frontend received 4.7 to 5.3 million datagrams in each run and dropped about 0.1% of them. The drops did not show in the tail latency.
• Velo over QUIC is ahead of Dynamo QUIC. With 2 holders, velo moved 2,742 requests/s against 2,328 to 2,496. It also had a lower first-token p99, a lower ITL p99, and 6% to 9% less CPU. The Dynamo frontend dropped 16 to 18 times as many datagrams with 2 holders, and 27 to 41 times as many with 1 holder. It received about 0.7 datagrams for each output token, against 0.07 to 0.08 for velo. With the batch interval at 0, Dynamo sends close to one packet for each token record. The velo mux puts many records in each frame. Dynamo's default interval of 5 ms was not measured.
• max_mtu 6550 was not measured. The path MTU of the cluster is 1500, so a larger max_mtu has no effect.