Velo

Velo is a distributed messaging library for Rust. It gives active messages, typed streams, distributed events, large-payload transfer, and work queues over pluggable transports. It has peer discovery and Prometheus metrics built in.

Velo is experimental. The APIs are not stable yet. Do not use Velo in production.

What a Velo instance holds

A Velo instance wraps three managers behind one API:

• Messenger: active messages with four patterns (fire-and-forget, sync, unary, typed unary) and named handlers.
• AnchorManager: typed streams. One producer attaches to one anchor and sends items to the consumer that owns the anchor.
• RendezvousManager: large payloads. The owner stages the bytes, and a peer pulls them by handle. Over UCX, the pull is a one-sided RDMA read.

You inject transports, discovery, and metrics when you build the instance.

graph TD
    App[Application] --> V[Velo]
    V --> M[Messenger]
    V --> A[AnchorManager]
    V --> R[RendezvousManager]
    M --> T["Transports<br>TCP, UDS, QUIC, NATS, gRPC, ZMQ, UCX"]
    A --> FT["Frame transport or<br>batched streaming (mux)"]
    R --> M
    R --> RDMA["RDMA GET (UCX)"]
    M --> D["Discovery<br>filesystem, NATS, etcd"]
    V --> O["VeloMetrics<br>(Prometheus)"]

Crates

Application authors depend on velo only. See Workspace crates for why the runtime is one crate, and why velo-hub is a crate of its own.

How to read this book

• Concepts explains each subsystem and the contract that it keeps.
• Guides gives procedures for common tasks.
• Operations covers metrics, saturation, benchmarks, and the measured performance.
• Development records the workspace rules and the design decisions, with the reasons and the alternatives that were rejected.

Getting started

Add the dependency

cargo add velo
cargo add serde --features derive
cargo add tokio --features macros,rt-multi-thread
cargo add anyhow

The default features are http, nats-transport, grpc, and services. Add other features as you need them:

Platforms

velo builds on Linux and macOS. On macOS every feature builds except ucx. There the ucx feature compiles but adds no transport, and the ucx-rs crate builds as an empty stub, so cargo build --workspace --all-features still works. Linux sockets also set TCP_USER_TIMEOUT, which macOS does not have.

TCP is always available. UDS is always available on Unix. Filesystem discovery needs services. Discovery and queue backend features enable services too.

nats-transport, nats-discovery, and nats-queue are independent. They share the async-nats dependency, but one does not enable the others.

A first program

This program connects two instances over TCP and makes one typed request.

use std::sync::Arc;
use serde::{Deserialize, Serialize};
use velo::transports::tcp::TcpTransportBuilder;
use velo::{Handler, TypedContext, Velo};

#[derive(Serialize, Deserialize)]
struct AddRequest { a: i64, b: i64 }

#[derive(Serialize, Deserialize)]
struct AddResponse { sum: i64 }

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let node_a = Velo::builder()
        .add_transport(Arc::new(TcpTransportBuilder::new().build()?))
        .build()
        .await?;
    let node_b = Velo::builder()
        .add_transport(Arc::new(TcpTransportBuilder::new().build()?))
        .build()
        .await?;

    // Register a handler on node B.
    let handler = Handler::typed_unary_async("add", |ctx: TypedContext<AddRequest>| async move {
        Ok(AddResponse { sum: ctx.input.a + ctx.input.b })
    })
    .build();
    node_b.register_handler(handler)?;

    // Without a discovery backend, register the peer by hand.
    node_a.register_peer(node_b.peer_info())?;

    let resp: AddResponse = node_a
        .typed_unary::<AddResponse>("add")?
        .payload(&AddRequest { a: 3, b: 4 })?
        .instance(node_b.instance_id())
        .send()
        .await?;
    assert_eq!(resp.sum, 7);
    Ok(())
}

Next steps

• Read Messaging for the four request patterns and the handler options.
• Read Streaming to move a sequence of items between workers.
• Read Run the examples to run the examples in the repository.

Messaging

The messenger sends active messages. An active message names a handler on a remote instance. That handler runs when the message arrives.

Four patterns

The patterns differ in what the caller waits for.

All four builders work the same way. Get a builder from the Velo instance, attach a payload, name the destination instance, then call .send().await.

// Fire-and-forget. Delivery is best effort.
node_a.am_send("notify")?.payload(&event)?.instance(b).send().await?;

// Sync. Returns when the handler finishes.
node_a.am_sync("process")?.payload(&job)?.instance(b).send().await?;

// Unary with raw bytes.
let reply: Bytes = node_a.unary("ping")?.raw_payload(Bytes::new()).instance(b).send().await?;

// Typed unary. Payloads are JSON (serde_json).
let reply: MyResponse = node_a.typed_unary::<MyResponse>("rpc")?.payload(&req)?.instance(b).send().await?;

A payload that is larger than the large-payload threshold does not travel in the message. The messenger stages it with rendezvous and sends a handle. The receiver pulls the bytes before the handler runs.

Timeouts

velo puts no deadline on a request. A sync or unary call waits until the remote answers or the send fails. To bound the wait, set .timeout(d) on an am_sync, unary, or typed_unary builder.

use std::time::Duration;
use velo::RequestTimeout;

match node_a.unary("ping")?.timeout(Duration::from_secs(2)).instance(b).send().await {
    Ok(reply) => { /* the answer */ }
    Err(e) if e.is::<RequestTimeout>() => { /* no answer in time */ }
    Err(e) => return Err(e),
}

The clock starts at send(). It covers the wait for a response slot, discovery, the handshake, the send, and the answer. When the time runs out, the call fails with RequestTimeout. Every other error is a failed send or an error from the remote. Only awaiting the result is bounded, not admitted().

At expiry, the call gives back its response slot, also when the caller keeps the result. A reply that arrives later is discarded. The remote can still run the handler. A call without .timeout makes no timer.

When the request's payload is staged, the timeout also bounds the receiver's fetch of it. At expiry the receiver gives up the fetch and its response slot, the handler does not run, and the sender frees the payload. See Transparent staging.

A remote that forgot the caller never answers. The remote sends a reply through its own peer table, not back over the connection of the request. So a call to a remote that called unregister_peer on the caller waits until its timeout, and without a timeout it waits for ever.

Handlers

You register a handler by name. Each handler kind has a sync form and an async form.

use velo::{Context, Handler, TypedContext};

// Sync unary handler that returns raw bytes.
node.register_handler(Handler::unary_handler("ping", |_ctx: Context| Ok(Some(Bytes::from("pong")))).build())?;

// Async typed handler. The input is deserialized and the output is serialized.
node.register_handler(
    Handler::typed_unary_async("add", |ctx: TypedContext<AddRequest>| async move {
        Ok(AddResponse { sum: ctx.input.a + ctx.input.b })
    })
    .build(),
)?;

// Async fire-and-forget handler.
node.register_handler(Handler::am_handler_async("notify", |ctx: Context| async move { Ok(()) }).build())?;

The handler context holds the payload, the message headers, and the messenger (ctx.msg). A handler can use the messenger to send messages, register handlers, or wait for events. ctx.sender_worker_id() is always available. ctx.sender_instance_id() returns the InstanceId of the sender after its handshake is complete.

Handler names that start with _ are reserved for system handlers.

Dispatch modes

.max_concurrent(n) limits how many lanes run the handler at the same time. A lane takes the permit for each message inside the lane, so the limit does not change the order in a lane. The builder rejects 0 because it would stop every lane.

let h = Handler::typed_unary_async("bulk", bulk)
    .ordered_with(
        OrderedConfig::by_sender()
            .with_idle_lane_ttl(Some(Duration::from_secs(300)))
            .with_max_queue_depth(Some(100_000))
            .with_overflow(OverflowPolicy::Reject),
    )
    .build();

Ordered dispatch has these limits:

• It keeps order. It does not create order. If a peer uses more than one transport, or a connection drops and reconnects, the arrival order was already lost before the messenger saw the messages.
• Large payloads are not ordered. A rendezvous-staged message resolves before dispatch. The dispatcher logs a warning once when it sees one.
• Lane queues have no hard bound. max_queue_depth is a soft limit for each lane. It drives OverflowPolicy. It does not bound the channel. One slow peer cannot cause the messages of other peers to be shed.
• A unary handler serves one request at a time for each sender. A client that makes 100 concurrent calls gets them one after the other.
• Idle lanes stop. A lane with no messages for idle_lane_ttl (30 s by default) stops, so short-lived peers do not leak tasks.

Watch velo_messenger_ordered_lane_depth and velo_messenger_ordered_lane_wait_seconds to find a lane that falls behind. Under OverflowPolicy::Reject, velo_messenger_dispatch_failures_total{reason="ordered_lane_shed"} counts the shed messages.

Events

The event APIs require the default services feature. Messenger ACK/NACK messages are part of core and do not require it.

Velo has a generational event system to coordinate async tasks. An event has a compact u128 handle. You can share the handle between threads, or send it to another instance.

Local events

use velo::EventManager;

let manager = EventManager::local();
let event = manager.new_event()?;
let awaiter = manager.awaiter(event.handle())?;

// trigger() consumes the event, so an event cannot complete twice.
event.trigger()?;
awaiter.await?;

If you drop an Event without trigger() or poison(), the drop poisons it. A waiter never waits for an event that nobody can complete.

merge_events makes an AND gate. The merged event completes after all its inputs complete.

let weights = manager.new_event()?;
let tokenizer = manager.new_event()?;
let ready = manager.merge_events(vec![weights.handle(), tokenizer.handle()])?;
weights.trigger()?;
tokenizer.trigger()?;
manager.awaiter(ready)?.await?;

When an input is poisoned, the poison reason goes to all its waiters, and to every merged event that uses the input.

Distributed events

In a Velo instance, the event manager is distributed. The handle encodes the instance that owns the event. When you wait on a remote handle, Velo subscribes to the owner with active messages.

// Node A creates the event and sends the handle to node B.
let event = node_a.event_manager().new_event()?;
let handle = event.handle();

// Node B waits on the remote event.
let awaiter = node_b.event_manager().awaiter(handle)?;

// Node A triggers the event, and the waiter on node B wakes.
event.trigger()?;
awaiter.await?;

A remote wait looks in three places, in this order:

1. A cache of completed events. A wait on an event that is already complete returns immediately, with no network round trip.
2. An existing local subscription to the same event.
3. A new subscription to the owner, over the network.

Streaming

A stream moves a sequence of typed items from a producer to a consumer. The consumer owns a StreamAnchor<T>. A producer attaches to the anchor and gets a StreamSender<T>. The anchor has a compact u128 handle, so the consumer can send the handle to a producer on a different worker.

The handle is all that a peer needs to attach to the anchor, cancel it, or open its slot on the mux, and velo does not authenticate peers. So the local ID in the handle is random: 63 bits, and never 0. With a counter, any host that could reach a node could attach to the next handle and take over the stream of another request. Send the handle only to the producer that needs it.

Single producer (SPSC)

A default anchor accepts one producer at a time.

use futures::StreamExt;
use velo::StreamFrame;

// Consumer: create an anchor and send its handle to the producer.
let mut anchor = node_b.create_anchor::<String>();
let handle = anchor.handle();

// Producer, on the same node or another node.
let sender = node_a.attach_anchor::<String>(handle).await?;
sender.send("hello".into()).await?;
sender.finalize()?;

// Consumer.
while let Some(frame) = anchor.next().await {
    match frame? {
        StreamFrame::Item(s) => println!("{s}"),
        StreamFrame::Finalized => break,
        _ => {}
    }
}

The consumer can request graceful stop with anchor.controller().request_stop(). The producer observes sender.stop_token() and can send its final output, then call finalize(). Stop is idempotent and does not discard buffered output.

The consumer can cancel the stream with anchor.cancel() or a cloned StreamController. Cancel triggers both producer tokens and ends delivery. Dropping the consumer also cancels. Stop and cancel made before attach or ticket open are retained; the producer observes them when it opens, even if it sends no data. Transport failure remains a stream error.

Know when the output has left

finalize returns before the stream's output leaves the producer's node. Under the messenger mux, records wait on the producer for the consumer's credit, and a slow consumer can hold them there for as long as it does not read. A producer that shuts its node down in that time cuts the stream short.

sender.retired() returns a future. The future resolves once velo on the producer's node has handed on all of the stream's output; the table below says what that means on each path. It resolves after finalize, detach, or drop. Get the future before any of the three, because each one consumes the sender.

let retired = sender.retired();
sender.finalize()?;
match retired.await {
    Retirement::Sent => {}      // Handed on; see the table and the staged-item exception below.
    Retirement::Discarded => {} // Output was thrown away before it left this node.
    _ => {}                     // `Retirement` is not exhaustive.
}

Retirement::Sent means that velo handed the terminal (Finalized, Detached, or Dropped) on behind every record before it. It is a statement about the producer's node, not about the consumer: it does not say that the consumer received anything, and what "handed on" means depends on the path of the stream:

Retirement::Discarded means that the output was thrown away before velo handed it on. Over the messenger mux: the consumer cancelled or dropped its anchor, the consumer's process went away, the transport refused the batch, or the producer's node shut down while it held the output. On the same worker: the consumer dropped its anchor, so the terminal had no channel to enter; a cancel through a StreamController while the anchor lives leaves the terminal in the anchor's channel, which is Sent. On a per-stream transport: the pump's write failed, the connection ended first, or the gRPC server did not take the stream; a per-stream TCP pump cannot see a consumer that left before it writes (known limitation). A task or batch that dies with the terminal also gives Discarded, so the future does not wait forever.

The future shares the state that the sender already keeps for the stream. A sender can hand out any number of them, and each one wakes. A stream that nobody asks about pays one compare-exchange when it retires. Batched streaming shows how the mux reports it, and what it costs.

Byte items

Some streams carry items that the application has already serialized, such as response chunks. For these, use Bytes as the item type, send with send_bytes, and read with next_bytes. Inside a hand-written Stream, use poll_next_bytes.

use bytes::Bytes;
use velo::StreamFrame;

// Producer.
let sender = node_a.open_anchor_stream::<Bytes>(handle, ticket).await?;
sender.send_bytes(&chunk).await?;
sender.finalize()?;

// Consumer of `node_b.create_anchor::<Bytes>()`.
while let Some(frame) = anchor.next_bytes().await {
    match frame? {
        StreamFrame::Item(chunk) => write(chunk),
        StreamFrame::Finalized => break,
        _ => {}
    }
}

send_bytes writes the frame by hand, and the frame is the same as the one send writes for a Bytes item. So each end can use either form. next_bytes handles every frame other than an item in the same code as next, so heartbeats, errors, Detached and the end of the stream behave the same.

The difference is the copies of each payload. Over the mux:

These counts are from the source, not measured. On the same worker, there is no batch and no ingress, so send_bytes and next_bytes make one copy in total.

A kept item keeps its frame buffer in memory. The frame is the item plus at most 11 bytes.

Write an item in place

send_bytes copies a payload that the application encoded into its own buffer. To skip that copy, encode straight into the frame with ByteItemBuilder, and send the result with send_frame. The builder needs no sender, so the encode can run before the producer holds one.

use velo::streaming::ByteItemBuilder;

let mut item = ByteItemBuilder::with_capacity(size_hint);
rmp_serde::encode::write(item.buffer(), &token)?;
sender.send_frame(item.finish()?).await?;

buffer is a Vec<u8> that already holds the 11-byte item header. Append the payload to it. finish writes the payload's length into the header, and refuses a frame whose header was changed. send_frame has the rules of send_bytes for a cancel, a full channel, and the item limit.

A payload that fits the capacity costs one allocation for the whole frame, and no copy before the batch. A larger payload grows the buffer as a Vec grows.

The frame decodes as the send_bytes frame does, through next, next_bytes, and a typed consumer. The bytes differ in one way: the length is always a 5-byte bin32, because it is written after the payload. So an item under 64 KiB is up to 3 bytes longer on the wire. On the same worker, a frame with spare capacity costs the consumer one small allocation when it takes the buffer as Bytes. Over the mux, the batch copy removes that cost.

Why separate methods: next cannot skip serde for Bytes alone. Rust has no specialization, and a type check at run time needs T: 'static on the Stream impl of StreamAnchor, which is a breaking change. A marker item type would still go through serde on the producer. Why the mux still copies each record out of its batch is in Batched streaming design.

Many producers (MPSC)

create_mpsc_anchor makes an anchor that accepts many producers. Each producer gets a SenderId when it attaches, and each frame carries the SenderId of its producer. Detached and Dropped from one producer do not end the stream. The consumer ends the stream: it cancels, drops the anchor, or lets its unattached timeout (MpscAnchorConfig::unattached_timeout, none by default) fire after all producers leave. A producer cannot finalize an MPSC anchor.

How frames travel

Frames travel on one of two paths:

• A frame transport. The builder sets it with stream_config. StreamConfig::Tcp is the default. StreamConfig::Grpc needs the grpc feature. Each stream sends its own frames. This path carries a stream when the peer does not offer the mux, or when the mux is off.
• Batched streaming (the mux). The builder installs it by default. messenger_mux(MuxConfig) changes its settings or turns it off. Records from many streams to the same peer share one messenger frame, with credit-based flow control for each stream. See Batched streaming.

let node = Velo::builder()
    .add_transport(tcp)
    .stream_config(StreamConfig::Tcp(None))? // TCP streams on a port that the OS assigns
    .build()
    .await?;

Zero-RTT setup

attach_anchor makes one _anchor_attach round trip before the first item can move. Sometimes the consumer knows which worker will produce. For example, the consumer is about to send that worker a request. Then the consumer can make a ticket and put it in its own request, and the worker opens the stream with no round trip.

// Consumer: make a ticket.
let mut anchor = node_b.create_anchor::<String>();
let handle = anchor.handle();
let Some(ticket) = node_b.prebind_anchor(handle) else {
    // No ticket. Attach the usual way.
    let sender = node_a.attach_anchor::<String>(handle).await?;
    /* ... */
    return Ok(());
};

// The ticket travels inside the request of the application.

// Worker: open the stream on the ticket.
let sender = node_a.open_anchor_stream::<String>(handle, ticket).await?;
sender.send("hello".into()).await?;
sender.finalize()?;

prebind_anchor returns None when it has nothing to make. The caller then uses attach_anchor as usual:

• If no mux is installed, None is the normal result, and Velo records nothing.
• For all other causes, Velo logs at debug level and counts outcome="error" under velo_streaming_anchor_operations_total{operation="prebind"}. These causes are an MPSC anchor, the handle of another node, an anchor that is already attached or pre-bound, and an anchor that was removed. Call prebind_anchor only on SPSC anchors that this node created.

Call prebind_anchor from a runtime context, because it starts tasks. A ticket that the worker never opens does not stay forever. After the configured accept window (180 seconds by default), the bind is reclaimed and the anchor sees SenderDropped. velo_streaming_unclaimed_bind_reaped_total counts these tickets.

Ticket senders support stop_token() and cancellation_token() through messenger mux version 2. The existing slot-open exchange carries the stream session identity; no extra stream registration is needed. A peer epoch and slot generation reject stale lifecycle signals. Applications that require this lifecycle must require messenger-mux-v2; legacy attach fallback does not establish that capability.

Reservation renewal

VeloBuilder::reservation_config sets the default unattached lifetime (120 seconds) and the accept window of the messenger mux (180 seconds). The accept window applies to every mux bind: a prepared slot and an ordinary attach. The per-stream TCP transport keeps its own fixed 60-second accept window. Per-anchor AnchorConfig::unattached_timeout overrides the ordinary lifetime. The unattached default applies to SPSC anchors only. An MPSC anchor has an unattached timeout only when MpscAnchorConfig::unattached_timeout sets one, and renewal does not extend it. To make anchors wait for a sender with no limit, set unattached_timeout to Duration::MAX. An AnchorManager that drops cancels the SPSC and MPSC anchors still in its registries. Unlike stop, a drop does not ask attached senders to cancel.

ReservationConfig is #[non_exhaustive], so start from the default and change its fields:

let mut reservations = ReservationConfig::default();
reservations.prebind_timeout = Duration::from_secs(600);
let velo = Velo::builder().reservation_config(reservations).build().await?;

Capture AnchorReservation::new(handle, ticket.as_ref()) when a request takes responsibility for a reply. Keep the reservation while that request waits or executes before attachment. Periodically pass held reservations to AnchorManager::renew_reservations. The call groups them by owner and sends at most 256 entries per remote batch. It does not attach a producer. The application owns the collection, renewal cadence, and call deadline. Velo does not schedule renewal. Choose a cadence shorter than the reservation lifetime.

Renewal extends only a live, unattached reservation. Attached, expired, cancelled, and replaced reservations are successful no-ops, including attachment races. Renewal cannot revive an expired stream. A transport failure returns an error. A peer that runs an older Velo has no _anchor_renew handler, so renewal to it returns an error too. Successful delivery does not prove that a reservation still exists.

Release the held reservation after downstream acceptance, attachment, cancellation, or failure. The accept window is the only limit on an unclaimed bind. A renewal loop that outlives its request holds the bind, its buffer, and its count on a mux lane until the loop stops. Renewal does not extend an application request deadline or replace producer heartbeats. A prepared ticket still identifies one slot generation; renewal does not make that ticket reusable after detach. After a detach, renewal with that ticket extends the unattached lifetime of the anchor.

Batched streaming

The messenger mux carries every stream to one peer over the Messenger connection that already exists to that peer. It packs the records for that peer into _stream_batch active messages. Its transport key is messenger-mux-v2. The mux is on by default and is negotiated per attach. Senders do not change: StreamSender::send stages a record, and the layer below it decides when to write.

This chapter describes how the mux works. The Tune batched streaming guide tells you how to configure it. The Batched streaming design chapter records why it works this way and which alternatives were rejected.

Why one connection per stream does not scale

Without the mux, a velo stream owns a connection. The consumer creates a StreamAnchor, the producer attaches to its StreamAnchorHandle, and FrameTransport::connect returns a channel over a dedicated TCP connection. This design is correct for a few long-lived bulk streams. Each stream gets its own socket, its own kernel buffers and its own failure domain.

LLM serving has a different shape. A decode engine holds X requests in flight and emits one token per request on each forward pass. The X anchors belong to a small set of Y frontends. In a typical disaggregated deployment X is 256 to 1,024 and Y is 4 to 16. Each forward pass therefore makes X writes and X TCP segments, where Y writes carry the same payload.

Each remote stream on the per-stream path costs the following:

Per token, the stream pays one rmp_serde allocation, one channel hop, one encode_frame and, because TCP_NODELAY is set, one syscall and one TCP segment.

This cost is a ceiling, not a slope. Each remote stream holds one socket, with one file descriptor in each of the two processes. Across both ends, 1,024 concurrent remote streams need 2,048 descriptors, about 4 GiB of requested socket buffer and about 4,096 tasks. Each process holds one descriptor per stream. At the default ulimit -n of 1,024, a process stops below 1,024 concurrent remote streams, less the descriptors that it uses for other things.

The rate of new streams is a limit too. On a cluster (2026-09-23, two Grace nodes on 200G Ethernet, 512 mock workers in 8 processes, concurrency 8,192), a response plane that opened about 2,500 streams per second on the per-stream path failed 80% to 92% of its requests. The workers could not get a local port (Cannot assign requested address). The mux carried the same load with no errors. For this reason the mux is the default.

Per-stream coalescing cannot reach the forward-pass shape

The per-stream egress pump packs every frame already queued on its stream into one write. The test lib/velo/tests/streaming/tcp_batching.rs measures this on TCP loopback. The ratio is velo_streaming_frames_written_total divided by velo_streaming_egress_flushes_total.

A forward pass puts one frame on each of X different streams. Each pump wakes with exactly one frame, so per-stream coalescing does nothing. The frames are on the wrong axis. Only bucketing by destination worker reaches them. The test forward_pass_shape_does_not_coalesce_per_stream asserts this limitation, so it fails if per-stream coalescing is ever mistaken for a solution.

Design overview

The mux has four mechanisms:

1. Multiplexing. Every stream to a peer becomes a slot on the Messenger connection to that peer. Streaming owns no sockets.
2. Batching. Many records go into one active message and one write_all.
3. Flow control. Per-slot credit keeps one slow consumer from stalling the shared ordering lane of its peer.
4. Flush policy. A policy decides when the batcher writes.

Bucketing by destination costs nothing. StreamAnchorHandle packs a WorkerId into its upper 64 bits, and WorkerId is a 1:1 hash of InstanceId. handle.unpack().0 is the batching key, with no map lookup and no discovery hop.

flowchart LR
    subgraph Producer node
        S1[StreamSender] --> I1[slot inlet C+1]
        S2[StreamSender] --> I2[slot inlet C+1]
        I1 --> B[PeerBatcher, one per peer]
        I2 --> B
    end
    B -->|_stream_batch active message| M[Messenger transport]
    subgraph Consumer node
        M --> L[ordered lane for this sender]
        L --> H[ingress handle_batch]
        H --> Q1[slot buffer C+1]
        H --> Q2[slot buffer C+1]
        Q1 -->|read directly| C1[StreamAnchor]
        Q2 -->|read directly| C2[StreamAnchor]
        H -.->|CreditUpdate replies| RB[PeerBatcher back to producer]
    end

Protocol

Riding the Messenger

MessengerMuxTransport implements the streaming FrameTransport contract. It has no dial, no listener, no acceptor and no connection manager. The sender's identity arrives in the active-message envelope, so credit has a return route without a handshake.

Egress is one PeerBatcher per remote instance and lane. The batcher is created on the first send to that peer on that lane, and evicted when it is idle with no live slots. A node that talks to Y peers on one lane holds Y batchers, whatever its stream count. The per-stream design is O(X) in tasks and sockets. With the mux, sockets and batchers are O(Y). Each stream still has its own stream watchdog on the consumer, so consumer tasks stay O(X). On the producer, one task per AnchorManager sends the heartbeats of all streams. The watchdog wakes once per heartbeat window, not once per record.

The cost of this design is that streaming no longer owns its wire. It shares queues, framing and backpressure with control traffic. Stream order is no longer a TCP guarantee. It is a protocol obligation, and every record carries a per-slot sequence number for this reason.

Frame envelope

A batch is the payload of one _stream_batch active message. The Messenger frames the active message, so the batch carries only its own header:

_stream_batch payload:
  [16 B batch header][record_count x record]

batch header:
  [u8 mux_version = 2][u8 flags][u16 record_count][u64 peer_epoch][u32 batch_seq]

record:
  [u8 record_type][u32 slot][u32 frame_seq][u32 len][len bytes body]

record_type: 0 = Data, 1 = OpenSlot, 2 = CloseSlot, 3 = CreditUpdate, 4 = SlotHeartbeat, 5 = LifecycleSlot

Every multi-byte field is big-endian, in the header and in each record. The low four bits of flags carry the lane of the batch. The high four bits are reserved and are zero. A lane-0 batch writes flags as zero, as senders did before lanes, and a receiver from before lanes ignores the byte.

The sender bumps peer_epoch each time it re-establishes its view of the peer. batch_seq advances within an epoch and is compared modulo 2^32. Together they let ingress discard a stale epoch's batches by header inspection, and they meter gaps. frame_seq is per slot and is the authority on stream order.

Each record header is 13 bytes. On a 40-byte token that is about 33% overhead, against 9 bytes on a dedicated connection. The two sequence numbers pay for ordering that a private TCP connection gave for free.

Record bodies:

• Data carries the rmp_serde-encoded StreamFrame bytes, identical to the per-stream path. is_terminal_sentinel() works unchanged on a record body, so terminal detection has one code path.
• OpenSlot carries [u64 anchor_id][u64 session_id]. This is the 16-byte attach handshake, moved into a record. The anchor id is random, so a peer cannot guess the pair. See Streaming.
• CloseSlot carries [u8 reason]: 0 terminal sent, 1 peer gone, 2 unknown slot, 3 protocol error.
• CreditUpdate carries [u32 delta] from receiver to sender.
• LifecycleSlot carries the 64-bit session identity from OpenSlot and a one-byte action (0 = stop, 1 = cancel). Stop leaves the slot open for remaining output. Cancel takes precedence for the same session. Both the queued signal and the live producer check session identity, so a delayed signal cannot target a reused slot even if its compact generation has wrapped. Early stop is retained by the pre-bind until OpenSlot claims it.
• SlotHeartbeat has no body. The decoder accepts it, but no sender emits it. See Heartbeats.

CloseSlot travels in both directions and has no direction bit. The reason carries the direction. TerminalSent and PeerGone travel from slot owner to receiver. UnknownSlot and ProtocolError travel from receiver to slot owner. Both sides can hold a slot at the same dense index, and the reason tells them apart.

Lanes

A lane is one ordered channel of batches from a sender to a peer. Order holds per (peer, lane), not per peer. Each slot stays on one lane for its whole life, so the order of each stream holds.

The mux keeps up to 16 lanes to a peer, and mux lane k rides transport lane k. The count follows the transport: Transport::lanes(peer), capped at 16. A QUIC or TCP transport built with lanes(8) gives 8 mux lanes. The default of one lane is the behavior from before lanes.

Each lane has its own batch handler, ordered per sender. Lane 0 keeps the name _stream_batch, so a peer from before lanes sends and receives on it. Lanes 1 to 15 use _stream_batch.1 to _stream_batch.15. Every node registers all 16 handlers when it is built. At that time the lane counts of its peers are not known, and they can be different for each peer and each transport.

Everything that depends on order is kept per (peer, lane): the batcher on the sender, and the epoch, batch_seq and slot table on the receiver. Slot ids are unique only within one batcher, so a table shared by two lanes would let one lane retire the slots of the other. Replies (credit, closes, stops) go back on the lane that the batch arrived on.

The receiver of a stream names its lane, in the attach response or in the StreamOpenTicket. The sender uses that lane modulo the lanes its own transport keeps to the receiver. A sender with one lane always uses lane 0. Any lane is correct for any stream, because every node takes batches on every lane. Only the spread changes.

The receiver chooses the lane once, when it binds the slot:

• With a key. Velo::attach_anchor_keyed(handle, key), Velo::attach_mpsc_anchor_keyed(handle, key) and Velo::prebind_anchor_keyed(handle, key) take a u64 key. The receiver puts the stream on lane hash(key) % lanes. The hash is splitmix64, fixed in the code, so one key gives one lane index in every build, on every receiver with the same lane count. The key is a placement hint, not an ordering guarantee. Streams with one key to one receiver share a lane, but the mux never orders the records of one stream against the records of another.
• Without a key, on attach. attach_anchor and attach_mpsc_anchor put the stream on the lane with the least load from that sender. The load of a lane is its live slots plus the binds that the receiver answered and no OpenSlot has claimed yet. The unclaimed binds count because an OpenSlot arrives only with the first batch of the sender. Without them, attaches answered at the same time all go to lane 0.
• Without a key, on pre-bind. prebind_anchor does not know the sender. It puts the stream on the lane with the least load on this node. The load of a lane is its live slots from every peer, plus the pre-binds on it that are not yet claimed, released or expired. The live slots count because a frontend's tickets are claimed within milliseconds and the streams then live for seconds. If only unclaimed pre-binds counted, almost every choice would see all lanes empty, and almost every stream would go to lane 0.

Ties go to the lowest lane. lanes is the mux lane count of the transport to the sender. A pre-bind has no sender, so it uses the most lanes that any installed transport keeps. This is correct while Transport::lanes() gives one count for all peers, as every transport in velo does.

A stream counts on the lane that the receiver chose, from the bind until its slot closes. That is not always the lane that its batches arrive on. A sender with fewer lanes sends lane k on lane k modulo its own count. If the load counted on the arrival lane, the lanes above the count of the sender would never get load, and every unkeyed stream would go to the first of them.

A receiver whose transport keeps one lane names lane 0 for every stream, with a key or without one. A default deployment therefore sends the same bytes as before lanes.

The new fields are last in each message, default to zero when absent, and are not sent when zero. A lane-0 ticket or attach message is therefore the same bytes as before lanes, in JSON and in MessagePack. A ticket that names another lane has one more field, which a worker from before lanes refuses under positional MessagePack. Upgrade the workers before the node that mints the tickets.

The batch header repeats the lane. Ingress drops a batch whose header lane is not the lane of its handler, and counts its records as lane_mismatch in velo_streaming_mux_records_dropped_total. The slot ids of such a batch belong to another batcher, so applying them could feed or retire the wrong streams.

Across two nodes over QUIC, 64 streams of 16 KiB items moved about 750 MiB/s on 1 lane and 5,300 MiB/s on 8 lanes. The setup and the full table are in QUIC performance.

Each (peer, lane) table takes the whole range of 65,536 slot indices. The sender picks the lane that a stream rides, so one table can hold every stream of a peer: a sender with one lane puts all its streams on lane 0. All the tables of one peer share one byte budget, peer_byte_budget. The bound for each peer is therefore the same at any lane count on either side, and it does not depend on when each table was made.

The batch size cap

Three numbers clamp a batch, and the smallest one binds:

• MuxConfig::max_batch_bytes, 60 KiB by default.
• The effective eager budget. This is the smaller of Transport::max_message_size(target) (where the transport reports one) and the rendezvous staging threshold, less the envelope overhead. It is the largest payload the Messenger carries inline to that peer.
• COALESCE_THRESHOLD, 64 KiB. The coalescing writer packs a frame into its buffered write_all only if the frame fits under this threshold. A larger frame is written in segments, which gives back the syscall saving.

A batch over the eager budget does not fail. It becomes a rendezvous transfer and pays a round trip for every slot packed into it. The cap exists to keep batches eager.

A single record larger than the eager budget goes through rendezvous alone, in its own batch. Other slots continue in eager batches while that transfer is in flight. Ordering describes how the slot stays in order.

Slots

SlotId = (u24 index, u8 generation), packed into a u32, scoped by the sender's peer epoch

  bits 31..8 : index        bits 7..0 : generation

The index sits in the high bits so that the raw u32 sorts by index. A dense index makes demux a Vec lookup instead of a hash. A 60 KiB batch holds about 1,100 records, so this lookup is on the hot path.

The generation is a correctness requirement. Without it, a stale record for a reused index goes to the stream that now holds that index. Request A's tokens then appear in request B's response, silently. Ingress drops a record whose generation does not match the current occupant and meters it in velo_streaming_mux_generation_mismatch_total.

The epoch scopes the whole table above the generation. A generation survives slot reuse within one sender's lifetime. The epoch survives the sender itself. Ingress therefore discards batches from an old epoch as a whole, and a u8 generation is enough.

Ordering is per slot

The mux registers each lane's batch handler (_stream_batch, _stream_batch.1, ...) with ordered per-sender dispatch. One task handles the batches from one peer on one lane, in arrival order. The general reordering problem does not arise, and no reorder window is necessary.

One exception exists. A rendezvous payload resolves in a detached task before dispatch, so an oversized record is not ordered against the eager batches around it. Two mechanisms bound this:

1. The egress fence. A slot has at most one fenced singleton outstanding. While the fence is up, the batcher withholds the later records of that slot until the staged send is admitted. This includes the slot's CloseSlot, because a close must not overtake the record in front of it. Only the resolution of that singleton lifts its own fence. A rendezvous record always fences its slot, even when the transport admits it synchronously, because the receiver resolves it outside the ordered lane.
2. The ingress hold. Ingress keeps a record that arrives ahead of its frame_seq in a per-slot hold, and applies it when the gap closes. Credit admission and the slot and peer byte budgets bound the hold. An overflow closes that slot with ProtocolError and meters velo_streaming_mux_hold_overflow_total. The consumer sees Dropped. Other slots and the lane continue.

The fence lifts when the rendezvous send is admitted, not when the receiver has the record. The slot's terminal and CloseSlot then go in an eager batch, and the sender frees the slot index. Its free list is last in, first out, so the next stream it opens on that lane takes that index at once. That OpenSlot can reach ingress while the old slot still waits for its rendezvous record, with its terminal in the hold and its close deferred behind it. Ingress has the close, so the sender is done with the old slot. Ingress moves it aside, kept by its full slot id, and opens the new slot at the index. The old slot then ends as it would have in the table.

An OpenSlot over a slot whose close has not arrived is still a protocol error. Ingress rejects the new slot with ProtocolError, leaves the old one alone, and counts slot_collision in velo_streaming_mux_records_dropped_total. One legitimate case stays a collision: a terminal larger than the eager limit (a TransportError with a very large message) goes through rendezvous together with its CloseSlot, so ingress has no close to see.

The generation in a slot id has 8 bits. After 256 streams on one index, the sender can open a slot with the id of an old slot that still waits aside. A record names its slot by id alone, so ingress cannot tell the records of the two streams apart. It rejects the new slot as a collision and fails the old one, so that neither stream gets the other's records.

Two other designs were rejected. The sender could keep an index until the receiver acknowledges the close, but that adds a record to every stream. The lane could wait for each rendezvous record before it handles later batches, but then every stream from the peer on that lane waits behind one large transfer.

A record refused for its size

A receiver built with VeloBuilder::staged_payload_max refuses a staged batch that states more bytes than that, before it allocates them (Rendezvous). It never reads the batch, so it cannot learn from the batch which slot the record was for. Each singleton batch that the messenger stages therefore names its stream in a header, _mux_slot: the sender's epoch, the slot's full id, and the routing session the slot was opened with, as <epoch>:<slot>:<session>. An inline singleton cannot be refused, so it carries no header: it may already be at the transport's frame limit. On a refusal, ingress finds that slot in the table of the (peer, lane) the batch came from, gives its consumer a TransportError that names both sizes as the terminal, retires the slot, and tells the sender to cancel that stream with LifecycleSlot, which names the session. A CloseSlot would name only the slot id, and a late one could end a later stream that reuses the id. The producer's next send fails. Other slots are not touched.

The slot ends at once. It does not wait for records ahead of the refused one that are still on their way, because the stream has already failed. A terminal that overtook the refused record waits in the hold and is dropped with the slot, so the error is delivered even when that terminal has already spent the slot's terminal reserve: the consumer always ends with the error, never with nothing. All three are checked under the table's lock. A resolve can outlive its stream, and a slot id alone does not name one lifetime: a replaced batcher reuses ids, and within one epoch a cancelled slot's index is reused and its eight-bit generation wraps. The session is the receiver's own u64, unique on its node and never reused, so a stale refusal finds no slot and does nothing. Without the header, the slot would keep a gap that never closes, and its consumer would wait for the stream watchdog. The id is the sender's word, but it is looked up only among that sender's own slots, so a sender can end only its own streams.

The refusal comes from the resolver, not along the ordered lane, so it can arrive while the stream's OpenSlot still waits on that lane. The slot is not open yet, but the stream's bind is waiting for it, so the refusal waits on the bind, and the OpenSlot that claims the bind ends the stream at once. It must match the (peer, lane), the epoch and the slot id that the refusal named. A session is not secret, so the bind keeps one waiting refusal for each (peer, lane): another peer that names the same session can neither replace nor block it. Both happen under the table's lock, so the claim cannot fall between the check and the mark. Without this, the refusal found no slot, the OpenSlot opened one, and its consumer waited for the stream watchdog.

Batches from one peer on one lane must not be reordered. The worker writes each OpenSlot in its own batch. A Data batch that overtakes it finds no slot and is dropped as closed_slot. The slot then opens with its sequence past the lost record, which silently truncates the head of the stream.

Opening a slot

The sender writes OpenSlot when the stream attaches, in a flush of its own. It does not wait for the first data record. bind() starts an accept window of ReservationConfig::prebind_timeout (180 seconds by default). The window measures the time until a batch that carries the OpenSlot arrives. A lazy OpenSlot makes the window measure the time until the first token, and a queued request with a long prefill then expires.

An OpenSlot for an (anchor_id, session_id) pair that was never registered does not fail the peer. The receiver replies CloseSlot{UnknownSlot} and discards the records of that slot. When the accept window closes on an unclaimed bind, the stream watchdog (or the consumer, if it sees the close first) reaps the registry entry, injects Dropped and increments velo_streaming_unclaimed_bind_reaped_total.

By default, connect returns after the transport admits the OpenSlot. MuxConfig::async_open_ack changes this. The OpenSlot still goes out in a batch of its own before connect returns, but the acknowledgement does not wait for admission. If the transport admits the frame synchronously, per-target FIFO already orders the slot's later records behind it, and no fence goes up. Otherwise the slot is fenced until the admission resolves. A failed admission is epoch death in both modes. Measured at load, this option did not improve first-token latency. See Response plane performance.

Zero-RTT stream setup

The receiver chooses every field of the attach response without input from the sender. It can therefore bind a slot before any sender asks. AnchorManager::prebind_anchor does the work of the attach handler at request registration. It binds the slot, allocates the routing session, takes the drain signal, installs the direct feed and spawns the stream watchdog. It returns a StreamOpenTicket with the six values an attach response carries. The application puts the ticket in the request envelope that it already sends to the worker.

The worker calls AnchorManager::open_anchor_stream with the ticket. Its first batch carries an OpenSlot, which claims the pre-bound slot the same way an attached sender's does. No _anchor_attach crosses the wire. The wire format does not change: StreamOpenTicket is a separate type in the application's envelope. When no mux is installed, prebind_anchor returns None and the stream attaches the ordinary way.

The pre-bind has these rules:

• An attach adopts an unclaimed pre-bind. A sender that attaches the ordinary way receives the waiting slot and the ticket's session id. An attach against a pre-bind whose OpenSlot already arrived is refused. A sender that does not offer the pre-bound key is refused, and the pre-bind is released.
• A same-worker attach releases an unclaimed pre-bind. A same-worker sender writes into the anchor channel directly and never claims the slot.
• A dead request gives the bind back. PreBind's Drop releases an unclaimed bind. Every path that kills an anchor removes its registry entry, so this needs no extra call site.
• Cancellation posts a close. A zero-RTT anchor never learns a StreamCancelHandle. When a claimed pre-bind drops, it posts CloseSlot{UnknownSlot} to the producer through close_claimed_slot. An idle producer learns of the cancel without sending another record.
• Detach gives the slot back. Both places that handle Detached release the pre-bind and resume the unattached timer.

The ticket can wait in an envelope for the full configured accept window (180 seconds by default). Heartbeat detection starts only when the OpenSlot that claims the bind arrives. Before that, the stream watchdog does not count a silent window as a miss.

Credit is exact by construction. prebind sizes its buffer from the same NegotiatedLimits that the ticket quotes. open_slot emits no CreditUpdate on the claim, so the sender never holds 2C credit against a C+1 buffer.

The rollback is not symmetric. Disable the mux on the minting side first, or on both sides together. A producer that disables the mux alone still advertises its default transport key. A consumer that still pre-binds refuses that attach, because the key does not match the pre-bind.

The rollout has the same asymmetry in reverse. The mux is on by default, so a consumer mints tickets as soon as it runs a version with the mux. A producer without the mux cannot open them. Upgrade the producers first, or keep the mux off on the consumers that mint tickets until every producer has it.

Peer loss

While a batcher has live slots, it checks the health of its peer every 5 s. A failed health check, peer eviction and batcher eviction end in epoch death. A failed admission of a batch is also epoch death. The transport refused the batch, so it never reached the wire and left a frame_seq gap in every slot that it carried. The mux does not retransmit, so those slots cannot make progress again.

At epoch death the batcher closes every live slot of its (peer, lane) and starts a new epoch. The receiver retires the slots of the old epoch when the first batch of the new epoch arrives on that lane. Every retired slot that has not seen a terminal receives an injected StreamFrame::Dropped. The consumer sees StreamError::SenderDropped, the same as on the per-stream path. TransportError stays reserved for protocol violations. If no batch of the new epoch arrives, for example because the lane still refuses, the stream watchdog ends each slot.

A connection that closes after the transport admitted a batch is not epoch death. The mux does not wait for a response to a batch: the send drops its response awaiter at admission. A later failure of the send goes only to the process-wide error handler of the Messenger, which logs it. QUIC and TCP dial the lane again on the next send, and the batcher keeps its epoch. A batch that still waits for admission when the connection closes is refused (ChannelClosed or ConnectionReplaced), and that refusal is epoch death. The receiver sees the lost batches as a gap in batch_seq, and as a gap in frame_seq in each slot that they carried.

A slot with a gap cannot make progress. Its stream watchdog ends it after DETECTION_MULTIPLIER heartbeat windows, 15 to 20 s at the defaults. A slot whose OpenSlot was lost never opens. If the slot was pre-bound, its consumer waits until the configured accept window (180 seconds by default) closes the bind; an attached slot is ended by its stream watchdog, like a slot with a gap. A lost reply batch loses the credit and the cancels that it carried. The consumer's account then shows that credit as out, so its watchdog ends a slot whose grant was lost. Slots that lost nothing continue.

Each lane has its own batcher, connection and epoch. A failure on one lane therefore touches only the slots of that lane. Streams to the same peer on other lanes continue, unless the peer fails its health check.

Flow control

The shared resource is the ordering lane of one (peer, lane). A batch handler that awaits holds that lane, and every slot from the peer on that lane stalls behind it. Lane channels are unbounded, so a blocking handler turns backpressure into unbounded memory growth. With a blocking handler, one saturated anchor stalls every stream from that peer on its lane and fires all of their heartbeat watchdogs at once. For inference, one slow HTTP client then throttles the GPU. Ingress is therefore bounded and nonblocking, on per-slot credit.

Credit against a mux-owned buffer

Credit is issued against a mux-owned per-slot buffer, never against the anchor's frame_tx. frame_tx has other writers: the same-worker attach path, the detach and finalize handlers, the watchdog's Dropped injection and M concurrent MPSC senders. Any proof of "C credits against a C-deep channel" fails when a second writer exists. bind returns a receiver of depth C+1. The StreamAnchor reads that buffer itself, before its own frame_tx, and counts each record it takes. No task sits between the buffer and the consumer.

The mux publishes the buffer to the anchor through a replaceable feed. Retiring the bind (detach, a released pre-bind, a same-worker attach over a pre-bind) or removing the anchor withdraws the feed. The consumer drops a withdrawn feed on its next poll, before it reads that buffer again, so records still unread there are discarded. While a feed is installed, nothing on the anchor channel can jump ahead of a record the consumer can still read.

Invariant: a slot never has more than C data records outstanding against its C+1 buffer. The ingress handler only calls try_send into space that credit already reserved, and it never blocks the lane. velo_streaming_mux_reader_stall_total > 0 is a bug, not a tuning signal.

The initial credit C is negotiated. The receiver advertises initial_credit and slot_byte_budget in AnchorAttachResponse::Ok and in the MPSC equivalent. A slot opens already holding that window, so no round trip occurs before the first token. Continuing credit returns ride CreditUpdate records.

Each slot reserves one credit that only a terminal sentinel can spend. Data spends only C. One reserved credit is enough because sent_terminal allows at most one terminal per slot. SenderError is not a terminal and spends data credit. Control records (OpenSlot, CloseSlot, CreditUpdate) do not occupy the buffer, so data exhaustion never blocks them.

Byte budgets

Frame credit alone bounds memory at slots × C × the maximum frame size, which is not a useful bound. The per-stream socket enforced about 1 MiB per stream for free. The mux shares one connection, so it enforces its own limits:

• MuxConfig::slot_byte_budget, 1 MiB per slot by default.
• MuxConfig::peer_byte_budget, 8 MiB per peer by default. All the lanes of a peer share it.

Frame credit proves that no head-of-line blocking occurs. Byte credit bounds memory. The two grants can disagree, for example C records of 1 MiB each against a 1 MiB slot cap. The byte side wins by withholding the next grant while the slot is over its byte watermark. It does not refuse a record whose frame credit was already granted. The ingress hold is the one place where a byte reservation refuses a record, because the alternative is unbounded growth behind a gap.

Egress: the withheld queue

The batcher pulls every slot's inlet, whether or not the slot has credit, up to the slot byte budget. A slot with no credit keeps its records in a per-slot withheld queue. The queue is FIFO, so a terminal in it still waits for the records in front of it. finalize, detach and Drop reach the inlet from synchronous code. When the inlet is full, the terminal waits in a task, so these calls do not block.

A producer that runs a slot's byte cap ahead of its consumer waits. The batcher stops pulling from that slot's inlet, the inlet fills, and StreamSender::send waits until credit returns. Other slots continue. Stream saturation describes this backpressure from the operator side.

The batcher's own control inlet is coalesced state, not a queue. Credit returns accumulate into a u32 per slot. A close supersedes the credit of its slot, and a failed singleton supersedes a successful one. The batcher is woken, never fed. A queue is unbounded exactly when a flush parks on admission, which is when the peer is busiest returning credit.

Each control map is bounded by what the batcher allocated, not by a size cap. One map holds the control that a peer sends about this side's slots. It accepts a key only for an index that this batcher opened, at its live generation. A key for an index never allocated is refused and counted in velo_streaming_mux_control_refused_total. A key for a retired generation is dropped silently, because that is an ordinary close-then-reopen race. Replies that this side's ingress writes for slots it admitted are never refused. Rejections of an OpenSlot that ingress never admitted go on a separate reject lane, capped at 8,192 entries. A dropped rejection costs no credit, which is why a cap is safe there and nowhere else.

Admission

Transports expose an ordered admission gate, per target and lane, as SendOutcome::{Admitted, Pending(SendAdmission)}. A batcher learns at the send site, in order, that its peer is congested, and parks itself instead of a runtime worker. A batcher parked on admission stops draining its inlets. The producer then waits on a full inlet. This wait is bounded by the progress of the transport, which is the same position a socket was always in.

Credit return

Credit comes back from three paths. Two of them visit only slots that something named. The third walks the whole table as a backstop.

• Drain signal. When the consumer takes a record out of the slot buffer, it increments an exact count on the slot's DrainSignal. It then sets the slot's bit in the peer's dirty set, a lock-free bitmap. Only the drain that sets the bit posts the peer; a drain that finds the bit already set changes nothing shared. The consumer takes no lock.
• Arrival path. On every inbound batch, handle_batch reconciles the slots that the batch delivered into and the slots in the dirty set. The credit that a stream's tail waits for rides the peer's next batch, which arrives in tens of microseconds.
• Doorbell. The sweep task answers a peer wake by reconciling the slots in its dirty set. MuxConfig::drain_visit_floor (2 ms by default) limits it to one visit per (peer, lane) per floor. Each lane has its own table and lock, so a peer with N lanes can take up to N visits per floor, one on each lane. This path covers a peer that sends no further batches.
• Periodic tick. Every MuxConfig::credit_sweep_interval (200 ms by default), the sweep walks every slot of every ingress peer. This covers a slot that nothing names: parked, with nothing arriving and nothing being taken out. The same tick evicts idle batchers. Each walk is skipped when it has nothing to visit: the slot walk while no slot is live, the eviction check while no batcher exists. So an idle node's tick allocates nothing.

The dirty set carries an index and no quantity. The quantity is the count on the slot's own DrainSignal, and IngressSlot::reconcile swaps it to zero. A redundant visit therefore finds a count of zero and grants nothing. The three paths can run concurrently without double-counting. A stale listing costs a visit, never credit.

sequenceDiagram
    participant P as Producer batcher
    participant I as Consumer ingress
    participant R as Consumer (StreamAnchor)
    participant S as Sweep task
    participant B as Consumer batcher
    P->>I: _stream_batch (Data records)
    I->>I: deliver into slot buffer C+1
    R->>R: take record from slot buffer
    R->>R: DrainSignal count += 1, list slot in dirty set
    R-->>S: post peer wake
    P->>I: next _stream_batch
    I->>I: reconcile touched slots and dirty-set slots
    I->>B: CreditUpdate replies
    S->>I: doorbell visit (at most once per drain_visit_floor)
    S->>I: periodic whole-table walk (every credit_sweep_interval)
    B->>P: _stream_batch (CreditUpdate records, after reply_linger)

Credit for a record returns when the consumer takes it, not when it enters the mux buffer. A single-sender consumer that stops polling therefore holds its sender to the credit window C: no task moves records onward and returns credit for them. An MPSC anchor still has a pump that moves records into the anchor channel, so its senders can run ahead by that channel's depth as well.

A producer that ran out of credit sends no batches, so the arrival path does not run for its slots. It waits for the doorbell floor and then for the reply linger. Per record this costs (drain_visit_floor + reply_linger) / initial_credit. At the defaults (2 ms, 1 ms and 32) that is about 94 µs per record.

The periodic walk does not reclaim credit for a slot whose consumer is gone, because a consumer that is gone counts no drains. The slot is closed instead: removing a single-sender anchor closes its slot and tells the sender, whichever way the stream ended. An MPSC anchor closes the slot of each sender when that sender's pump ends without a terminal, for example when the anchor is dropped. A sender parked at its byte cap then wakes with an error.

Reply linger

A CreditUpdate does not write a batch of its own. MuxConfig::reply_linger (1 ms by default) starts when a reply with no window already running joins the batch. The window belongs to that reply. A record that joins later does not cancel it.

What ends the wait early depends on the flush policy:

• Under Auto { on_admission: true }, any record that is not a credit reply ends the wait. A batch that holds only replies waits out the window, or until a close, a terminal or an application flush.
• Under Manual and Auto { on_admission: false }, ordinary staging does not end the wait. Data that joins a pending reply is written with it when the window ends.
• Under every policy, opening a slot on the peer ends the wait, because OpenSlot has its own flush.

No policy holds a reply past its window. A reply held for good starves the peer's sender, and no application on this side knows that it owes the peer anything. Duration::ZERO makes each reply urgent again. CloseSlot replies are always urgent.

Terminal sentinels

On the per-stream path, the egress pump writes a terminal, discards anything queued behind it and closes the socket. A frame sent after a terminal races the consumer's cleanup. The mux keeps this rule, scoped to one slot:

1. Egress sees is_terminal_sentinel(body) for slot S, appends the record to the current batch, marks S draining and drops the inlet receiver of S. Frames queued behind the terminal for S are discarded. Other slots continue.
2. Egress appends CloseSlot{TerminalSent} in the same batch, right after the terminal. Terminal and close are atomic.
3. The batcher frees the slot, bumps its generation and releases its credit and byte budget. The peer batcher and the peer's ordering lane continue.
4. On the receive side, the terminal spends the reserved credit. Then CloseSlot drops the mux-side sender. The consumer reads the terminal and then sees the buffer close, the same as a receiver does when a socket closes, and the stream watchdog exits.
5. A CloseSlot with any reason other than TerminalSent, and every epoch death, injects Dropped for a slot that has not delivered a terminal.

finalize, detach and Drop hand the terminal to the inlet without blocking a runtime worker. They call try_send first. On a full channel, they hand the record to a task that awaits space. The task holds a clone of the sender, so the channel stays open until the sentinel is in it. The task runs on the caller's runtime, or on the sender's own runtime when the caller has none, so no caller thread blocks. detach clears the attachment flag only after the sentinel is in the channel, so a re-attach cannot put records ahead of the Detached frame.

velo_streaming_mux_live_slots must return to zero at teardown. A leaked slot holds credit and byte budget for the rest of the epoch, and unlike a leaked socket it does not show in lsof.

Retirement

StreamSender::retired() resolves when the producer's node has handed on all of the stream's output (see Streaming). Under the mux, the batcher reports it:

• When egress writes a terminal into the open batch, it moves a guard for the stream into the batch. When the batch is admitted, each guard reports Sent. A terminal that goes alone through rendezvous gives its guard to the task that waits for its admission.
• A guard that is dropped reports Discarded. A refused batch, an epoch death, a teardown, and a batcher task that is aborted all drop the guards that they hold.
• An item larger than the eager budget goes alone, as a handle the consumer pulls. Its guard reports Sent when that handle's batch is admitted, while the item is still staged on the producer (known limitation).
• A slot that closes while it still holds its producer's lifecycle reports Discarded. The terminal path takes the lifecycle out of the slot first. So every other close reports the loss: the CloseSlot{UnknownSlot} that a consumer sends when it cancels or drops its anchor, an epoch death when the consumer's process goes away, and the teardown of the producer's node.

The state is three bits in the per-stream allocation that the sender and its slot already share (the outcome, and whether anyone waits), and a list of wakers that stays empty until a Retired future waits. No message crosses the wire for it. The first outcome stands. A stream that nobody watches pays one compare-exchange when it retires, and no wake.

The alternative is an acknowledgement from the consumer in the application's own protocol. It costs one more active message for each stream, a table of waiters on the producer, and a round trip before the producer knows.

Measured with the stream_lifecycle example (Benchmarking): two nodes in one process over loopback TCP, 200,000 streams of four 16-byte items, 64 at once, on the example's 4 worker threads. Provisional: an Apple M5 laptop (10 cores, macOS), shared with other builds and test runs (load average 2.3 to 3.3), release build, /usr/bin/time -l target/release/examples/stream_lifecycle --mode <mode>. The modes ran interleaved, so contention hit each alike, and the figure is the process's instructions divided by the streams, median of nine runs (three for ack), with the spread of the runs. These costs are to be confirmed on dedicated hardware, and only instruction counts are compared, not wall time or tail latency.

Awaiting retired() costs about 600 instructions per stream here, under 0.5 percent; the acknowledgement costs about 38,000, 30 percent, and a round trip. A stream that nobody watches pays one compare-exchange when it retires; this run has no build without retirement, so it does not measure that.

A second run compares a build without retirement. It ran on an idle Linux host (20 Arm cores: Cortex-X925 and Cortex-A725), release build, with the same harness and defaults. Hardware counters are not readable there, so the figure is the process's CPU time (user and system) divided by the streams, median of fifteen runs, with the builds and modes interleaved. The build without retirement is the same harness on the commit before retirement, with the retired mode left out.

A stream that nobody watches costs the same with and without retirement, on both paths: the medians differ by less than the spread of the runs. Throughput is the same too (86,000 and 87,500 streams per second on the mux; 15,160 and 15,146 per stream). Awaiting retired() costs about 3 µs of CPU per stream on this host.

Heartbeats

One task per AnchorManager sends the heartbeats of its remote senders. Once per heartbeat interval that the consumer advertised, it calls try_send with a StreamFrame::Heartbeat into the channel of each sender. On a full channel the heartbeat is dropped. A sender on the anchor's own worker gets no heartbeats, because no watchdog reads them there. Under the mux, a heartbeat is an ordinary Data record. It spends data credit and shares the peer's queues with data.

On a single-sender mux stream, the stream watchdog detects silence. It is one task per stream, and it holds no data. It wakes once per heartbeat window on its own timer, or when the mux closes the bind. A window is live if the ingress delivered anything to the slot during it, or if the sender holds no data credit and nothing waits behind a sequence gap in the reorder hold. A consumer that is behind leaves its sender without credit, by leaving its window unread or by holding the byte budget so that credit is withheld, and a sender without credit cannot send a heartbeat, so that silence is not the sender's. After DETECTION_MULTIPLIER dead windows (3 × 5 s at the manager default), the watchdog injects Dropped, removes the anchor and increments velo_streaming_heartbeat_watchdog_firings_total. The windows run on the watchdog's own clock, so detection lands between DETECTION_MULTIPLIER and DETECTION_MULTIPLIER + 1 windows after the last arrival. Before a pre-bound slot has a sender, windows do not count. An MPSC anchor over the mux keeps its reader pump and that pump's watchdog. On any transport, an MPSC pump neither logs a firing nor counts it in velo_streaming_heartbeat_watchdog_firings_total.

A sender that still holds credit can heartbeat, so its silence counts even with records unread: a worker that dies behind a slow reader is detected on time. A sender that died while holding no credit is detected once its consumer reads enough to return credit to it.

On the per-stream path, the reader pump detects silence. It arms one pinned timer per stream. On each received frame, it stamps the time after the forward completes. It moves the deadline only when the deadline is within half a window. Under steady traffic the timer never fires and moves at most twice per deadline. When the timer fires with no frame inside the window, the pump counts a miss. After DETECTION_MULTIPLIER misses, exactly that many windows after the last frame, it injects Dropped and increments the same counter.

A per-stream heartbeat does not detect a hung producer, because it runs on a separate task. It detects process or host death, connection death and sustained saturation. Saturation shows because a full channel drops heartbeats. Under the mux, the health check of the batcher detects a dead peer on the sending side, and the mux learns of it through epoch death. The health check runs only while a batcher has live slots to send on. On a node that only receives, a dead producer is still the watchdog's job: its slots retire only when a batch of a new epoch arrives, or when the watchdog fires. A connection that closes after admission is different: nothing reports it to the mux, and the stream watchdog ends the slots that lost records (see Peer loss). The other signal that a stream heartbeat carries is per-slot saturation upstream of the consumer, such as a backlog on the producer's egress. For this reason heartbeats are not in the reserved control class. A consumer that has fallen behind is not this signal: it leaves its sender without credit, and the watchdog exempts a sender that holds none.

The wire reserves SlotHeartbeat (record type 4) for a cheaper heartbeat. In that design, the batcher emits it only for idle slots, on one peer-level tick. Ingress decodes and applies it, but no sender emits it yet. The Batched streaming design chapter records that design.

Flush policy

Senders stage records, and the peer batcher decides when to write. The policies differ only in who decides.

AutoFlush is a struct, not more enum variants, because its conditions compose. A batcher can hold several. on_admission is named for its mechanism. A flush parks until the transport admits it, so "at the end of every wake" means "as soon as the peer took the last batch". The default cannot make latency worse. The batcher never waits for work that has not arrived. It only takes everything that is already queued.

Ending the window early

A linger window runs from the oldest record in the batch. A record that is alone on its lane therefore waits the whole window for records that will not come. Dynamo's response plane runs with a 2 ms window. With one request in flight, the time from the worker's first chunk to the frontend's first token was 3.13 ms longer ([3.12, 3.14] ms, over 278 requests) than over Dynamo's per-request TCP path: the window, plus about 1 ms of timer rounding. That was Qwen3.8 on SGLang, with the frontend and the worker on one VR200 node. The same 3 ms showed at every single-request point measured, on TRT-LLM, vLLM and SGLang. Two optional conditions end the wait when more waiting cannot help:

• on_complete. Every live slot on the lane already has a data record in the batch. More waiting could only add a second record from one of them, and for a token stream that is a forward pass away. The batch goes at the later of its oldest record and one window after the last write that carried data. The second bound keeps a stream that sends faster than the window at one write per window. A write without data does not count: an empty flush (such as flush_batch() with nothing staged), an OpenSlot, or a batch of credit replies. Counting every write was rejected. flush_batch() kicks every batcher on the node, so one empty call held the next lone record on every idle lane for up to a window, and the OpenSlot held a stream's first record until a window after its open. On a busy lane, some slots are always idle (in prefill, or between turns), so the batch is never complete and the window applies as before. Credit replies and closes do not make a batch complete.
• at_bytes. The batch has reached about one transport segment. A longer wait then saves at most one partly filled packet per write, and adds latency. This is not max_batch_bytes: that cap is also the eager limit, and a record larger than it goes through rendezvous. at_bytes only ends the wait.

The rejected alternative wrote a batch at once whenever the lane had been quiet for a window. A forward pass reaches the batcher over a fraction of a millisecond, so that rule writes the first records of a pass alone and the rest a window later: two writes per pass where the window made one. on_complete writes early only when the batch holds a record from every live slot, so it never writes the start of a pass alone. A pass can still be split at a window edge, as under the plain window: a complete batch that waits one window from the last write that carried data can take the first records of the next pass.

In the same deployment with a 2 ms window, worker batches averaged 1.4 to 3.1 records and 120 to 410 bytes, with a 99th percentile of 7 to 16 records. That was at 96 concurrent requests on TRT-LLM (about 38 live slots per worker) and at 1,882 on SGLang with 4 workers (135 to 720 live slots per worker). Batch sizes in bytes were measured only as averages. At 85 to 200 bytes per record, a 99th-percentile batch is about 1.4 to 2 KB, so an at_bytes of one 1,448-byte segment would bind at most on that tail, and one of 8,948 bytes not at all.

Two conditions write a batch under every policy:

• A batch at a clamp goes. The byte cap, the record cap and the eager budget each cut a batch where they bind. A full batch gains nothing by waiting.
• Records that carry liveness go. OpenSlot has its own flush on the awaited path. A CloseSlot or a terminal makes its batch urgent. A CreditUpdate goes within reply_linger.

A slot with no credit also cuts a batch: it adds nothing, and its records wait in the withheld queue.

The flush API

#![allow(unused)]
fn main() {
for (sender, token) in outputs {
    sender.send(token).await?;   // stage
}
velo.flush_batch();              // one write per peer
}

flush_batch() is synchronous and does not block. It kicks each batcher and returns. It is not a backpressure point: per-slot credit and transport admission slow a producer, the same as when nobody calls it. It takes no argument, because a producer holds StreamSenders and cannot know which batcher each one feeds. It is valid under every policy and never returns an error. Under Manual it is the write. Under Auto it forces a write before the batcher's own conditions. With no mux installed it does nothing.

A burst between two calls is a hint, not a frame boundary. A clamp or a liveness record can cut a wire batch inside the burst. A caller must not assume that the records it bracketed arrive in one _stream_batch.

The kick survives a clamp. A wake that stages 100 records can hit a clamp at record 60 and flush inline. The kick stays set, and the batcher consumes it at the end of the wake, after it writes the other 40.

Why a call and not a guard

An earlier design used an RAII gate: start_batch() and end_batch() around a StreamBatch guard, reference-counted for nesting, flushing on Drop. The call replaces it:

• No open state. A gate can be held across an .await, which made a deadlock possible and required two rules to prevent it. A flush holds nothing and nests nothing.
• No guard type. One method replaces a guard, wrapper functions and a reference count.
• It fits the loop. A forward pass is imperative: stage the sends, flush, start the next pass.

The deadlock that the gate rules existed for cannot occur. A producer that sends C+1 frames to one slot has frames 1 to C staged and frame C+1 in the withheld queue. The flush that returns credit is one the producer makes itself, at the end of the pass.

The failure mode of Manual

Under Manual, the application's own records that are staged after the last flush_batch wait for the next call. No timer moves them. This is what makes "one write per pass, carrying that pass" a property of the code and not of the scheduler. The usual cause is a producer that stopped calling flush_batch. A slot starved of credit is a second case: a grant that arrives after the flush releases withheld records into the next pass's batch. A terminal and an inlet close move on their own, so the end of the stream bounds this case.

The cost is latency, not memory, because the same clamps bound staged records. velo_streaming_mux_staged_records shows a plateau. A deployment that wants a safety net can configure Auto { on_admission: false, max_linger: Some(w) }, which is the same batching with a window.

When an explicit flush helps

A forward pass that sends to X streams with no .await between the sends puts them all in the shared egress queue. The default policy then sees all of them. A producer that awaits between sends (a tokenizer, a sampling callback, anything that yields) delivers each send to the batcher alone, and the ratio falls toward 1.0. Only an explicit flush groups sends that the runtime scheduled apart.

The second reason is determinism. How many records share a batch under Auto depends on how the runtime scheduled the batcher against the producer. The batched_streaming example (three anchor hosts, two engines, loopback TCP, 20-core arm64 machine, credit window 256) shows the difference in tokens per wire write:

At serving depth, Manual is both higher and repeatable. A batcher that writes at every wake sometimes wakes mid-pass and writes half a pass. Auto packs harder only when the producer outruns the batcher with no gap between passes. That surplus is throughput bought with per-token latency, which is the wrong trade for a decode engine. Use Auto to let the batcher do its best with what it finds. Use Manual to know what it will do.

Negotiation and compatibility

The attach selects the mux. No wire magic exists. AnchorAttachRequest and MpscAnchorAttachRequest carry supported_transport_keys (with #[serde(default)], so an older sender deserializes as advertising nothing). The attach handler intersects the sender's keys with its installed transports. It picks messenger-mux-v2 only when both sides name it. Otherwise it selects the per-stream default if the sender offers that key, or sends an attach error. An empty offer from an older sender retains the legacy default selection.

The sender reads the answer:

• A key other than messenger-mux-v2 is the per-stream path, and the credit fields do not apply. Every older receiver answers this way.
• messenger-mux-v2 with a window opens a slot that already holds that window.
• messenger-mux-v2 with no window (initial_credit zero) is refused. No shipped receiver answers this, and a node with a mux cannot advertise a zero window. A fallback to another transport reaches nothing that listens, and the stream hangs until the watchdog fires.

By default, a node with the mux enabled registers both messenger-mux-v2 and its configured per-stream transport, so it still serves older peers. A node built with .mux_only() installs no per-stream listener and refuses peers that need one. resolve_transport fails on an unknown key in a non-empty registry, so a receiver that answers the mux on its own breaks every older sender. SPSC and MPSC anchors both negotiate the mux.

StreamSender::negotiated_transport() returns the key that the attach settled on. It returns None for a same-worker attach, which uses no transport. Compare it with the public constant MESSENGER_MUX_KEY.

For a node with a per-stream listener, MuxConfig::enabled = false, or VELO_MESSENGER_MUX_DISABLE=1 at startup, is the rollback. The node stops advertising messenger-mux-v2, and the next attach negotiates the per-stream path with no code or wire change. For mux-only nodes, either setting makes startup fail. Remove .mux_only() to restore the listener before using this rollback. See Zero-RTT stream setup for the order when tickets are in use.

Observability

The mux series all start with velo_streaming_mux_. The most important ones:

• velo_streaming_mux_records_per_batch{direction} and velo_streaming_mux_batches_total{direction} show how much batching occurs.
• velo_streaming_mux_live_slots returns to zero at teardown.
• velo_streaming_mux_reader_stall_total and velo_streaming_mux_credit_lost_total must stay at zero.
• velo_streaming_slot_credit_exhausted_total counts credit starvation on the sender.
• velo_messenger_ordered_lane_wait_seconds{handler="_stream_batch"} shows the ingress lane wait, one sample per batch. Lanes other than 0 report under their own handler names, _stream_batch.1 to _stream_batch.15.

The Metrics reference lists every series. Stream saturation explains how to read them under load.

Peer loss

A batcher with live producer slots checks its selected message transport every five seconds. A failed check closes the producer slots and triggers their cancellation and stop tokens, including idle producers. The check is per peer, not per token or stream. It does not change the selected transport or permit a fallback.

Transports

A transport moves messenger frames between instances. You add transports when you build the instance. More than one transport can be active at the same time.

let node = Velo::builder()
    .add_transport(Arc::new(TcpTransportBuilder::new().build()?))
    .build()
    .await?;

Available transports

The http feature exists, but the HTTP messenger transport is not built at this time.

Peer routing

Velo registers each peer with every transport that the peer and the local instance both support. For each peer, Velo selects a primary transport: the compatible transport with the highest priority. Messages to that peer go on the primary transport.

Velo::unregister_peer undoes this. It removes the peer's routes, and then each transport drops the peer's address and closes its connections. See Forget a peer. Dropping a connection's map entry does not close it: the writer task holds a sender of its own channel, so it would wait for frames forever. So TCP, UDS, QUIC, and gRPC give each connection its own cancel token, a child of the transport's token. UCX closes the endpoint on its progress thread, at the next point in the loop where closing one is safe. ZMQ closes the DEALER socket on its sender thread. NATS keeps no connection per peer.

A WorkerAddress is a MessagePack map from TransportKey to endpoint bytes. Each transport adds its own entry, and a peer reads the entries for the transports that it has.

The transport contract

Every transport implements the Transport trait from velo-ext. The runtime relies on these rules:

• Sends do not wait for the wire. send_message takes the frame and reports when it reached the send channel for the target. Failures after that point go to a TransportErrorHandler callback.
• Order holds per lane. lanes(target) tells how many ordered channels the transport keeps to a peer. The default is 1. send_message_on_lane keeps order within one lane only, and send_message sends on lane 0. A lane never fails over to another lane's connection, because that would reorder it. The messenger sends its own traffic on lane 0, because ordered handlers need the messages of one peer on one lane. QUIC and TCP implement lanes. The other transports keep one lane.
• Inbound frames go to four streams: message, response, event, and shutdown. The TransportAdapter routes each frame to its stream.
• Admission owns the in-flight count. Each inbound MessageType::Message goes through TransportAdapter::admit_message. See Shutdown and drain for why a transport must not check is_draining() itself.
• Unregister forgets a peer. unregister(instance_id) drops what the transport keeps for the peer and closes its connections. The runtime calls it on every transport, also on one that never registered the peer, so it must be cheap and idempotent. The default does nothing, which is right only for a transport with no per-peer state.
• Metrics use one handle. The runtime gives each transport an observability handle through set_observability. In-tree and out-of-tree transports write the same velo_transport_* series.

Coalescing writer

TCP, UDS, and QUIC use a coalescing writer. Small frames that wait in the send channel go out together in one write.

A frame above 64 KB (COALESCE_THRESHOLD) goes out directly, as two writes. The first write is the preamble and the header, staged in a 256 B stack buffer. The second write is the payload, with no copy. Three writes cost two extra syscalls on a TCP_NODELAY socket, and they put small segments on the wire before each large payload. The writer does not use write_vectored, because TCP can return a short write for it past about 128 KB.

These costs are for TCP and UDS. On QUIC, quinn copies each write into its own send buffer, so the payload write is not free of copies there, and the three writes cost no extra syscalls.

The reader reserves the full frame length as soon as it has parsed the preamble. Without this, the read buffer grows by doubling from 8 KB, and each step copies all the bytes received so far. On loopback, the two changes together cut the 8 MB round trip from 6.46 ms to 4.70 ms.

The writer publishes velo_transport_frames_written_total, velo_transport_write_duration_seconds, and velo_transport_egress_queue_wait_seconds. gRPC, NATS, ZMQ, and UCX have no such writer and do not publish these series. See Observability.

Socket buffers are set before data flows

TCP sets SO_RCVBUF and SO_SNDBUF on the listening socket, so each accepted socket inherits the sizes at handshake time. The dial side sets them before its first write. The size is 2 MiB by default.

An explicit size turns off the kernel's autotuning, and Linux clamps it to net.core.rmem_max and wmem_max. With the common value of 212,992, 2 MiB becomes a locked buffer that reports 425,984 bytes (the kernel doubles the value for its own use). This caps the TCP window at about 208 KiB. TcpTransportBuilder::socket_buffers(None) sets no size, so the kernel autotunes the buffers up to net.ipv4.tcp_rmem and tcp_wmem. The choice is a trade-off.

The table below was measured on 2026-09-30 with the throughput example in its two-host mode, over one connection. The nodes were of the same type as in the two-node table: Grace aarch64, 200G Ethernet at MTU 1500, rmem_max at 212,992. Each process had a whole node and was not pinned. Each cell sent 20,000 messages. Three reps, with the two settings interleaved.

The NUMA node that the processes run on also moves these numbers. The two-node table gives 3,920 MiB/s for one TCP connection, 64 KiB pipelined. In a separate run with both processes pinned to the NUMA node of the NIC, that case moved 3,220–3,910 MiB/s with the fixed size and 4,440–4,450 MiB/s with autotuning. Pinned to the other NUMA node, it moved 2,370–2,400 MiB/s with the fixed size.

On loopback, autotuning lost 10% for 64 KiB pipelined, gained 10 to 30% for 256 KiB pipelined, and doubled the p99 for 256 KiB with 64 in flight. Autotuning suits one-way bulk traffic across nodes, and 256 KiB messages on loopback. The default suits request and reply. The option applies to the TCP messenger transport only: the per-stream TCP transport keeps a fixed 1 MiB. Every example that builds its transport with new_transport reads VELO_TCP_SOCKET_BUFFERS: auto for autotuning, or a size in bytes.

The old code set the sizes on the accepted socket, one task spawn after accept. By then the peer was already sending. On Linux, SO_RCVBUF at that point turns off receive autotuning and clamps the buffer to net.core.rmem_max. The advertised window then collapses for the life of the connection. Throughput fell about 100 times, to 22.9 MB/s, and the fault was racy, so it showed up in some runs only. The tx_budget example measures this path and guards against the fault.

TCP lanes

With TcpTransportBuilder::lanes(n), the dialer keeps up to n connections to each peer, one for each lane that it sends on, dialed on the first send on that lane. The default is 1. send_message uses lane 0, so ordinary traffic keeps one ordered channel for each peer. Each lane has its own send channel and admission gate. If the connection of a lane closes, the next send on that lane dials a new one. A lane never uses the connection of another lane, because that can reorder its messages.

• One connection is limited by its receiver. On the receiving side, one reader task does the whole receive copy for a connection, and by default the fixed socket buffers cap the TCP window (see Socket buffers). Lanes spread that work over more tasks and cores.
• The listener needs no change. The kernel gives each accepted connection its own socket, and the listener reads each connection on its own task. A peer can open as many lanes as it needs. Only the count on the dialing side matters.
• The health check counts any lane. A caller can send on any lane, so a peer can have live connections with none on lane 0. check_health reports such a peer as healthy.

A prototype with one connection per lane sent 64 KiB messages, pipelined, between two nodes. It used the throughput example on nodes of the same type as the two-node table, and sent each message on the next lane in turn. The prototype was a patched build, not the code in the tree. The throughput example in the tree sends through the messenger, which uses lane 0 only, so it cannot repeat this measurement. Each cell has two reps. The NUMA node that the two processes run on changes the result by 1.3 to 1.6 times:

The 3,920 MiB/s for one TCP connection in the two-node table is at the top of the range for processes on the NUMA node of the NIC.

QUIC

The QUIC transport uses quinn. It has the same shape as TCP: one connection for each peer, lane and direction, dialed on the first send on that lane. With the default of one lane, that is one connection for each peer and direction.

graph LR
    subgraph Dialer
        W[Coalescing writer] --> S["Bidirectional stream<br>(send half)"]
        R[Dialed reader] --> A1[Shutdown stream]
    end
    subgraph Listener
        E["Server endpoints<br>(one port each)"] --> F[Stream reader]
        F --> AD[admit_message and route_frame]
        F -. "ShuttingDown echo" .-> R
    end
    S --> E

• One stream for each connection. The dialer opens one bidirectional stream and writes velo frames on it with the TCP frame codec. One stream keeps the order of the messages on one connection, that is on one lane. Ordered handlers and batched streaming rely on that order. Several streams would remove head-of-line blocking after a packet loss, but they would also reorder the messages of one peer without the caller choosing it. Lanes (below) reorder only across lanes, and the caller picks the lane for each ordered flow.
• Shutdown waits for acknowledgement. When a writer stops, even at teardown, it finishes its stream and waits up to 1 second for the peer to acknowledge what it wrote, and then closes its connection. A writer that is blocked because the peer stopped reading is closed by force at 2 seconds instead. The dial sockets close when every writer has finished, or at the latest after 2 seconds. A QUIC close discards unacknowledged data, which TCP would still deliver after a close. Transport::closed() waits for this, and graceful_shutdown waits for closed(), so a process can exit when graceful_shutdown returns.
• The reverse direction carries only drain echoes. The listener writes a ShuttingDown frame back on the same stream when it refuses a request during drain. The dialer reads it with the same code as TCP.
• The certificate is pinned. Each transport makes a self-signed certificate and puts its SHA-256 fingerprint in its WorkerAddress entry. A dialer accepts only that certificate and checks the TLS 1.3 handshake signature. A different listener on a reused port fails the handshake.
• Lanes are separate connections. With lanes(n), the dialer keeps up to n connections to each peer, one for each lane that it sends on, each from its own UDP socket. One QUIC connection does its packet and crypto work on one task, so it is bound to about one core. Lanes spread that work over more cores. send_message uses lane 0, so ordinary traffic keeps one ordered channel for each peer.
• Each server socket has its own port. server_endpoints(n) binds n UDP sockets, each on its own port, and the WorkerAddress entry lists the ports. A dialer sends lane k to socket (offset + k) % n, where offset comes from the dialer's own certificate. So the lanes of one dialer land on different sockets, and the peers are spread over the sockets too. Each socket has its own quinn endpoint driver, receive queue and buffer ceiling. The lanes of a dialer are spread evenly, so a socket can carry two lanes: 8 lanes into 4 sockets moved as much as into 8. The default is 4. Each socket costs a port, a quinn endpoint and its buffers. A peer that does not list ports is dialed on its one advertised port for every lane.
• Dial sockets are separate. Dials use their own sockets on ephemeral ports, one for each lane, so each lane has its own endpoint driver for its replies.
• UDP buffers are checked. The transport requests 8 MiB receive and 4 MiB send buffers on each socket, reads back what the kernel granted, and logs a warning when net.core.rmem_max or net.core.wmem_max clamped the request. A clamped UDP buffer shows up later as dropped datagrams, not as an error.
• Packets are at most 6550 bytes. quinn sends up to 10 packets in one GSO batch, and a larger packet makes the batch exceed the UDP datagram limit. The batch is then lost with no error. max_mtu is lowered to 6550.
• quinn 0.11.12 is the minimum. It pulls quinn-proto 0.11.18. Older quinn-proto can fail an ordered, lossless stream of many small chunks with too many gaps in stream buffer.

For the tuning settings and the measured cost against TCP, see QUIC performance.

Frame transports for streams

Streams without the mux use a FrameTransport. TCP is the default. gRPC is available with the grpc feature. See Streaming.

Add a transport

• For an in-tree transport, follow the procedure in the project CLAUDE.md under "Adding a new in-tree transport".
• For a transport in another crate, see Write an out-of-tree transport.

Discovery

Discovery finds the addresses of peers. A backend implements the PeerDiscovery trait. It resolves an InstanceId or a WorkerId to a PeerInfo, which holds the transport addresses of the peer.

use velo::discovery::FilesystemPeerDiscovery;

let discovery = Arc::new(FilesystemPeerDiscovery::new("/tmp/peers.json")?);
let node = Velo::builder()
    .add_transport(tcp)
    .discovery(discovery)
    .build()
    .await?;

// Find a peer and register it.
node.discover_and_register_peer(peer_instance_id).await?;

The discovery traits and custom implementations are available without services. Built-in backends require it; nats-discovery and etcd enable it automatically.

ServiceDiscovery is a second trait. It maps a service name to the instances that provide it.

A registration returns a guard. When the guard drops, the backend removes the registration.

A hub from the velo-hub crate is a discovery service for a whole application, with registration, liveness, and hosted extensions. A program uses it as its peer discovery through HubClient::velo_builder(), and as its service discovery through HubClient::services(). See The hub.

Without a discovery backend, register each peer by hand:

node_a.register_peer(node_b.peer_info())?;

Forget a peer

Velo::unregister_peer(instance_id) undoes register_peer. It drops the peer from every transport, closes the connections to it, and forgets its handler list. A send to the peer then fails as a send to a peer never registered does. A later register_peer, with the same id or a new one, starts fresh. Other peers are not affected. The call is synchronous, cheap, and idempotent.

node_a.unregister_peer(node_b.instance_id());

A restarted peer has a new instance id. So a long-lived node whose peers restart, such as a worker that serves many frontends, keeps the state of every peer it ever saw unless it calls this when a peer leaves.

Unregister only an idle peer. A send, stream, or handshake in flight to it may fail, and one that finishes after the call can register the peer again. A _hello handshake from the peer registers it again too. The messenger mux keeps small per-peer receive tables, and this call does not drop them.

The etcd backend uses leases. If a test process stops and does not revoke its lease, its keys stay until the lease expires.

The hub

The velo-hub crate builds a hub: one service that is the peer discovery for a velo application and also hosts extensions. A program builds the hub in main, adds extensions or bundles of extensions, picks a store, and serves it. velo does not depend on velo-hub. An application that does not use a hub picks any velo discovery backend, as before.

The crate has three cargo features. server builds the hub. client builds the hub client, velo_hub::client, without the server. test-support adds the store conformance suite and FaultStore, a store that fails on request. velo_hub::protocol (paths, request and response types, ids) builds under either feature, and with no feature. It is the velo-hub-ext crate, re-exported. velo-hub-ext depends on velo-ext and not on velo, so a program that only speaks the hub's wire protocol can depend on it and skip the velo runtime.

use velo_hub::{Hub, HubConfig};

let config: HubConfig = serde_json::from_str(&std::fs::read_to_string("hub.json")?)?;
let hub = Hub::builder(config)
    .velo(velo::Velo::builder().add_transport(tcp))
    .bundle(MyBundle::new())
    .serve()
    .await?;
hub.run_until_signal().await?;

The library reads no files and parses no flags. The program builds the HubConfig. examples/examples/hub.rs is a complete hub program: two bundles, one requiring the other, configured by examples/hub.json; --demo serves on free ports, runs a client against it, and stops. Without .store(...), the hub uses an InMemoryStore. Without .velo(...), the hub only does discovery: it does not probe registrants, and liveness is the store's TTL.

Listeners

A hub serves two HTTP listeners:

Extension routes are under /v1/ext/{name}/. An extension's public_routes go on the public listener and its control_routes on the control listener. The hub checks that each extension name is one URL segment of [a-z0-9_-], so an extension cannot add a route to a core path.

A successful registration returns a credential and an epoch. The credential goes in the x-velo-hub-credential header of every mutation that the registration owns. Each registration gets a new credential and a new epoch, and the previous credential stops working. Every error that a core route answers is an ErrorBody with a stable code, also for a body or a path that the hub cannot parse, and for a route that does not exist. A client acts on the code: busy means retry, and register_again means register again without a credential. Extension routes get the same behavior when they use the HubJson, HubPath, and Authorized extractors and return HubError.

Store

The hub keeps registrations in a HubStore. The hub's routes, its registration transaction, and its own velo instance all use the same store. The hub's velo instance uses the store as its PeerDiscovery. InMemoryStore is the store that ships. It expires an entry that nobody touched for the TTL (30 s by default).

The store contract:

• Each write of an instance gets a new incarnation, greater than every earlier one of that instance. The hub and the extensions compare incarnations to tell a stale removal from a current one.
• Writes, removals, and touches are conditional on the current incarnation. Work that runs later carries the incarnation that it saw, so it cannot change a later registration of the same instance.
• The removal hook and its snapshot share one point in time. Each entry present at that point is in the snapshot, and the hook hears each later removal once.
• The store reports its continuity. When the store is not Live, the hub refuses every credential with 503 and refuses new registrations. Peer lookups continue from the store's local view. A Fenced store (another hub holds it) stops the hub.
• Every removal that a call causes or races reaches the removal hook before the call returns. The hub reads the hook's record right after a write, a read-back, or a remove.
• A write can answer Indeterminate: a remote write can apply even when its answer is lost. The hub then asks the store which write produced the current record, by the write's op id.

The record holds the peer, the epoch, the extension payloads, a pinned flag for the hub's own entry, and SHA-256 digests of the credentials. The store never holds a credential, so a copy of the store does not give authority.

Each backend runs the conformance suite in velo_hub::store::conformance (feature test-support). FaultStore in velo_hub::store::testing is an in-memory store that fails, stops at a gate, or answers Indeterminate on request. It passes the same suite when no fault is set.

The etcd store

EtcdStore (feature etcd) keeps the registrations in etcd, under a prefix (/velo-hub/default by default), so they survive a hub restart.

• An entry's incarnation is the etcd revision of its key. etcd revisions only grow, so incarnations do too.
• Every write is one etcd transaction, conditional on the incarnation. When a write's answer is lost, the hub's read-back first writes a tombstone for the write's op id, and every write requires that tombstone to be absent. So a read-back that finds nothing is final: the write will not apply later.
• The store watches its prefix and keeps a local copy, which serves the sync reads and the peer lookups. A call waits until the watch has applied its own transaction and run the removal hook for it, so the hook rule above holds.
• One hub per prefix. The store takes a lock key on a short lease (10 s) when it connects, and every transaction checks that the lock is still its own. If the lock is lost, for example after a partition, the store is Fenced as soon as etcd tells it so, and the hub stops. A hub that cannot reach etcd at all learns this only when it reaches etcd again. It notices that it is cut off when its watch breaks: at once if connections are refused, and after up to about 9 s with the defaults if packets are dropped (the 5 s connection keep-alive plus the request deadline). From then on its store is Degraded: it refuses every credential, but it still answers peer lookups from its local view, which may be stale. Its lock can expire before that, two thirds to one lock TTL after the last renewal, so for a few seconds a second hub can take the prefix while the first one still accepts its credentials. A deployment with a standby hub must allow for that window. A clean stop frees the lock. After a crash, the next hub waits for the lock with wait_for_prefix.
• Liveness does not use a lease per entry. A touch updates the local copy, and the store's reaper removes an entry that nobody touched for the registrant TTL (30 s). Every key of a hub is on one custody lease (5 min), kept alive by the store. So the entries of a hub that crashed stay for the custody TTL, which is how long a restarted hub can recover them. A lease per entry was rejected: it needs one etcd keep-alive stream per touch (about 1,000 a second for 10,000 registrants at the default interval), and a healthy registrant could expire after one late probe.
• A watch that falls behind etcd's compaction loses history. The store is then Lost, reloads the prefix, reports every entry that went away in the gap as a removal, and is Live again. The hub needs nothing more: the store is fenced to it, so no other writer added an entry in the gap.

Measured against a local etcd 3.5 in Docker (release build, one call at a time): a put takes 2.8 ms and a remove 2.5 ms, against 2.2 ms for a plain etcd put; a touch takes 0.17 µs, because it does not reach etcd; a reaper pass that expires 10,000 entries takes 165 to 187 ms.

Recovery after a restart

A hub that starts on a store with registrations in it reserves each of them: nobody can claim the instance without a credential, and peer lookups keep working. It also tells its velo instance about each peer, so its probes reach them.

The registrant gets back in with its credential. The hub's heartbeat carries the hub's id, and the client notices that the id changed (HubClient::recovered()); a route the registrant calls with its credential answers recovered. The client then registers again presenting its credential. The hub checks the credential's digest against the stored record (the credential the registrant presented, or the one that registration minted, whose answer may have been lost), and the registration proceeds as a re-registration of the stored one, with a new credential and a new epoch. Extension state is not replayed: extensions see the owner as new. The registration carries the client's services, so they come back with it. A recovery that fails before its write (an extension refuses it) leaves the instance reserved, so the credential can try again. The same holds for a registration that presented a credential and failed after its write, when the hub could not remove the entry it wrote: the hub answers recovered, not register_again, so the client keeps the one credential that can recover the entry. The credential the registrant presented stays able to recover until the registrant registers once more, because the record keeps it for the case where the answer to that registration was lost.

Authority

Only the hub holds credentials. Authority::authorize(owner, credential) returns a Lease with the incarnation and the epoch of the registration that the credential belongs to. While a registration of that owner is in progress, authorize answers Busy.

A lease alone does not fence a mutation. A registration can start after authorize returns and before the mutation applies. So an extension keeps the epoch that it committed for each owner. It applies an owner's mutation only if the lease's epoch is equal to that epoch, and it does this check under its own lock, in the same critical section as the write.

authorize takes a read lock for one hash lookup and a 16-byte compare in constant time, plus one atomic load for continuity. The Authorized axum extractor does this for a route with an {instance_id} path parameter.

Extensions

An extension implements Extension. It has a wire name, a payload type, a staged type, the extensions it requires, and these hooks:

Each extension sees each registration. The payload is None when the registrant did not opt in. Then commit drops what the owner had before. commit, on_registered, and on_removed all run under one hub-wide registration lock, so nothing changes extension state between prepare and commit.

requires sets the attach order and the start order. At attach, AttachContext::extension::<T>() returns an extension that is already attached. It fails if T is not in requires, so a missing declaration fails at once.

A bundle (Bundle) adds a set of extensions and can require other bundles. A bundle never builds another bundle: the program adds each bundle once, and two bundles that require a third share one instance of it. A bundle reads its own section, bundles.{name}, of HubConfig.

The hub checks the graph before it binds a socket. These are errors: two bundles or two extensions with one name, a required bundle or extension that is not added, a cycle, a name that is not one URL segment, and a config section that no bundle owns.

Registration

handler   spawn the transaction task, and wait for it
task      decode payloads; check registration_requires; admit           400
          take the lock (503 after lock_timeout_ms); begin               401, 409 busy
          prepare each extension, each under step_timeout_ms
            failure: abort; the prior registration is not changed
          write to the store, under step_timeout_ms
            Indeterminate or timeout: resolve the write by its op id
          record the write; velo register_peer; commit each extension;
          commit the credential; on_registered each extension; unlock     200
            any failure after the write: remove the new entry,
            on_removed on each extension, forget the velo peer,
            refuse the credential                                        409 register_again

The transaction is a task of the hub. hyper drops a request handler when its client closes the connection. If the handler ran the transaction, a disconnect could leave the owner busy for ever. Decoding and admit run in the task as well, so a panic in any extension hook answers 500.

Each store call that the hub awaits (under the lock, in a request such as the HTTP heartbeat, or in the heartbeat task), and each on_registered, runs under step_timeout_ms. A store or an extension that never answers cannot hold the lock. A remove that times out counts as indeterminate: the hub reads the store to learn whether it applied.

A failure before the store write leaves the prior registration exactly as it was. A failure after the write removes the registration, including the prior one, and the client registers again. The alternative is to write the prior registration back. kvbm-hub did that, but the restore is a second write that can fail too, and it gives the registration a new incarnation.

If an extension panics, the hub settles the owner. Before the write, it aborts. After the write, it refuses the credential and queues the removal of the entry.

When the hub runs velo, it decodes the peer's address before it takes the lock and answers 400 if the address is not valid. Without velo, the hub stores the address and does not read it.

Removal

The store calls the hub's removal hook when an entry leaves: an unregistration, a TTL expiry, or a failed heartbeat. The hook revokes the owner's authority at once and puts the removal on a queue. It never waits for the registration lock. A lifecycle task takes the lock and delivers the removal: on_removed on each extension in reverse start order, then it forgets the velo peer, then the owner can register again. Until then, the owner's requests answer busy.

The lifecycle task forgets the velo peer only if no later incarnation replaced it. So a late delivery does not disconnect a replacement.

DELETE /v1/instances/{id} takes the lock and delivers the removal itself. When it answers, each extension has dropped the owner. The hook also queued a delivery, and that second delivery changes nothing. Like a registration, the unregistration runs as a task of the hub, so a client that disconnects cannot leave the owner busy.

Heartbeat

When the hub runs velo, it sends a heartbeat to each registration every heartbeat_interval_ms (10 s by default). The registrant answers on the velo_hub_heartbeat handler. Each probe times out after one interval. After heartbeat_max_failures (3 by default) failures in a row, the hub removes that incarnation. A success touches that incarnation. An answer with ok: false counts as a failure: a registrant answers that while it holds no registration with this hub, for example after the response to its registration was lost. The hub committed that registration, but nobody holds its credential, so the registrant could never register again while its probes kept the entry alive. The hub does not probe an instance while its registration transaction runs: the registrant has no answer yet, so it cannot tell its new registration from one it never got, and a slow but conforming on_registered would let the probes remove the entry before the answer arrives. Probes run in parallel, so a stopped peer does not delay the others. The hub does not probe its own pinned entry. Without velo, liveness is the store's TTL.

The heartbeat task holds a WeakVelo. Extension handlers and tasks that use the hub's Velo must also hold a WeakVelo, and upgrade it on each call. A handler that holds a Velo is in that instance's own handler registry, so the instance holds itself and is never dropped.

Client

A program that uses a hub builds a HubClient (feature client), builds its Velo from the client, and registers:

use velo_hub::client::HubClient;

let hub = HubClient::builder().host("hub.example").build()?;
let velo = hub.velo_builder().add_transport(tcp).build().await?;
hub.register(&velo, Default::default()).await?;
// velo now finds every other registered peer through the hub.
hub.unregister().await?;

velo_builder() sets the hub as the velo discovery. velo resolves a worker target through it, and Velo::discover_and_register_peer resolves an instance id through it.

register(&velo, extensions) installs the heartbeat handler on velo first, so the hub's probes get an answer as soon as the hub answers the registration. After the registration, register registers the hub's velo peer with velo (velo does not ask discovery for a call to an instance id). It does this again for a new hub after a restart, and if this step fails, it logs a warning and keeps the registration. register_peer(peer, extensions) registers a PeerInfo without a handler, for a hub without velo. A client holds one registration. A second register of the same peer presents the credential, and the hub replaces the registration: a new credential, a new epoch, and the old credential stops working. If the hub answers unauthorized or register_again, the client drops the registration it holds, and the next register starts again without a credential.

The client's heartbeat handler answers ok: true only while the client holds a registration or is getting one. So when the response to a registration is lost, the hub removes the registration that nobody can use after heartbeat_max_failures probes, and the client can register again. A response is lost when the network drops it, and also when the caller drops the register future (a timeout around it): the hub runs the registration as its own task and commits it anyway. A guard clears "getting one" on every exit from register, also when the future is dropped, so the handler never answers ok for a registration the client does not hold. A re-registration that gets no answer from the hub fails closed. No answer means anything but the hub's own response or a request that never left: its future is dropped, its request times out or breaks after it was sent, or a proxy in front of the hub answers 502 or 504. Then: the hub may have committed it and rotated the credential, so the client drops the registration it held, and the hub frees whatever it holds after heartbeat_max_failures probes. A register without a credential that the hub refuses with unauthorized means the hub holds a registration of this instance that the client never got. Until a register succeeds, the handler then answers not ok even during a call, so a caller that retries in a loop cannot keep that registration alive. The retry gets in after about heartbeat_interval_ms times heartbeat_max_failures (30 s by default).

An answer that says the hub does not accept the credential (unauthorized, register_again, on any call, also an extension call) drops the registration the client holds, and so does not_found for a heartbeat or an unregister. So is_registered() tells the truth again, and the next register starts without a credential.

The client is a cheap handle; its clones share the one registration. Dropping the last clone sends DELETE in the background on the current runtime, as a best effort: without a runtime, or when the runtime shuts down first, the request is not sent, and the hub's probes or the TTL remove the entry. A program that can await calls unregister. If DELETE fails, the client keeps the registration, so a retry or the drop sends it again.

velo keeps its discovery for as long as it lives, so the discovery that velo_builder() installs is a separate handle, HubDiscovery, that holds no registration. If velo held the client itself, the registration would live as long as velo does, and dropping the client would not unregister. The heartbeat handler and ExtClient hold the registration weakly for the same reason.

client.ext(name) makes JSON calls under /v1/ext/{name}. get and post use the public listener. control_get, control_post, and control_delete use the control listener and send the registration's credential; once every HubClient clone is dropped, they answer NotRegistered. A bundle crate builds its typed client on it. Every error that the hub answers is a ClientError::Hub with the hub's ErrorCode, so a caller acts on the code.

The default HTTP client has a 5 s connect timeout and a 60 s request timeout. A registration on a hub with default settings and up to two extensions of its own takes at most 50 s: 5 s to wait for the lock, and 5 s for each step under it (the built-in service directory counts as an extension). A caller that passes its own reqwest::Client sets its own timeouts.

An extension can also serve velo handlers on the hub's velo instance, and its client calls them with velo requests, not over HTTP. velo puts no deadline on a velo request, and the hub sends each reply through its own peer table. When the hub removes a registration, it forgets the velo peer. After that, a velo request from that instance gets no reply.

So a client that makes velo calls to the hub sets .timeout on each call. It reads RequestTimeout as "the hub can have forgotten this instance", and it calls register again. That call presents the credential that the client holds. If the hub answers register_again or unauthorized, the client drops the registration, and the next register starts without a credential. is_registered() does not show an eviction, because only an answer from the hub changes it. recovered() shows only a hub restart.

Services

A registered instance can serve named services. Another program asks the hub which instances serve a name, or watches the name. A service name is 1 to 128 of A-Z a-z 0-9 . _ -, and not . or .., so it is always one path segment and a URL resolver cannot turn it into another route.

The services are a built-in extension, named services, first in the start order. So a service entry follows the same transaction and removal rules as an extension's state: it goes when its registration goes (unregister, expiry, or heartbeat eviction), and there is no second cleanup path that could leave an orphan. A registration may carry the names the instance serves, as the services payload: the hub then sets the instance's services to exactly those names. Without the payload, a re-registration with the credential keeps the instance's services, because the peer still serves them, and a registration without a credential starts with none. A PUT or DELETE applies only if its lease's epoch is the one the directory committed, under the directory's own lock. The hub does not list the built-in in GET /v1/hub, and no other extension can take its name.

A watch takes its snapshot and subscribes in one step, under the directory's lock, so a watcher never misses a change or sees one twice. A watcher that falls more than 128 changes behind gets disconnected, and its stream ends: its set is stale, so it watches again. At shutdown the directory's shutdown hook ends every watch stream, before the hub drains its listeners, so an open watch does not hold the drain. The hub sends a keep-alive comment every 10 s. A watch of a name that nobody serves costs the hub a small entry and a 128-slot event ring until the watch ends.

The client implements velo's ServiceDiscovery with client.services(). client.register_service(name) returns a guard; the client holds one guard per name and refuses a second. The client keeps the names it serves, and every registration carries them, so after any registration (fresh, re-registered, or on a hub that restarted) the hub's set is the client's, with no request per name. Between registrations, guards change only the client's set, and one writer, sync_services, sends the changes one request at a time while no registration runs. So two requests from one client never race. A request the hub refused is answered; one with no answer from the hub (a dropped future, a timeout, a proxy's 504) leaves its name unsettled, and the next pass sends it again as the client's set says. So a guard never removes another guard's entry, and the hub does not keep an entry nobody serves. register answers Ok once the registration is held. A failure to send a change made during the call is left for sync_services, which the caller can also run to retry. Dropping a guard stops serving the name, in the background.

Services are kept in the hub's memory, like extension state. A hub that restarts with a persistent store restores peer discovery only; the client's next registration carries its services back.

Startup and shutdown

HubBuilder::serve does these steps in this order:

1. Check the bundle and extension graph and the config. This step does no I/O.
2. Bind the two listeners.
3. Build the hub's velo instance, with the store as its discovery.
4. Write the hub's own pinned entry.
5. Attach each extension, in dependency order.
6. Start each extension, in dependency order. Then build both routers, which runs each extension's route code.
7. Install the store's removal hook, and reserve each entry that the store already holds.
8. Start the reaper, the heartbeat, the lifecycle task, and the HTTP servers, then accept registrations.

If a step fails or panics, the hub undoes each earlier step in reverse order. Step 1 also refuses a config with a zero interval or timeout. The removal hook cannot be removed, so the hub installs it last. A store given to a hub that failed to start can go to another hub.

Hub::shutdown stops new registrations, waits for the registration that holds the lock, stops extension background work, delivers the queued removals, shuts each extension down in reverse order, and stops the heartbeat and the reaper. Then it releases the lock, so a request that waits for it answers 503 at once. It closes both listeners and waits up to shutdown_timeout_ms for the requests in flight. Then each request still in flight answers 503 and its handler is dropped, and the hub waits up to shutdown_timeout_ms again for the connections to close. So the listener drain can take two times shutdown_timeout_ms. The wait for the lifecycle task, each extension's shutdown, each hub task, and the velo drain also get shutdown_timeout_ms each. The wait for the lock gets at least as long as a registration can hold it: a prepare and an on_registered for each extension, and a write, a read-back, and a remove, each step_timeout_ms. So only a sync hook that blocks its thread, against the extension rules, can make the stop go on without the lock. Then it delivers no queued removal, because without the lock on_removed could run while that hook still runs. Last, it removes the pinned entry, closes the store (an etcd store frees its prefix, so the next hub can take it over), and shuts the hub's velo instance down. When the drain finishes within its budgets, the listener ports are free when it returns. If a connection outlives both budgets, the hub aborts its server task, and the port is freed when that abort takes effect.

The stop runs as its own task. A caller that stops waiting does not stop it, and a second call waits for the same stop. Dropping the last Hub clone runs the same stop in the background. serve also runs as its own task: if its caller stops waiting, startup finishes and the new hub stops in order.

Cost of the request paths

Two reads are on the request path: Authority::authorize, for each owner-scoped mutation, and the peer lookup. Both read a sharded copy of the state, and the writers update that copy while they hold the one lock that orders all writes. A design with the readers on the writers' RwLock was measured first and rejected: every read writes the lock's cache line, so more threads gave fewer calls in total.

Setup: release build, 10,000 owners or peers, 2,000,000 calls spread over the owners, a 20-core host. The tests are authorize_cost and lookup_cost, run with --ignored --release --test-threads=1. The "one RwLock" column is the same two tests run on the code before the sharded copy existed.

get and list copy each record with its payloads. With a 2 KiB payload, get took 248 ns, and list took 14 to 19 ms for 10,000 entries. The heartbeat uses peers(), which does not copy payloads: 0.3 ms for 10,000 entries.

A write holds the store's write lock and a shard of the copy, so its cost must not grow with the number of entries. With 10,000 entries (write_cost), a put takes 1.0 µs and a remove 0.6 µs, and a reaper pass that expires all 10,000 takes 7.2 ms. An index update that scanned every entry was measured first and rejected: 21 µs for each put and each remove, and 109 ms for that reaper pass, all of it with the write lock held.

Versioning

velo-hub and velo-hub-ext take the workspace version and depend on each other and on velo with a caret on that version. See Versioning.

Work queues

Work queues require the default services feature. The queue-messenger and nats-queue backend features enable it automatically.

A work queue is a named queue of typed work items. You create or connect to a queue by name, and then you get a typed sender and a typed receiver for it.

use velo::queue::{backends::memory::InMemoryBackend, receiver, sender};

let backend = InMemoryBackend::new(1024);
let tx = sender::<Job>(&backend, "my-jobs").await?;
let rx = receiver::<Job>(&backend, "my-jobs").await?;

tx.enqueue(&Job { id: 1 }).await?;
let job = rx.next().await?.unwrap();

Shutdown and drain

Velo::graceful_shutdown(policy) stops an instance in four phases. With ShutdownPolicy::WaitForever, it does not lose a request that it already accepted. With ShutdownPolicy::Timeout(d), teardown drops any accepted request that is still queued when d expires.

sequenceDiagram
    participant App
    participant Velo
    participant Adapter as TransportAdapter
    participant Peer
    App->>Velo: graceful_shutdown(policy)
    Velo->>Adapter: 1. Gate: begin_drain()
    Peer->>Adapter: new request
    Adapter-->>Peer: ShuttingDown (request header echoed)
    Velo->>Velo: 2. Drain: wait until in-flight = 0 (or the timeout)
    Velo->>Velo: 3. Teardown: cancel tokens, stop transports
    Velo->>Velo: 4. Close: wait until each transport's close is on the wire

1. Gate. The drain flag goes up. New inbound requests are refused. Responses, acks, events, and the messages of open streams continue to flow, so in-flight work can finish.
2. Drain. Velo waits until no admitted request is in flight. ShutdownPolicy::WaitForever waits with no limit. ShutdownPolicy::Timeout(d) bounds the drain at d. The drain counts requests in flight, not the payloads that the messenger staged for messages over the eager budget: a consumer that has not pulled one yet loses it at teardown (known limitation). Joining the mux's batchers adds up to 0.5 seconds, the time a batcher stalled on a slow peer may take to send what it holds, and the close step adds the close bound of each transport (QUIC: 2.5 seconds; TCP and UDS: their teardown grace, 1 second unless the builder set it, plus 1.5 seconds). Teardown adds the time that the shutdown hooks of the transports take. A hook may block, so d does not bound it. The ZMQ hook takes at most about 6 seconds: up to 5 seconds for a send already under way to a peer that has gone, and up to 1 second for the frames that were queued before the stop. ZMQ reports each frame that it cannot send in that second as failed. Unless a hook blocks, the whole call takes at most d plus these bounds.
3. Teardown. Velo cancels the tokens and stops the transports. Each transport's shutdown hook runs once, on a dedicated thread, and the call waits for all of them. (A hook runs on the caller instead if the thread cannot be created, and on the building task if the build fails or is cancelled while transports start.) If a hook panics, the other hooks still run, and the call then panics: it cannot report the instance, or its RDMA memory, as released. A later shutdown call panics at once, without draining again.
4. Close. Velo waits for Transport::closed() on each transport. TCP and UDS first wait for their connection writers, for at most their teardown grace plus 0.5 seconds. At teardown, each writer finishes the write in progress and writes the frames already in its channel, until the grace has passed, and then fails the rest. The grace is 1 second unless the transport's builder set it (teardown_flush_grace on TcpTransportBuilder and UdsTransportBuilder). It covers only what is already queued when teardown starts (known limitation). A writer that is still dialing its connection keeps dialing until the same deadline, so the frames queued behind the dial are written too. The transport admitted those frames, and a sender can act on that at once: the messenger mux reports a stream sent when the batch that ends it is admitted. TCP and UDS do not wait for what they wrote, because the kernel delivers it after the process exits. Then they wait until their accept loop has closed its listening socket, for at most 1 second, so that no peer can connect after graceful_shutdown returns. The loop stops at its next poll, so the wait runs out only when nothing drives the runtime that runs it. In that case the transport logs a warning, and the socket can stay open until the loop runs. QUIC keeps written data in user space until the peer acknowledges it, so it waits up to 2 seconds for its writers to finish their streams and its connections to close. Then it closes the rest by force and waits up to 0.5 seconds more, so that each frame a writer still held is reported as failed. A frame still queued at teardown is failed (known limitation). A frame that quinn accepted but the peer did not acknowledge is lost, as a frame in the kernel send buffer is lost when a TCP peer stops reading. The writer logs a warning when that can happen. The exception is an application close from the peer: a peer that closes has gone, as a TCP peer that closes has, so that end is not a warning. A process can exit when graceful_shutdown returns.

Velo is Clone. If two clones call graceful_shutdown at the same time, the first runs the sequence and the second waits for it.

Shutdown does not remove handlers. A handler that holds a Velo clone of its own instance keeps that instance alive, so its final drop never runs. A handler that needs its own instance holds a WeakVelo from Velo::downgrade and calls upgrade on each request.

Work that was accepted before the drain keeps flowing through it. The messenger mux sends its records and its credit as active messages, so the gate lets through the handlers that serve accepted work:

• The mux batch handlers (_stream_batch, and _stream_batch.1 to _stream_batch.15, one for each lane), which carry the records and credit of open mux streams.
• _stream_cancel and _stream_stop, the detach, finalize and cancel handlers of SPSC anchors, and the detach and cancel handlers of MPSC anchors.
• _anchor_renew, which extends a reservation that this node already made. It opens no stream.
• The rendezvous handlers that pull a staged payload and end its lease (_rv_acquire, _rv_pull, _rv_detach, _rv_release, _rv_lease_renew). A record or response too large for one message is staged, and the receiver pulls it with these handlers. A draining owner answers the pull chunked, never by RDMA. _rv_metadata and _rv_ref start a new consumer, so the gate refuses them. A handle that an application staged itself can still be pulled with get during the drain.
• The event handlers _event_trigger, _event_trigger_request and _event_subscribe. _event_trigger completes an awaiter of work that this node already accepted. _event_trigger_request completes an event that this node already created, and it acknowledges the requester. The requester sends it fire-and-forget, so a refusal would never reach the requester's wait. _event_subscribe answers at once for an event that is already complete, or records one subscriber for a pending event. None of the three starts new work.

Each handler declares its exemption in the code where it is registered, so the gate cannot disagree with the code. The list above is kept by hand. An attach opens a new stream, so the gate refuses it. A detached SPSC anchor therefore waits out its unattached timeout during a drain, because no new sender can attach. A zero-RTT stream is not an attach: its first record opens the slot that prebind_anchor bound, during the drain as well. prebind_anchor does not check the drain, so a node that keeps making pre-binds while it drains keeps accepting streams.

The drain counts an exempt message while its handler runs. It does not count the stream that the message serves. This has two results:

• The drain finishes at the first moment that no message is in flight. A stream message is in flight only for microseconds, and consecutive batches are at least a credit round trip apart. An open stream, busy or quiet, therefore does not hold graceful_shutdown open.
• Velo then stops and joins mux sends before transport teardown. Ingress slots stay in place until full shutdown detaches their readers, so orderly shutdown does not report a false sender-drop error. The shutdown timeout bounds the drain; joining mux tasks comes after it, takes up to 0.5 seconds more for a batcher stalled on a slow peer, and needs the owning Tokio runtime to keep running.

To let open streams finish, call begin_drain, wait until your streams end, and then call shutdown. The application must wait for its streams: the messenger drain does not count their full lifetime.

Close an instance while Tokio keeps running

Use Velo::shutdown(policy) when an application removes a Velo instance but keeps its Tokio runtime. It first runs graceful_shutdown, then cancels live anchors and senders, stops the builder-owned per-stream listener, and joins the messenger receive loops and streaming tasks. Pending remote event waits fail at teardown, and their subscription tasks stop. Local event completion remains available. Custom frame transports remain the caller's responsibility.

If the sender of a stream is on the same instance, shutdown cancels that sender. The cancellation_token of the sender fires, and later sends fail. The reader of the stream ends when the application drops or finalizes the sender. Shutdown does not end the reader itself. That needs a check on every read.

Call shutdown explicitly to drain and join. Final Velo drop cancels its streams and stops builder-owned streaming services. It also tells each remote producer that its stream ended, while the transports are still up: an attached producer through _stream_cancel, and a zero-RTT producer through its mux slot close. This is best effort. It needs the Tokio runtime to keep running, a stalled peer gets the slot close only if it admits it within 0.5 seconds, and without a retained Messenger the transport can close before the notice is written. A producer that is told nothing waits on its next send. When remote producers must be told, call Velo::shutdown and end their streams first. A retained Arc<Messenger> keeps active messaging available, but does not keep those streams alive. Final Messenger drop starts transport teardown on an owned thread. Explicit shutdown waits for this cleanup, including native transport joins, before it waits for transport close. Neither Drop path waits for work to finish or reports RDMA memory as released. See Ownership changes in 0.19.

graceful_shutdown keeps its existing behavior: it drains and closes the messenger and RDMA services, but does not close the per-stream TCP or gRPC transport. shutdown is the complete instance shutdown operation. Application handlers that exceed a timeout can still be running after it returns.

The public task tracker includes receive loops, ordering lanes, and tasks added by the application. close() allows wait() to finish when all tracked tasks exit; it does not cancel them. Ensure application tasks and idle ordering lanes can exit before waiting. An ordered handler with with_idle_lane_ttl(None) can leave a lane waiting forever while its router remains owned. Internal shutdown joins its own receive loops separately, so an application task cannot extend its timeout.

A handler panic fails its waiting caller in every dispatch mode, provided the program unwinds panics. The error reply remains counted work until the transport accepts it. An ordered lane continues with the next message.

Teardown does not interrupt a blocked TCP or UDS write, or a dial in progress, at once. The writer finishes it and writes the frames already in its channel, until the transport's teardown grace has passed (1 second unless its builder set it). At that deadline it fails the write and the frames that are left. A connection that closes alone, because its peer was unregistered or closed the socket, fails its frames at once. The connection is discarded if a write may be partial. A reported write failure does not prove that the peer received no bytes; applications must not treat it as permission to retry a non-idempotent request.

A refused request fails fast

A transport that has a return path answers a refused request with a MessageType::ShuttingDown frame. The frame echoes the header of the rejected request, so the sender can find the waiting caller and fail it at once. Without the echo, the caller waits for its own timeout. The echo uses the request header, not a response header, so ShuttingDown frames have their own inbound stream.

gRPC has no return path on its client-side read half. There, the transport records the rejection and drops the frame.

Admission owns the in-flight count

The rule is: a message on the inbound queue is counted work. TransportAdapter::admit_message is the only way onto the inbound queue. It takes the in-flight guard first, and then it reads the drain flag. The queued message carries the guard, and the guard is not optional.

This count starts at server admission. It does not include requests still in a client or network queue. To settle a known set of client calls before shutdown, call begin_drain, keep the server alive until those calls finish or reach their deadlines, then call shutdown.

Each caller must bound its response wait, for example with tokio::time::timeout. A peer disconnect does not complete every outstanding response slot. A request admitted before peer failure can therefore wait until its caller's deadline. Dropping its response awaiter releases the slot.

Two faults made this the rule:

• The consumer took the guard after it dequeued the message. A message in the queue was then invisible to the drain. The fast path of graceful_shutdown does not yield, so it could finish inside the gap between enqueue and dequeue. The handler then ran after the instance had declared itself stopped.
• A transport that checked is_draining() before it enqueued had a check-then-act race. A request could pass the check, the drain could start and finish, and then the request was enqueued.

is_draining() is for reports only. Never use it to decide admission.

The admission check is a store-buffer pattern. The producer increments in_flight and then loads the flag. Shutdown stores the flag and then loads in_flight. All four accesses must be SeqCst. With Acquire and Release, both sides can miss the store of the other side.

A consumer that stops with messages still in its queue must drain the queue and drop the messages. flume keeps the buffer alive while any sender clone is alive. Guards left in the buffer hold in_flight above zero, and every later drain waits forever.

These tests pin the contract:

RDMA registrations go first

When the RDMA registration layer is installed, shutdown has four steps:

1. The gate closes. No new request can start. The pull of a payload staged before the drain still passes the gate, but a draining owner answers it chunked, never by RDMA.
2. The registry sweep runs. New registrations are refused, in-flight transfers drain, and each region and arena is unmapped. Anything staged in registered memory first moves to the heap, so an admitted chunked transfer can finish.
3. The messenger gate, drain, teardown, and close run as usual.
4. Each registration that survived step 2 is declared released, but only if the backend reports that nothing is still registered.

The order of steps 1 and 2 matters. An RDMA GET is issued by the NIC of the peer, so it never shows in the in-flight count of this instance. If the transport stopped first, Velo would unmap memory that a peer is still reading.

Step 4 makes RegionGuard::deregistered safe to wait on. After an abnormal teardown, such as a progress thread that panicked, step 4 declares nothing. That memory leaks on purpose, because Velo does not tell a caller to free pages that it cannot prove are released.

Under ShutdownPolicy::Timeout, the sweep and the messenger drain share one deadline. Under WaitForever, the sweep still uses RdmaConfig::shutdown_timeout, so a peer that crashed during a transfer cannot stop shutdown forever. See Rendezvous and RDMA.

Rendezvous and RDMA

Rendezvous moves a large payload by handle. One worker (the owner) stages the bytes and gets a compact DataHandle. The owner passes the handle to consumers by any means: a message field, an event, or a typed response. Each consumer then pulls the bytes from the owner. The consumer drives every transfer. The owner only answers.

The bytes move on one of two paths. The chunked path works everywhere and pulls the payload in 512 KiB messages. The RDMA path lets the consumer's NIC read the owner's memory with one UCX ucp_get_nbx call. The owner decides the path for each transfer, and a caller writes the same code for both.

The RDMA path needs the ucx feature on Linux. See Run rendezvous over RDMA for the build and the runtime setup. See RDMA performance for measured numbers, and RDMA design for why the design has this shape.

The handle API

A handle is a u128: the owner's WorkerId in the upper 64 bits and a local slot id in the lower 64. Serde writes it as {hi, lo} because MessagePack has no clean u128 encoding.

The owner registers seven handlers: _rv_metadata, _rv_acquire, _rv_pull, _rv_ref, _rv_detach, _rv_release, and _rv_lease_renew. The control plane uses JSON over ordinary messenger calls, so any transport carries it.

One transfer, both paths

_rv_acquire takes the read lock and chooses the path in the same round trip.

sequenceDiagram
    participant C as Consumer
    participant O as Owner
    participant NIC as Consumer NIC
    C->>O: _rv_acquire {handle, rdma: offer}
    alt owner answers RDMA
        O-->>C: Rdma {lease_id, descriptor, lease_timeout_ms}
        C->>NIC: ucp_get_nbx(addr, len, packed rkey)
        NIC->>O: one-sided read of owner memory
        loop every lease_timeout / 2 while the GET runs
            C--)O: _rv_lease_renew {handle, lease_id}
        end
        alt GET or decode fails
            C->>O: _rv_detach {lease_id}
            C->>O: _rv_acquire {handle, rdma: none}
            O-->>C: Ready {...}
            C->>O: _rv_pull (chunked)
        end
    else owner answers chunked
        O-->>C: Ready {lease_id, transfer_id, chunk_size, chunk_count}
        loop chunk_count times
            C->>O: _rv_pull {transfer_id, index}
            O-->>C: chunk bytes
        end
    end
    C->>O: _rv_detach or _rv_release {lease_id}

Both answers end in the same detach or release. The owner checks the registration again at every acquire, so no cross-node invalidation protocol exists.

When the owner answers RDMA

The owner answers AcquireResponse::Rdma only if every check below passes. The checks run in this order. Each refusal increments velo_rendezvous_rdma_path_total{path="chunked", reason=...} on the owner.

Check 7 comes after the facts about the configuration and the slot, so draining counts only the acquires that the drain changed. A heap-staged slot on a draining owner still counts as not_pinned. Check 8 catches an external region that was deregistered under the slot.

A lease answered with a descriptor always gets a deadline. A chunked lease gets one only when its consumer asks for it (see What a fetch that gives up leaves behind).

What the consumer offers

The consumer builds its offer from three local facts. It asks the owner nothing.

• This instance has an RDMA registry.
• The kill switch on this instance is off.
• The owner is registered on a transport whose key matches the registry backend.

The GET uses this consumer's own UCX endpoint to the owner. So "UCX is registered for that peer" is the whole condition. UCX does not have to be the primary transport. The expected deployment is a TCP control plane with UCX beside it.

The rule has a cold-start effect. A peer that is not registered yet translates to nothing. The first get to an unknown owner therefore offers nothing and pulls chunks. Later gets take the RDMA path.

Fallback is a routing decision

After the owner sends a descriptor, four failures can occur on the consumer. They map to three reason labels:

For each failure, the consumer detaches the lease, acquires again with no offer, and pulls chunks. This always works, because a pinned slot also serves the chunked path.

The fallback happens exactly once. The second acquire carries no offer, so an owner that answers it with another descriptor is broken. The consumer returns an error in that case. A retry loop turns one bad owner into an unbounded storm of round trips.

A pinned slot is never RDMA-only. Old consumers, consumers without UCX, consumers below the threshold, and consumers whose GET failed all read the same slot by chunks. The owner does not need to know in advance which kind of consumer it talks to.

Staging on the owner

register_data_pinned

This call copies the bytes into a pool buffer that the NIC can read. It returns a DataHandle, not a Result. If the pool cannot give memory, the call stages the bytes in plain memory and records why:

• The registered-bytes budget is spent: budget.
• The pool cannot allocate for another reason: pool_exhausted.
• The kill switch is on: kill_switch.
• The instance has no UCX transport: the data goes to plain memory.

Zero-length data always goes to plain memory, because a GET has nothing to transfer. The call always costs one copy. The zero-copy form is register_data_in_region over memory that the caller registered.

register_data_in_region

This call stages a range of memory that the caller already registered with register_external_memory or register_owned. The range is relative to the registered pointer, not to the page-rounded effective range. A caller therefore cannot name a byte that is inside the registration but outside its own allocation.

The call returns a Result. A fallback to plain memory copies bytes that the caller asked velo not to copy. The kill switch does not affect staging here. It only decides whether _rv_acquire answers with a descriptor.

The slot holds an in-flight guard on the region. RegionGuard::unregister therefore waits for the anchors inside the region before it unmaps. The slot takes the guard first and then reads the region state. The reverse order leaves a gap in which the whole gate, drain, and unmap sequence can run.

Chunked reads of an external slot never form a Rust reference into the region. A peer holding the key can write the region at any time. The read copies with ptr::copy_nonoverlapping into a fresh buffer, under the region's copy gate, 512 KiB at a time. The copy gate orders each read against the deregistration latch. A copy in progress delays the latch. A copy that starts after the latch sees the flag and refuses.

Transparent staging

The messenger stages any payload above 256 KiB through rendezvous and replaces it in the frame with a handle. The receiver resolves the handle before the handler runs, so handler code never sees the difference.

The transparent stager runs inside the synchronous send_message. It uses only arenas that the pool has already mapped. It never maps a new arena, because mapping is an ibv_reg_mr whose cost is linear in size. A send must not stall on that.

The result: a process whose only staging is transparent never maps an arena and never takes the RDMA path. One explicit register_data_pinned call maps the first arena. After that, transparent payloads use it. Growing the pool from a send on a background task was rejected. A caller cannot reason about a message send that causes a 64 MiB pin.

The fetch has a deadline

The receiver fetches the payload with its own calls to the sender (_rv_acquire, then _rv_pull). Each call holds one of the receiver's response slots, and the request keeps its drain guard, until the fetch ends. A fetch with no deadline ends only when the sender answers. So a sender that stops answering took one slot from the receiver for each staged request, after its own caller had given up.

The fetch now ends at the first of two bounds:

• The caller's. A sync or unary call with .timeout sends what is left of it with the handle, in the _rv_budget_ms header, rounded up from nanoseconds to whole milliseconds. The receiver starts the bound when it takes the message off its queue, so it does not end before the caller's own timer. How much later it ends depends on transit, queueing and scheduling, which nothing bounds. When it runs out, the receiver drops the fetch and sends no reply: the caller has its RequestTimeout already, and a reply that raced the caller's timer would replace that with a different error.
• The receiver's. VeloBuilder::staged_fetch_timeout bounds every fetch on this instance. It covers what carries no bound: fire-and-forget sends, calls without a timeout, and senders from before the header. When it runs out, a sync or unary caller gets an error reply that says the fetch did not finish in time. It does not apply to the items of a stream. A stream item over the threshold is a fire-and-forget batch, so a dropped fetch lost the item with no word to the stream. Its consumer got SenderDropped from the stream watchdog about 15 s later, while the producer was still attached.

The bound is a duration and not an instant, because the two nodes do not share a clock. It is its own header because a receiver from before it reads the _rv value as a bare number. Such a receiver ignores the new header.

The staging headers belong to the send that staged

The receiver takes _rv and _rv_budget_ms out of a staged message before its handler runs, so a handler never sees them. A handler that passes its headers on in a send of its own therefore passes neither. A forwarded handle made the next receiver fetch a payload that was not its message's. A forwarded budget made it give up at another call's deadline.

A staged send writes both headers itself, after the tracing propagator, so the ones on the wire are its own. A budget without a handle means nothing to a receiver, so an eager send needs no such care. An encoder that owned the budget key was rejected: it compared every header key of every send with _rv_budget_ms, and it left a forwarded _rv in place.

What a fetch that gives up leaves behind

The receiver tells the sender nothing when it gives up a fetch. A slow sender then grants a lease that nobody pulls or releases, and a payload whose fetch never got a lease stays staged. Two things end both on the sender:

• A time-to-live. A staged payload whose call has a .timeout gets that deadline as its RegisterOptions::ttl. Past it, the slot is freed with every lease on it, whether or not a consumer is still reading. The caller has given up by then.
• A chunked lease deadline. A fetch with a bound sends an _rv_lease_ms header with its _rv_acquire. The owner ends the lease when no _rv_pull renews it within that time. An owner from before the header ignores it. An acquire without the header, from an older consumer or from a public get, gets a lease with no deadline, as before. A get caller can still hold its lease after the last pull.

A sweep in the store enforces both, every 100 ms. It starts when a deadline or a time-to-live is set, and it exits when none is left, so an instance that stages nothing with a deadline arms no timer. A UCX instance stops it at shutdown, together with the RDMA reaper.

The lease deadline is a header, not a field of RvAcquireRequest, because a new public field breaks code that builds the request. A message from the receiver that abandons the fetch was rejected. Before _rv_acquire returns, the receiver has no lease to name, and a lost message still leaves the payload staged. What still stays staged is in the limitation.

The receiver's limit

A staged payload's size is its owner's word: the acquire answer states total_len, or a descriptor states its length. With no limit, the consumer allocates that size before it pulls a byte, so one peer can make another allocate any amount for one message. VeloBuilder::staged_payload_max sets the largest payload a node takes this way. The transparent resolver checks the stated size against it before it allocates anything, on the chunked path and on the RDMA path.

A payload over the limit is refused:

• The consumer releases the handle, so the owner frees the staged copy. Transparent handles are single-use, and a detached one would stay on the owner with nobody left to read it.
• A request that expects a reply fails with an error that names both sizes. Its handler never runs.
• A stream item on the messenger mux ends its own stream with a TransportError that names both sizes, and the producer is told to stop (Batched streaming). Other streams from the same peer go on.
• Any other fire-and-forget message is lost, and the refusal is logged.

The limit does not bound a payload under the staging threshold. Such a payload travels inline, in one transport frame, and the frame limit bounds it.

The size is checked after the acquire, from its answer, not by a _rv_metadata query first: that query would add a round trip to every staged payload to catch the rare one that is too large. The check is not after the allocation, because by then the memory is spent.

Reading on the consumer

• get GETs into a pool buffer on the consumer, then copies into Bytes so the pool space returns at once.
• get_pinned returns the pool buffer itself. On the chunked path it copies the chunks into a pool buffer, so the return type is the same on both paths.
• get_into with a PinnedWriter (from alloc_pinned_writer) GETs straight into the caller buffer. Other RendezvousWrite destinations get a GET into a pool buffer and one copy.

The GET runs on a spawned task that owns the destination reservation (a TransferHold). If the caller drops the get future, the task keeps running and the granules stay reserved until the backend reports completion. Without this, a dropped future returns the granules to the free list while the NIC still writes into them.

Leases and deadlines

The owner sees every chunk request, so it sees an abandoned chunked transfer stop. It does not see an RDMA GET, because the consumer's NIC issues it. Without a deadline, a consumer that crashes mid-GET holds the read lock and the refcount forever. So RDMA leases carry a deadline.

• lease_timeout defaults to 30 s. The owner sends it to the consumer as lease_timeout_ms.
• The owner reaper scans every lease_timeout / 2, with a 10 ms floor. A silent lease is force-released between one and one and a half timeouts after its last renewal.
• While a GET runs, the consumer sends _rv_lease_renew every lease_timeout / 2, with a 5 ms floor. A lost renewal is harmless, because the next one is half a deadline away.
• Renewal stops when the transfer ends. Holding an RDMA lease idle past its deadline is not supported. A caller that holds data across a long pause releases it and acquires again.

On the wire, 0 means "no deadline". A consumer that gets 0 starts no renewal ticker. A sub-millisecond lease_timeout therefore clamps to 1 ms at build time. Without the clamp, the owner arms a deadline that the consumer was told does not exist, and the lease never renews. The clamp makes small values defined, not useful. Renewals go out at most every 5 ms and the reaper scans at most every 10 ms, so values below about 100 ms reap live transfers.

The descriptor

The RDMA answer carries a velo-owned binary descriptor, not JSON:

backend:    u8      1 = ucx
version:    u8      1
flags:      u8      0
generation: u64le   owner's registration generation
addr:       u64le   owner-authored absolute address
len:        u64le   bytes to read
rkey_len:   u16le   length of the packed key that follows
rkey:       [u8; rkey_len]

The header is 29 bytes, then exactly rkey_len key bytes, then nothing.

The format is binary because the packed rkey goes to ucp_ep_rkey_unpack, which has no length parameter. A truncated or corrupt blob is an out-of-bounds read inside UCX, and no Rust wrapper can make that safe after the fact. The framing must therefore be exact and owned by velo.

Decoding refuses anything that it cannot account for byte for byte:

• an unknown backend or version
• a non-zero flags
• a zero len
• an rkey_len that disagrees with the bytes that follow
• any trailing byte

A refusal is not an error for the caller. It costs one extra round trip on the chunked path.

The key is bounded at 4096 bytes by the descriptor and at 1024 bytes by the UCX backend. The backend checks the bound at map time and again before unpack. Before any pointer reaches ucp_ep_rkey_unpack, the UCX backend also walks the two stages of the UCX 1.22 packed-rkey format. It refuses a blob unless UCX's own parse ends inside the blob, and it refuses an out-of-range mem_type. This pre-parse checks framing only. A stale key with perfect framing passes it. See RDMA design for why that matters.

generation is in the descriptor for diagnostics. The consumer does not send it back in detach or release.

Version skew

Two #[serde(default)] fields make a mixed-version deployment fall back to chunked instead of failing:

• RvAcquireRequest::rdma (the offer). An old owner ignores it and answers Ready. An old consumer omits it, and an owner that sees no offer never answers Rdma.
• AcquireResponse::Rdma::lease_timeout_ms. If a future owner omits it, the default 0 means no deadline, which is the old behavior.

The rule for new fields: every field is #[serde(default)], and every default means what the protocol did before the field existed. A default that means "the new behavior" makes an old peer's silence look like consent.

Memory registration

Velo registers memory in two ways. Both end in the same lifecycle.

The arena pool

Velo sets UCX_RCACHE_ENABLE=n and UCX_MEM_EVENTS=n by default, so UCX installs no malloc hooks in the process. With those settings, every ucp_mem_map is a fresh ibv_reg_mr whose cost is linear in size, and UCX caches nothing. Pre-registered arenas are therefore required, not an optimization.

• The pool registers a few large arenas. Each arena is one page-aligned allocation, one ucp_mem_map, and one packed rkey.
• The first arena is 64 MiB (initial_arena_bytes). Later arenas grow geometrically up to 1 GiB (max_arena_bytes).
• Inside an arena, offset-allocator hands out 4 KiB granules in O(1). Its float bins waste at most about 12.5%.
• A request of 64 MiB or more (dedicated_arena_min) gets its own arena. Without this, the 12.5% round-up costs 128 MiB on a 1 GiB object.
• The suballocator node pool is sized to the granule count, about 28 bytes per granule (0.7% of the arena). A smaller node pool makes a fragmented arena report "full" while it still has room. The pool then maps an arena that it does not need.
• The free token (the private Allocation inside a PinnedBuf) and the wire descriptor are different values. A peer with a descriptor can read the range. It cannot free it, and the descriptor does not keep it alive.

registered_bytes_budget (1 GiB by default) caps mapped bytes across the pool and external regions together. It counts what the kernel pins: whole pages, reconciled to the effective range that UCX reports. A mostly empty arena costs its full size. Over the budget, pinned staging falls back to plain memory. Budget exhaustion is never a hard error.

Pages outlive their registration. Arena memory is freed only after the backend confirms the unmap. A registry torn down without its shutdown sweep leaks its arenas and logs at error. A leak is the safer failure, because freeing pages that UCX still pins is the hazard this layer exists to prevent.

External registration

Velo::register_external_memory(ptr, len) registers memory that the caller owns and returns a RegionGuard. The function is unsafe. Until RegionGuard::deregistered() resolves, the caller guarantees all of these:

• ptr is valid for reads and writes of len bytes. Registering a read-only mapping is undefined behavior.
• ptr + len does not wrap.
• The allocation is not freed, moved, remapped, or reallocated.
• No Rust reference into the range exists. A peer can write at any time.

Velo::register_owned(Box<[u8]>) is the safe form. Velo holds the buffer until an unmap is confirmed. On failure, the buffer comes back inside RegisterOwnedError, because BudgetExceeded is a routine refusal and must not consume the allocation. RegionGuard::unregister_owned gives the buffer back after release.

The guard does not borrow. Its lifecycle:

• deregistered().await is the single release signal. It resolves on a confirmed unmap, or at the end of graceful_shutdown, whichever comes first. Only then can the caller free the memory.
• unregister(timeout) gates new use, drains in-flight work, and unmaps. It returns Deregistered::Drained or Deregistered::DrainTimedOut. Both mean the memory is released. DrainTimedOut means velo did not wait for in-flight work. An earlier Err(Timeout) shape mixed up "still mapped" with "unmapped early", and only the first means "do not free".
• Drop without unregister starts a background deregistration, logs at warn, and returns at once. It never blocks, because blocking in Drop inside a tokio runtime deadlocks the worker. The memory stays pinned until the background task finishes. An early drop is a liveness bug for the caller, not a soundness bug for anyone else.
• watch() returns a RegionWatch. It carries the same observers but does not own the release, because unregister consumes the guard.
• If the progress thread dies while the backend still reports registrations, deregistered() never resolves. Velo cannot prove the memory was released, so it leaks the memory on purpose.

Registered means remotely writable

UCP has no enforced protection field. Any holder of a region's key can write the region, not only read it. The GET-only shape of the rendezvous protocol is a convention, not an enforcement. Registration pins whole pages, so bytes next to the allocation share the pinning and the remote writability. RegionGuard::effective_range reports what was pinned.

Registering memory is therefore a trust decision about the peers that this instance talks to. Key material only goes to peers that this instance already talks to. A &mut [u8]-shaped API is an aliasing lie, so velo does not offer one.

Lifecycle pressure

Registrations are evicted by a byte budget, never by an inactivity timer. A timer that evicts a registration while a peer still holds its descriptor creates a stale key. On InfiniBand, a stale key aborts the process. No surveyed system (UCX rcache, libfabric, MPICH, NCCL) evicts registrations on a timer.

Two timers exist anyway, on objects that no peer can hold a reference to.

Arena reclamation

One periodic sweep runs on the lease reaper's task. A second timer needs its own ordering against the first at shutdown.

• Empty pooled arenas, only when arena_reclaim_after is set (off by default). An arena with no suballocation has no live descriptor into it, so an unmap is safe. retain_arena_bytes (64 MiB by default) keeps a warm floor mapped, so an idle-then-busy workload does not pay a fresh registration. The sweep considers the newest arenas first, so the small first arena survives as the floor.
• Empty dedicated arenas, always, with no timer and no floor. A dedicated arena serves one request and is never offered again, so after its buffer drops it only consumes budget. Without this, a workload that stages 64 MiB objects repeatedly exhausts a 1 GiB budget after 16 of them and falls back to chunked for good.

The sweep period is the smaller of lease_timeout / 2 and arena_reclaim_after / 2, with a 10 ms floor. An arena is unmapped between one and two arena_reclaim_after intervals after it becomes empty.

Endpoint idle reaper

UcxTransportBuilder::ep_idle_timeout(Some(d)) closes UCX endpoints that nothing used for d. It is off by default and experimental. Values below 500 ms rise to 500 ms, which is about 35 times the 14 ms that the first RDMA GET on a fresh two-process pair over CX-7 InfiniBand took, wireup included.

• "Used" means both directions. Our sends, GETs, pings, and eager wireup stamp the endpoint. Every frame the peer sends us stamps it too, of any kind. Each frame header starts with the sender's 8-byte incarnation, and the receiver stamps the endpoint to the peer with that incarnation. The receive callback passes senders to the main loop through 8 slots, one per sender. If more than 8 different peers send frames between two drains of those slots, the main loop stamps every endpoint instead, so no sender's stamp is lost. On a worker that hears from many peers at once, idle endpoints can therefore stay open longer.
• Why a tag and not UCX's reply endpoint: UCX tells the receive callback which endpoint a frame came from only when the sender sets the REPLY flag. Velo sets that flag on Messages and pings only. A stamp that used it missed Responses, Events, Acks, Pongs, and ShuttingDown echoes, so a peer that streamed Responses back to us had our endpoint reaped under the stream. Setting the flag on every frame was rejected: after a reap, the peer's REPLY-flagged frames are lost (see below), so with the flag the next Response to a new request was lost too (3 of 3 in a probe). Without the flag, Responses kept arriving.
• An endpoint is idle only when no send posted on it is in flight and none has completed for the timeout. The reaper counts the sends in flight on each endpoint, and an asynchronous completion of the last one restarts the idle clock. A send that completes inline, while it is posted, records no completion time; the stamp taken when it was posted stands in for it. Replies on the endpoint that UCX hands the receive callback (Pongs and ShuttingDown echoes) are not counted. The inbound frame that caused each reply refreshes the endpoint in the same loop pass, before the reaper scans.
• The reaper never closes an endpoint while an RDMA operation to that peer is outstanding. When the operation completes, the idle clock starts again, as it does when a send completes.
• The scan runs every half timeout, and at least once per second. On an idle worker a due scan runs at most 100 ms late (the progress thread's park timeout). An endpoint closes between one timeout and one timeout plus one scan period plus 100 ms after its last use. A long pass on a busy worker delays the close further.
• The progress thread reads the reaper's clock a few times per pass, not once per use: at the start of the pass, after ucp_worker_progress returns, and after each ucp_ep_create. An inbound frame is stamped after it arrives, which keeps the endpoint open a little longer. A send can be stamped up to one command drain before it is posted, which can shorten the timeout by that much. A send whose asynchronous completion empties its endpoint's count also reads the clock once, in stamp_ep_drained, to record when it completed. With the reaper off, the progress thread does not read the reaper's clock per pass, and a send pays only two None checks, plus one clock read per ucp_ep_create.
• Idle time starts when ucp_ep_create returns, not before the call. In the in-process test harness over TCP on a GB200 node, with several UCX workers per process, one ucp_ep_create call took 110 to 150 ms. Across 31 creates in one process during the parallel UCX tests, the median was 630 ms, which is longer than the floor. The cause is not isolated. An endpoint with no send on it, such as an eager one, has only this stamp to keep it open.
• A first send between fresh workers also waits while the peer sets up its own endpoint back to us. With the CPUs oversubscribed, a fresh pair's first frame took 360 to 420 ms. With many UCX workers in one process, the peer's progress loop that contained that step took 526 to 573 ms. The count of sends in flight keeps the endpoint open for that time. The count costs no measurable time per send: on a GB200 node, in a release build with 6 alternating runs, the reaper-on send cost moved +1% to +3%, inside a noise floor of at least ±6%.
• The next use wires up a new endpoint with no error. The peer stays registered.

Closing an endpoint costs the peer. UCX pairs endpoints by remote worker. The peer's own endpoint back to us rides the same connection. After a reap, the peer's next Message to us is admitted and silently lost, with no error at either end (measured). By the mechanism inferred below, its next ping gets no Pong (not measured), so its check_health returns Timeout (or ConnectionFailed if keepalive fails the endpoint while the probe is waiting). Every Message the peer sends after that is lost too, until keepalive fails its endpoint: retrying does not help, and neither does a fresh endpoint on our side (measured by hand; the in-tree test asserts only the first loss). Pings are expected to be lost the same way (inferred, not measured). Its Responses and Events still arrive (measured: 8 of 8 arrived in every recorded run). Acks carry no REPLY flag either, so the same is expected, but it was not measured. The likely reason, inferred from the UCX source and a measurement but not proven: a REPLY-flagged frame carries an endpoint id for the reply path, and UCX drops a frame whose endpoint id no longer resolves. UCX keepalive (about 20 s) then declares the peer's endpoint failed, and the frame after that arrives. The cost per reaped endpoint is that the peer's Messages and pings to us are lost, silently, until keepalive (about 20 s) fails its endpoint. The first loss is a Message with no error, or a ping that times out. The disruption self-heals.

The reaper is meant for peers idle in both directions. That case still carries the cost above, whichever side speaks first, when the peer keeps its endpoint to us: if we speak first, our send arrives with no error, and the peer still loses its next Messages to us, and, by the same inferred mechanism, its pings. If the peer also runs the reaper and has closed its own endpoint, its next send creates a fresh one; that case is not measured. Send-only fan-out costs nothing, because no peer uses a path back. The reaper is unsafe for peers that this instance only health-probes. Each probe creates an endpoint, and each reap costs the peer its next Messages and pings to us, if it sends any before keepalive fails its endpoint.

The sender tag changes the wire format. The UCX blob version is 2. A version-1 node and a version-2 node refuse each other at register(), so all UCX peers must be upgraded together. See Version compatibility.

Eager endpoint wireup

UcxTransportBuilder::eager_endpoints(true) creates the endpoint at register() instead of at first use. It is off by default. The first GET on a fresh peer pair costs about 14 ms of UCX wireup, against about 108 µs for a warm GET (measured on 2026-08-29, see RDMA performance). Eager wireup moves that cost off the first transfer. The hint is fire-and-forget: register() does not wait, and a dropped hint falls back to lazy wireup.

With both knobs on, a peer that is registered but never used is wired up once and closed about one timeout (see the close window above) after ucp_ep_create returns. The UCX wireup itself finishes later, in the background.

Shutdown ordering

An RDMA GET is invisible to the owner's in-flight counters. An unmap after transport teardown can remove memory under a transfer that nobody on the owner can see. graceful_shutdown therefore runs the RDMA sweep before messenger teardown, under one shared deadline.

sequenceDiagram
    participant App
    participant Velo
    participant RV as RendezvousManager
    participant Reg as RdmaRegistry
    participant UCX as UCX progress thread
    App->>Velo: graceful_shutdown(policy)
    Velo->>Velo: begin_drain (messenger gate)
    Velo->>RV: stop lease reaper, demote pinned slots to heap
    Velo->>Reg: 1. Gate: refuse new registrations and allocations
    Velo->>Reg: 2. Drain: wait for in-progress registrations and region in-flight counts
    Velo->>Reg: 3. Deregister: unmap every region and arena
    Reg->>UCX: unmap (regions before endpoints)
    Velo->>Velo: messenger graceful_shutdown (remaining budget)
    UCX->>UCX: teardown force-unmaps anything left
    Velo->>Reg: latch deregistered() for every region

• Demotion copies, it does not drop. Each pinned slot is copied to the heap. A chunked pull admitted before the gate still finishes. A peer mid-GET holds a descriptor for the old address. That GET fails at its own end on InfiniBand and is silently lost over UCX_TLS=tcp.
• The gate is an ordering, not a flag. Admission takes an in-flight guard first and then reads the gate with SeqCst, as TransportAdapter::admit_message does. A plain token check followed by a map is a check-then-act race. A registration can land after step 3 and leave pinned memory with no tracking entry.
• The drain is bounded. When the budget runs out, velo warns and force-unmaps. The same straggler rule applies.
• RdmaConfig::shutdown_timeout (30 s) bounds the sweep when the policy is WaitForever. RdmaConfig::drop_dereg_timeout (30 s) bounds a background deregistration from a dropped guard.

See Shutdown and drain for the messenger half.

Configuration

Pass RdmaConfig with VeloBuilder::rdma_config. It has no effect without VeloBuilder::add_ucx_transport. A UCX transport added with add_transport gives full messaging and no RDMA registry.

The defaults keep the order rdma_min_bytes <= DEFAULT_THRESHOLD < DEFAULT_CHUNK_SIZE (64 KiB, 256 KiB, 512 KiB). A transparently staged payload is then large enough for the RDMA path from its first byte over the threshold. If rdma_min_bytes is above the transparent threshold, a band of payloads sits in pinned memory and never uses it.

The kill switch

VELO_RDMA_RENDEZVOUS_DISABLE=1 forces RdmaRendezvousConfig::enabled off. Velo reads it once, in VeloBuilder::build, so one process cannot answer half its acquires one way and half the other. Only 1, true, yes, and on (any case) count. Any other value leaves the path on. A switch that fires on a typo silently costs performance, but a missed typo shows in the metric.

The switch acts on both roles. An owner with it on never answers Rdma and stages new pooled slots in plain memory. A consumer with it on never sends an offer. Either side alone turns the path off, so a rollback works one node at a time. Pinned slots keep answering the chunked path, so no staged data becomes unreachable.

Metrics

Both sides record into path_total, at different points:

• The owner records at staging time, when register_data_pinned falls back (kill_switch, budget, pool_exhausted). It also records at each acquire (no_offer, not_configured, kill_switch, not_pinned, below_min, draining, ok).
• The consumer records when it builds its offer (not_configured, kill_switch, no_offer), after a failure (decode_error, pool_exhausted, get_failed), and when a GET completes (ok).

The series counts decisions, not transfers. With the owner's kill switch on, one slot adds kill_switch once at staging and again at each acquire. A fallback acquire carries no offer, so the owner also counts no_offer for a consumer whose GET failed. The owner counts ok when it sends a descriptor, before the GET runs. The consumer's ok is the proof that a GET completed.

This series is the only honest answer to "did the RDMA path run", because the chunked path also returns correct bytes. See Metrics reference.

Not yet built

• PUT in the other direction. The owner writes into a consumer-supplied buffer, flushes, and sends a completion message. The current decoder refuses any non-zero flags, and RdmaOffer carries backend names but no version. A direction flag is therefore not additive: every current peer refuses it and falls back to chunked during a mixed-version window. Either a version field in the offer ships one release ahead, or the rollout accepts chunked fallback everywhere in that window.
• A goodbye message before an idle close. It removes the Messages and pings that a peer loses after the endpoint reaper closes our endpoint to it.
• A deadline for every chunked lease. A chunked lease has a deadline only when its consumer asks for one. A consumer's LeaseGuard releases a chunked lease on every error path except one. That path is a release that cannot spawn because the runtime is shutting down. Nothing on the owner reclaims that lease, unless its slot has a time-to-live.
• A consumer-side unpacked-rkey cache. It removes both halves of the stale-key safety argument at once. An rkey outlives its operation, and the owner no longer revalidates each transfer. It is a safety change, not a profile-driven optimization.
• Register-in-place for large owned buffers. It needs an explicit API. Registering arbitrary caller Bytes by default is unsound: provenance is unknown and UCX pins past the caller's range.
• Metrics: an arena-utilization gauge, and a Prometheus series for endpoints closed by the idle reaper (the count exists only as an internal atomic).
• GPU memory, scatter-gather anchors, and an RDMA surface in velo-ext for out-of-tree transports.

Migrate ownership, dispatch, and cancellation to 0.19

Velo 0.19 makes runtime ownership explicit and removes duplicate dispatch and cancellation APIs. Stream wire formats and velo-ext traits do not change.

Runtime ownership

Keep a Velo owner for as long as its streams are needed. Velo is a handle, and clones share one instance. Dropping one clone does not stop streams. Dropping the final Velo owner cancels its streams and stops the builder-owned streaming listener and tasks, even if an Arc<Messenger> remains alive.

Messenger values no longer implement Clone. Clone Arc<Messenger> to share the instance:

let messages = Arc::clone(velo.messenger());
let streams = velo.clone(); // Keep this owner while streams are active.

The handlers that AnchorManager::register_handlers installs no longer keep their manager alive, so keep your own Arc<AnchorManager> for as long as you use it. The manager still keeps its Messenger alive: drop the manager too before you expect final Messenger drop to start teardown. RendezvousManager::register_handlers no longer keeps the Messenger alive. The public handler factories streaming::control::create_anchor_{attach,detach,finalize,cancel}_handler and streaming::mpsc::create_mpsc_anchor_{attach,detach,cancel}_handler are removed; call AnchorManager::register_handlers instead. A manager whose handlers owned it formed a cycle with its Messenger, so neither was ever dropped.

A retained Messenger supports active messages after the final Velo owner is dropped; it does not keep Velo streaming services alive. Stream handles and handler contexts can also hold Messenger references. Final Messenger drop starts immediate transport teardown and wakes remote event waits. It does not complete a response wait made through that Messenger: such a wait ends at the caller's own deadline. Velo's internal tasks that wait on a response end at teardown, except the handshake task for a peer that is not yet known, which ends at its 30-second handshake timeout. Local event completion remains available through retained event handles.

Drop requests cancellation and starts transport cleanup on an owned thread. It does not drain accepted work or wait for tasks or native threads to exit. A failed VeloBuilder::build also cleans up, in one of two ways. If anything fails while the transports start, the build stops the transports that already started on the building task, and waits for them to close before it returns the error, as step 3 of Shutdown and drain says. It skips that wait if a hook panicked, because a hook that panicked may never set up what its close waits on. If the build fails after its transports started, dropping what it built cleans up as Drop does, so a listener port can stay bound for a short time after the error returns. Explicit shutdown runs the same transport cleanup, and waits for it before it waits for transport close. Continue to call Velo::shutdown(policy).await when those guarantees are needed. RDMA users must use explicit shutdown before relying on deregistration; Drop does not report registered memory as released. Discovery registration guards and custom frame transports remain the caller's responsibility.

Handler dispatch

Replace .inline() with .spawn(), or omit the selector to use the default. The DispatchMode enum is removed. Select behavior with the builder methods: .spawn(), .ordered(), .ordered_global(), or .ordered_with(config).

Both old modes ran one task for each message. Neither guaranteed message order. Use .ordered() or .ordered_with(...) when handlers need to run in arrival order.

Accepted handler calls still count toward graceful shutdown until the handler and its response send finish. Messenger::tracker() remains available.

Stream cancellation

SenderEntry::rx_closer is removed, and SenderEntry has a new closed field. Code that constructs SenderEntry must omit rx_closer and set closed: Default::default(). Code that dropped its receiver to cancel a sender must call entry.cancel() instead. If you remove an entry from SenderRegistry to cancel its sender, call cancel() on it before you drop it.

The normal StreamController::cancel() and MpscStreamController::cancel() APIs need no changes. SPSC and MPSC senders check a cancelled flag before they send an item. SenderEntry::cancel sets the flag first, so the next send fails. Checking the token itself takes a lock on every item: 8.7 ns, against 0.3 ns for the flag. The token that cancellation_token() returns only observes the stream: it fires when the stream is cancelled, and cancelling it does not stop the sender. To end a stream from the producer, finalize, detach, or drop the sender. A cancelled stream also wakes a send that waits for channel space. Item and error sends return SendError::ChannelClosed after cancellation.

Graceful stop remains separate. For SPSC streams, request_stop() signals stop_token() and lets the producer send buffered output and call finalize(). Hard cancellation also signals the stop token, but rejects further item and error sends. Do not replace a graceful stop with cancel_token.cancel().

Migrate to the Velo handle in 0.21

Velo is now a cheap handle. VeloBuilder::build() returns Velo. It used to return Arc<Velo>.

Velo is Clone. A clone adds one reference count and does not copy any state. Every clone is the same instance.

How to migrate

• Replace Arc<Velo> with Velo in your types and signatures.
• Replace Weak<Velo> with WeakVelo in your types and signatures.
• Remove velo.as_ref() and &*velo. Pass &velo instead. Replace velo.as_ref().clone() with velo.clone().
• Keep .clone() as it is. It now clones the handle.
• Change Arc::clone(&velo) to velo.clone().
• Change Arc::downgrade(&velo) to velo.downgrade(). It returns a WeakVelo, and WeakVelo::upgrade() returns Option<Velo>. A handler that reaches its own instance must hold a WeakVelo. A Velo clone in the handler keeps the instance alive, so its final drop never runs.
• Change Weak::new() to WeakVelo::default(). It points at no instance, and upgrade() returns None.
• Change Arc::ptr_eq(&a, &b) to a.instance_id() == b.instance_id(). Clones of one instance share an InstanceId, and separate instances never do. Velo does not expose reference counts.
• The other Arc and Weak methods have no counterpart: try_unwrap, into_inner, get_mut, make_mut, strong_count, weak_count, and Weak::ptr_eq. For identity, compare instance_id(). To share, clone.
• Remove Arc::new(velo) if you wrapped a built Velo. Pass velo or velo.clone() instead.

// 0.20: `velo` is an `Arc<Velo>`
let velo = Velo::builder().add_transport(transport).build().await?;
let worker = Arc::clone(&velo);

// 0.21: `velo` is a Velo
let velo = Velo::builder().add_transport(transport).build().await?;
let worker = velo.clone();

Arc<Messenger> and the other Arc values that Velo hands out do not change.

What does not change

• Clones share one instance and one set of handlers, streams, and rendezvous state.
• The final drop of the last clone cancels the streaming services. Dropping any other clone does not.
• graceful_shutdown runs its teardown once, even when you call it through different clones. A later call waits for the first and then returns, unless the first teardown failed. Then a later call panics at once.
• shutdown calls graceful_shutdown, then closes the streaming services. It does the streaming step again on each call.
• Wire formats and velo-ext traits do not change.

Services and mux-only mode

Velo 0.19 groups distributed events, named work queues, and built-in discovery backends under one default feature, services. The default build keeps these APIs and the current stream listener behavior.

Select services at build time

For an application that registers peers itself:

[dependencies]
velo = { version = "0.19", default-features = false }

This build retains TCP and Unix sockets, typed handlers, ACK/NACK messages, SPSC and MPSC streams, per-stream TCP, the messenger mux, rendezvous, metrics, and discovery traits. Custom discovery implementations still work. Add ucx for UCX messaging and RDMA registration; those APIs do not need services.

The following APIs need services:

• velo::events, its root re-exports, VeloEvents, and event methods on Velo and Messenger.
• velo::queue, including the in-memory queue backend.
• Filesystem peer and service discovery.

nats-discovery, etcd, nats-queue, and queue-messenger enable services automatically. They still select their own backends. nats-transport does not enable services. This is a deliberate coarse boundary: selecting a discovery or queue backend also includes the other service APIs and their shared dependencies. It keeps the supported feature combinations small. Simulation remains independent of this feature.

If an existing dependency disables defaults and uses these APIs, add the feature:

velo = { version = "0.19", default-features = false, features = ["services"] }

Cargo combines features requested by all dependents. Another crate that enables Velo's default features also enables services for the shared Velo package. Without services, the messenger does not create an event manager or register the _event_* handlers. The fs4 and lru dependencies are omitted. Core request acknowledgements and errors keep their current wire format.

Select mux-only streams at runtime

The default builder creates a TCP stream listener beside the messenger mux. That listener allows streams from peers that do not offer the mux. If every peer supports the mux, use:

let node = velo::Velo::builder()
    .add_transport(transport)
    .mux_only()
    .build()
    .await?;

This starts no extra TCP or gRPC stream listener and advertises no stream endpoint. Ordinary SPSC attach, prebound tickets, MPSC attach, and large-payload rendezvous remain available. All remote streams use the messenger transport.

Do not combine mux_only() with stream_config() or stream_bind_addr(). build() returns an error for that combination, a disabled MuxConfig, or an active VELO_MESSENGER_MUX_DISABLE switch. It validates these choices before starting transports. Peers that need a per-stream transport receive an attach error. There is no silent fallback. To restore that fallback, remove mux_only() and restart the instance. The fleet-wide mux kill switch is therefore not a fallback for mux-only nodes.

The compile-time services choice and the runtime stream choice are independent. A node can keep all services and use mux-only streams, or omit services and keep the default stream listener. Use Velo::shutdown(policy) to close either kind of instance. See Shutdown and drain.

If a peer accepts an attach but the producer cannot connect, the producer waits up to one second for rollback before it returns the original connect error. SPSC rollback checks the producer identity so a late cleanup cannot detach a replacement. MPSC uses its sender ID. Rollback does not emit a Detached frame.

Older peers may not support the SPSC rollback handler. Cleanup failure is logged; there is no unqualified detach fallback. Existing receiver watchdogs and accept timeouts remain the backstop. A legacy transport's pending bind can remain until its accept timeout even after anchor rollback succeeds. Dropping the attach future, or losing the attach reply, is outside this cleanup guarantee.

Write an out-of-tree transport

A transport, frame transport, or discovery backend in another crate depends on velo-ext only. It does not depend on velo. velo-ext has no Prometheus, no Tonic, no NATS, and no dependency on velo.

[dependencies]
velo-ext = "0.5"

Pick the trait

Procedure for a messenger transport

1. Implement Transport for your type.
2. Send each inbound MessageType::Message frame through TransportAdapter::admit_message. Do not check ShutdownState::is_draining() first. See Shutdown and drain.
3. If admit_message returns AdmitOutcome::Draining and your transport has a return path, send a MessageType::ShuttingDown frame with the header of the rejected request. If there is no return path, record the rejection and drop the frame.
4. If admit_message returns AdmitOutcome::Disconnected, the runtime has stopped. Record TransportRejection::RouteFailed through the observability handle and drop the frame.
5. Route the other inbound types to their streams on the adapter: Response to the response stream, Ack and Event to the event stream, and ShuttingDown to the shutdown stream.
6. Store the handle that set_observability gives you. Record each inbound frame as you route it.
7. On the Admitted arm, and only there, record the frame as inbound message.
8. In shutdown, stop your tasks and threads. The runtime calls it at most once, with its Tokio runtime handle entered, on a dedicated thread, so you may block there, for example to join a thread. Two cases run it on the caller instead. Any failure while the transports start (another transport fails to start, a duplicate key, an address that cannot be merged) or a build that is cancelled then runs it on the building task. A thread that cannot be created runs it on whatever dropped or shut down the instance. After the final owner is dropped, the runtime may already be gone, so a task that shutdown spawns may never run. If shutdown panics, the runtime still calls the other transports' hooks, and then the shutdown call panics.
9. If your transport keeps state for each peer, override unregister to drop it and close the peer's connections. It must be cheap and idempotent. See Peer routing.

Step 7 matters because the inbound queue depth is a difference between two counters. If your transport admits a frame and does not record it, the depth goes negative. See Observability.

Step 6 matters because the runtime creates a series for each direction and message type in advance. If only message moves, the dashboard reads "no responses arrived", not "responses are not instrumented".

use std::sync::{Arc, OnceLock};
use velo_ext::{Transport, TransportObservability};

struct MyTransport {
    obs: OnceLock<Arc<dyn TransportObservability>>,
    /* ... */
}

impl Transport for MyTransport {
    // ... the required methods ...
    fn set_observability(&self, obs: Arc<dyn TransportObservability>) {
        let _ = self.obs.set(obs);
    }
}

Stability of velo-ext

New trait methods in velo-ext always have a default implementation, so a new release does not break your implementation. A change to an existing signature is a breaking release. See Versioning.

A transport with one ordered channel to each peer keeps the defaults of lanes and send_message_on_lane. Override both only if your transport keeps several ordered channels to one peer. The two methods need velo-ext = "0.5.4" or later. unregister needs velo-ext = "0.5.6" or later.

Tune batched streaming

Use this guide to change the settings of the messenger mux, choose a flush policy, or turn the mux off. Read Batched streaming first for the mechanisms that each setting controls.

The mux is on by default, and most deployments change nothing. Keep the defaults until a measurement shows a reason to change one.

The mux is on by default

A Velo builder installs the mux with MuxConfig::default(). The per-stream transport stays configured beside it, because negotiation needs it to serve peers that do not offer the mux. An attach uses the mux only when both sides advertise messenger-mux-v2. Every other pair uses the per-stream path.

To turn the mux off without a code change, set VELO_MESSENGER_MUX_DISABLE=1 and restart the process. Only 1, true, yes and on (any case) count. Velo reads the variable once, when it builds the node. The variable wins over enabled: true set in code, so a benchmark that must measure the mux must not inherit it.

A node with the mux runs one sweep task, which wakes five times a second, and registers 16 batch handlers (_stream_batch for lane 0 and _stream_batch.1 to _stream_batch.15), even if it never streams. A handler costs a map entry until its first batch.

Call messenger_mux to change a setting, or to turn the mux off:

#![allow(unused)]
fn main() {
use velo::streaming::MuxConfig;
use velo::{StreamConfig, TcpConfig, Velo};

let velo = Velo::builder()
    .add_transport(transport)
    // The per-stream path stays configured. Negotiation picks per attach.
    .stream_config(StreamConfig::Tcp(Some(TcpConfig::new(bind_addr))))?
    .messenger_mux(MuxConfig {
        enabled: false, // Turn the mux off. Omit this line to keep it on.
        ..Default::default()
    })?
    .build()
    .await?;
}

Call messenger_mux once per Velo instance. A second call returns an error.

Make sure that the mux carries your streams

1. On the sender, read StreamSender::negotiated_transport() after the attach.
2. Compare the result with velo::streaming::MESSENGER_MUX_KEY.

#![allow(unused)]
fn main() {
let sender = velo.attach_anchor::<Token>(handle).await?;
match sender.negotiated_transport() {
    Some(key) if key.as_str() == velo::streaming::MESSENGER_MUX_KEY => { /* multiplexed */ }
    Some(_) => { /* one connection per stream */ }
    None => { /* same worker, no transport */ }
}
}

3. On either node, read velo_streaming_mux_batches_total{direction="sent"}. The value increases only when the mux carries data.
4. Read velo_streaming_mux_records_per_batch{direction="sent"} to see how many records each batch carries.

negotiated_transport() exists on the SPSC StreamSender only. For an MPSC sender, use the metrics in steps 3 and 4.

If a benchmark must measure the mux, fail each request whose negotiated key is not MESSENGER_MUX_KEY. A silent fallback measures the per-stream path under a mux label.

Select a flush policy

If you do not own the send loop, use the default FlushPolicy::Auto. The batcher writes when the peer admits the previous batch.

If you own a loop that produces one record per stream per pass, use FlushPolicy::Manual. A decode engine is this kind of loop. Call flush_batch() once at the end of each pass.

#![allow(unused)]
fn main() {
let velo = Velo::builder()
    .messenger_mux(MuxConfig {
        flush_policy: FlushPolicy::Manual,
        ..Default::default()
    })?
    // ...
    .build()
    .await?;

loop {
    for (sender, token) in forward_pass() {
        sender.send(token).await?;   // stage
    }
    velo.flush_batch();              // one write per peer
}
}

CAUTION: Under Manual, call flush_batch() after every pass. Without the call, the last records of each stream stay staged and do not arrive.

flush_batch() is synchronous and returns at once. It flushes every peer, because a producer cannot know which peer each sender feeds. It is valid under Auto too, where it forces an early write. It does nothing when no mux is installed.

If you want batching with a time limit, use FlushPolicy::Auto(AutoFlush { on_admission: false, max_linger: Some(window), ..Default::default() }). The batcher then holds a batch for up to window after its oldest record. Two optional conditions end the wait early:

• Set on_complete: true when a lane often carries a single stream, such as a serving worker at low concurrency. The batch then goes once every live slot on the lane has a record in it, but not sooner than window after the last write that carried data. An empty flush_batch() and the write that opens a stream do not count. Without it, each record of a lone stream waits the whole window.
• Set at_bytes: Some(n) to write a batch once it reaches n bytes. Choose n near one transport segment of payload (1,448 bytes at MTU 1500, 8,948 at MTU 9000). Past that, a longer wait saves at most one partly filled packet per write, and adds latency. Unlike max_batch_bytes, it does not send larger records through rendezvous.

See Ending the window early.

Settings reference

The settings below are fields of MuxConfig. Always build it with ..Default::default() so that a new field does not break your code. The accept window of each bind is not a MuxConfig field: ReservationConfig::prebind_timeout sets it (180 seconds by default).

Change the credit window

initial_credit sets how many records a sender can send on one slot before it needs a grant. The consumer node's value is the one that counts: it advertises its window when a stream attaches, and the sender uses that.

• Keep the default of 32 for token streams. A larger window lets every stream fill the shared path between a producer and a saturated consumer node, and a new stream's first record then waits behind all of it. See The credit window and a saturated frontend.
• A stream longer than the window needs grants. The consumer node returns credit as its consumer drains, usually when the next batch from the producer arrives. A live consumer therefore rarely stalls its producer.
• The window is also how far a sender can run ahead of a consumer that stops polling. On a single-sender stream, credit returns only when the consumer takes a record from the slot buffer. An MPSC anchor's pump moves records into the anchor channel, so its senders can run further ahead.
• Do not set a small window to save memory. A small window raises the credit-return latency per record. For a producer that ran out of credit, it is (drain_visit_floor + reply_linger) / initial_credit.
• Raise the window for a few high-rate streams to an otherwise idle consumer. With no other traffic from the producer, a starved stream gets its credit back only through the doorbell and the reply linger, so a small window caps its rate.
• Do not set zero. Zero on the wire means "not offering the mux", and the build refuses it.

Each slot buffer holds initial_credit + 1 records. The byte budgets, not the record count, bound the memory.

Change the byte budgets

slot_byte_budget sets how far a producer can run ahead of its consumer before its send waits. A larger cap absorbs longer bursts, and costs that much memory per slot on the producer's node. A smaller cap makes the producer wait sooner.

If one peer carries many slots with large records, increase peer_byte_budget. It bounds only the receive side.

Read Stream saturation before you change either budget.

Leave the credit timers at their defaults

credit_sweep_interval, drain_visit_floor and reply_linger interact. Keep the defaults unless a measurement shows a reason.

• The arrival path returns most credit on the next inbound batch. The periodic sweep is a backstop. A shorter sweep interval costs CPU in proportion to peers × slots and gains little.
• drain_visit_floor limits contention on the ingress lock of each (peer, lane). Setting it to zero lets a fast consumer drive back-to-back walks under that lock.
• reply_linger lets credit replies share batches. The measurement used a 512-worker rig at 8,192-way concurrency. The 1 ms default cut the frontend's outbound batches by 4.3 to 5.9 times against zero. It also cut the frontend's total write time from 13.5–14.3 s to 2.8–3.9 s per run.

If you set reply_linger to zero, each credit reply writes its own batch again. Do this only to compare against the earlier behavior.

Do not enable async_open_ack without a measurement

async_open_ack returns from a slot open before the transport admits the OpenSlot. At 512 workers and 8,192-way concurrency, it made first-token p95 worse in every rep and did not improve p50.

It also parks producers behind a congested peer. On such a peer, a producer can fill the slot byte cap before its own OpenSlot is admitted, and then waits in send although its consumer is healthy. Nothing caps how many slots can wait this way against one congested peer, so egress memory grows with the number of opens.

Do not enable it unless both of these conditions are true:

• The queue depth of a congested peer is bounded.
• You measured the open time that it removes, at your own concurrency.

Use zero-RTT stream setup

Zero-RTT setup removes the attach round trip from each stream. If the consumer already sends each request to the producer in an envelope that you control, use it.

1. On the consumer, create the anchor.
2. Call velo.prebind_anchor(handle) from a runtime context.
3. If the call returns Some(ticket), put the ticket in the request envelope. StreamOpenTicket implements Serialize and Deserialize.
4. If the producer is a different worker and the envelope has a ticket, call velo.open_anchor_stream::<T>(handle, ticket) on the producer.
5. If the producer is the worker that called prebind_anchor, call velo.attach_anchor::<T>(handle). open_anchor_stream on the minting worker fails at once.
6. If the envelope has no ticket, call velo.attach_anchor::<T>(handle).

#![allow(unused)]
fn main() {
// Consumer, at request registration.
let anchor = velo.create_anchor::<Token>();
let ticket = velo.prebind_anchor(anchor.handle());
send_request(RequestEnvelope { handle: anchor.handle(), ticket }).await?;

// Producer, on receipt.
let minted_here = envelope.handle.unpack().0 == velo.instance_id().worker_id();
let sender = match envelope.ticket {
    Some(ticket) if !minted_here => velo.open_anchor_stream::<Token>(envelope.handle, ticket).await?,
    _ => velo.attach_anchor::<Token>(envelope.handle).await?,
};
}

prebind_anchor returns a ticket whenever the consumer has the mux and the anchor can be pre-bound, and the mux is on by default. A producer without the mux cannot open that ticket: open_anchor_stream fails, and the first attach_anchor is refused. The refusal releases the pre-bind, so a retry attaches on the per-stream path. To avoid these failures when you roll the mux out, upgrade the producers before the consumers that mint tickets, or keep the mux off on those consumers until every producer has it.

A ticket sender observes stop_token() and cancellation_token() through the mux's slot lifecycle. Stop leaves the stream open for final output. Cancel or consumer drop wakes an idle producer and ends delivery; it does not need a later send.

The ticket stays valid for the configured accept window (180 seconds by default). After the window, the consumer reaps the bind and sees SenderDropped.

Roll back

1. Set enabled: false, or set VELO_MESSENGER_MUX_DISABLE=1, on the nodes that mint tickets.
2. Restart those nodes.
3. Do the same on the producers.
4. Restart the producers.

CAUTION: Do not roll back a producer alone while its consumer still mints tickets. The consumer refuses the producer's attach, because the producer no longer offers the pre-bound key.

Rolling the mux out needs the reverse order: producers first, then the consumers that mint tickets. See Use zero-RTT stream setup.

Without zero-RTT setup, the order does not matter. Each new attach negotiates the per-stream path.

Settings that are not levers

These changes were measured and did not help. Do not try them again without a new reason:

• A grant threshold of half a window, measured at a window of 256. It cut credit traffic 19-fold, did not change CPU, and made inter-token p99 worse.
• A 500 µs data linger on the producers (Auto { max_linger }). It saved about 1 ms of frontend CPU per request and added about 10 ms to the request path.
• A shorter credit_sweep_interval for faster credit. The arrival path already returns credit on the next batch.
• async_open_ack, as described above.

Response plane performance has the measurements.

For work that exceeds the initial window, renew held tickets with AnchorManager::renew_reservations. Configure lifetimes through VeloBuilder::reservation_config; see reservation renewal.

Run rendezvous over RDMA

This guide builds velo with the ucx feature and wires the UCX transport for the rendezvous RDMA path. It then verifies that the path runs on the NIC you expect. For how the path works, see Rendezvous and RDMA.

What you need

• Linux on x86_64 or aarch64. On other platforms the ucx feature builds but adds no transport.
• A C compiler and GNU make. The vendored UCX tarball ships a generated configure, so autotools and cmake are not necessary for this crate.
• The rdma-core development headers. The build looks for three files: infiniband/verbs.h, infiniband/mlx5dv.h, and rdma/rdma_cma.h. On Debian and Ubuntu, libibverbs-dev and librdmacm-dev supply them.
• For real RDMA: an InfiniBand or RoCE port in state PORT_ACTIVE, and a memlock limit large enough for the registered-bytes budget.

The headers are necessary even for a build that only runs over UCX_TLS=tcp. The UCX tests and CI run that way with no RDMA hardware.

Build

1. Install the headers:

   sudo apt-get install -y libibverbs-dev librdmacm-dev
   

2. Turn on the feature in your Cargo.toml:

   velo = { version = "...", features = ["ucx"] }
   

3. Build:

   cargo build --features ucx
   

The first build compiles UCX 1.22.0 from the tarball inside crates/ucx-rs. This takes about 25 s on a 20-core machine (about 73 CPU-seconds). Later builds reuse it in about 0.3 s. A stamp file records the exact configure arguments and toolchain variables, so a change to either rebuilds UCX.

The result links UCX statically. At run time, no UCX shared objects, plugin directories, or LD_LIBRARY_PATH settings exist. The only dynamic dependencies are libibverbs.so.1, libmlx5.so.1, and librdmacm.so.1.

Why the build refuses a TCP-only UCX

If the verbs headers are missing, UCX's own configure prints a warning and builds a UCX with only TCP and shared memory. That build passes CI and then runs production traffic over TCP. ucx-rs refuses instead, with a panic that names the missing header.

For the same reason, ucx-rs passes --with-devx. Upstream defaults DEVX to check, which silently drops the accelerated mlx5 memory domain.

Build variables

The velo feature ucx turns on the ucx-rs default features: ib, rdmacm, and mt. You cannot turn off ib through velo.

Wire the transport

Add UCX with add_ucx_transport, not add_transport. Only add_ucx_transport creates the RDMA registry. Keep a TCP transport beside it for the control plane.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use velo::transports::tcp::TcpTransportBuilder;
use velo::transports::ucx::UcxTransportBuilder;
use velo::{RdmaConfig, Velo};

let tcp = Arc::new(
    TcpTransportBuilder::new()
        .from_listener(std::net::TcpListener::bind("0.0.0.0:0")?)?
        .build()?,
);
// Eager wireup moves the ~14 ms first-GET endpoint cost to register().
let ucx = Arc::new(UcxTransportBuilder::new().eager_endpoints(true).build()?);

let velo = Velo::builder()
    .add_transport(tcp)
    .add_ucx_transport(ucx)
    .rdma_config(RdmaConfig::default())
    .build()
    .await?;

// Owner: stage in registered memory. This call also maps the first arena.
let handle = velo.register_data_pinned(&payload).await;
}

The consumer calls get, get_pinned, or get_into as usual. Register the peers before the first get. A get to an owner that is not registered yet takes the chunked path.

If the only staging in a process is transparent (large messenger payloads), call register_data_pinned once at startup. The transparent stager never maps an arena, so without that call it never uses the RDMA path.

Set the runtime environment

Velo reads the UCX variables from the environment. The environment wins over the builder. UcxTransportBuilder::tls and net_devices apply only when UCX_TLS or UCX_NET_DEVICES is not set.

NOTE: No hardware run has used velo's build on rc_mlx5 yet. All two-node results so far are on rc_verbs (see RDMA performance). On rc_mlx5, do all steps in Verify the lane.

Velo sets UCX_RCACHE_ENABLE=n and UCX_MEM_EVENTS=n unless you set them. With these values, UCX patches no libc functions in the process. Keep them. Velo registers memory explicitly and never uses the UCX registration cache.

Do not set UCX_PROTO_ENABLE=n. It is a common workaround for protocol-v2 faults, and it silently disables all RMA. A multi-rail GET needs UCX_MAX_RMA_RAILS of 2 or more. The default is 1.

Prepare each node

1. Find the port:

   ibv_devinfo
   

2. Pick a port with state: PORT_ACTIVE and link_layer: InfiniBand. An active Ethernet port on the same card carries RoCE and looks like a success.

3. Set UCX_NET_DEVICES to that port, for example mlx5_2:1.

4. Set UCX_TLS. Include a ud transport beside the rc transport.

5. Raise the memlock limit:

   ulimit -l unlimited
   

6. If the IB partition gives this node partial pkey membership (0x7fff), set UCX_IB_PKEY explicitly. UCX_IB_PKEY=auto needs full membership, and raw verbs works where UCX fails.

Some systems ship a memlock limit of 8 MiB. With that limit, ibv_reg_mr fails at 32 MiB, and the first 64 MiB arena cannot map.

Verify the lane

A correct transfer does not prove that RDMA ran. The chunked path also returns correct bytes. Do all three steps.

1. Run with UCX_LOG_LEVEL=debug and find the memory domain line:

   UCX_LOG_LEVEL=debug ./your-binary 2>&1 | grep 'md open by'
   

   On a mlx5 NIC, the line must say uct_ib_mlx5_devx_md_ops. If it says uct_ib_verbs_md_ops, the accelerated transports have no devices.

2. Run with UCX_PROTO_INFO=y. Find the row rendezvous zero-copy read from remote. It must name the transport and device that you set, for example rc_mlx5/mlx5_2:1.

3. Read velo_rendezvous_rdma_path_total on the consumer. The series {path="rdma", reason="ok"} must increase with each transfer. Any other reason names why the path was not used. See Rendezvous and RDMA.

Do not use the system ucx_info to verify velo's build. The system UCX loads its transports as plugins and is a different library. On the same NIC, the system UCX can report rc_mlx5 while velo's static build does not.

Do not trust a configuration header that a program prints. The environment overrides the builder, so a header that says UCX_TLS=tcp can describe a run on InfiniBand. The UCX_PROTO_INFO table is the truth.

Version compatibility

Two velo nodes can talk over UCX only if they use the same UCX wire version. Each node publishes that version in its UCX address, and register() refuses a peer on a different one.

The current version is 2, from velo 0.15.0. Version 2 starts every frame header with the sender's 8-byte incarnation, so that the endpoint idle reaper can tell which peer sent each frame. A version-1 node would misread those 8 bytes, so the two versions refuse each other instead.

Upgrade all UCX peers together. During a rolling upgrade, an old node and a new node cannot reach each other over UCX. register() returns InvalidEndpoint, and the refusing node logs a warning: ucx: rejecting peer blob: unsupported ucx blob version 1 (expected 2).

Run the two-process example

The example stages 8 MiB on an owner and pulls it on a consumer, with a TCP control plane and UCX beside it. The consumer exits non-zero if the transfer did not take the RDMA path.

On one host, over TCP:

UCX_TLS=tcp cargo run --manifest-path examples/Cargo.toml --release --features ucx --example rendezvous_rdma_two_proc

With no --role, the example starts both roles as child processes.

On two nodes:

1. Build on both nodes:

   cargo build --manifest-path examples/Cargo.toml --release --features ucx --example rendezvous_rdma_two_proc
   

2. Pick a directory that both nodes can read and write, for example /shared/rv.

3. Start the owner on node A:

   UCX_NET_DEVICES=mlx5_2:1 UCX_TLS=rc_mlx5,ud_mlx5,self \
     ./examples/target/release/examples/rendezvous_rdma_two_proc --role owner --dir /shared/rv
   

4. Start the consumer on node B, with the same variables and --role consumer.

5. Read the velo_rendezvous_rdma_path_total lines that both sides print at the end.

Use --size <bytes> to change the payload. The two processes find each other through JSON files in --dir. On NFS, negative-dentry caching can delay that exchange by about 30 s. That delay is not a velo latency.

CI runs the single-host form with UCX_TLS=tcp on every pull request.

Run the tests

The UCX suites set tls("tcp") in code. The environment wins, so the same suites run on InfiniBand when UCX_TLS and UCX_NET_DEVICES are set.

# Rendezvous RDMA path, end to end
timeout 300 cargo test -p velo --features ucx,test-helpers --test rendezvous_rdma

# UCX transport and RMA plumbing
timeout 300 cargo test -p velo --features ucx,test-helpers --lib transports::ucx

# Registration layer
timeout 300 cargo test -p velo --features ucx,test-helpers --lib rendezvous::rdma

# The mlx5 memory domain sits ahead of the verbs one
timeout 300 cargo test -p ucx-rs --test ctor_order

# Registration and GET cost (prints numbers, asserts nothing)
timeout 300 cargo test -p velo --features ucx,test-helpers --lib bench_rma -- --ignored --nocapture

CAUTION: Do not run unusable_rkey_is_refused_by_ucx_over_tcp on InfiniBand. It aborts the process with SIGABRT there. Over TCP it passes.

Skip it on an InfiniBand run:

timeout 300 cargo test -p velo --features ucx,test-helpers --lib transports::ucx -- --skip unusable_rkey_is_refused_by_ucx_over_tcp

Common failures

Run the examples

The examples are in the examples/ crate. This crate is not a workspace member. Run the commands from examples/, or add --manifest-path examples/Cargo.toml from the repository root.

Messaging examples

cargo run --example ping_pong --all-features -- --rounds 1000
cargo run --example throughput --all-features -- --count 10000
cargo run --example mpsc_fanin --all-features -- --producers 4 --items 40

ping_pong and throughput take --transport {tcp,uds,zmq,nats,grpc,quic,ucx}. The default is tcp. The nats transport needs a nats-server on 127.0.0.1:4222.

The latency budget (tx_budget)

tx_budget adds one layer on each rung. The difference between two rungs is the cost of the layer that was added.

Each rung is an echo ping-pong of the same frame size in both directions, so the one-way time is half of the round trip. Use tx_budget to find which layer caused a change in latency.

Soak

soak messenger|stream|rendezvous|all --transport tcp|grpc --tier ci|nightly|long [--faults]

CI runs the ci tier on each PR. A nightly workflow runs the full set.

Benchmarks

For batched_streaming, response_plane_bench, stream_lifecycle and anchor_setup_bench, see Benchmarking. For rendezvous_rdma_two_proc, see Run rendezvous over RDMA.

Observability

velo::observability gives Prometheus metrics for all subsystems. You create a Registry and register VeloMetrics into it. Velo does not run an exporter. Serve or scrape the registry with your own code.

use prometheus::Registry;
use velo::{Velo, VeloMetrics};

let registry = Registry::new();
let metrics = Arc::new(VeloMetrics::register(&registry)?);
let node = Velo::builder()
    .add_transport(tcp)
    .metrics(metrics)
    .build()
    .await?;

For the full list of families, see the Metrics reference.

Families

With the distributed-tracing feature, Velo puts OpenTelemetry trace context in the message headers.

Read some metrics as differences

Some quantities have no gauge. A sampled channel length reads the wrong number under the load that makes the depth worth knowing. Velo gives two counters instead, and you subtract them.

Before you subtract, aggregate both sides to the same labels. The two sides often have different labels, so a bare a - b matches nothing.

Inbound queue depth

sum by (job, instance) (velo_transport_frames_total{direction="inbound",message_type="message",outcome="accepted"})
- sum by (job, instance) (velo_messenger_inbound_dequeued_total)

• The result can go negative for a short time. A transport counts the frame after admit_message has already put it on the queue, so the consumer can count the departure first. Clamp at zero.
• After a Timeout shutdown, the result stays high by the number of messages that were abandoned. Those messages never count as departures.
• The difference is correct only for transports that record what they admit. All in-tree messenger transports do. The simulation transport does not.

Egress queue depth, for each transport

sum by (job, instance, transport) (velo_transport_frames_total{direction="outbound",outcome="accepted"})
- sum by (job, instance, transport) (velo_transport_frames_written_total)

This is the number of frames in front of the socket: in the bounded send channel, or staged in the writer. It has these limits:

• It does not include frames held in the admission gate. velo_transport_send_backpressure_total counts those, and velo_transport_egress_queue_wait_seconds covers both the gate and the channel.
• It stops at the kernel. By default the TCP sockets request a 2 MiB send buffer (see Socket buffers). To see bytes below the counter, read tx_queue for the socket. On QUIC it stops at quinn's send buffer instead, in user space, and a frame counted as written can still wait there for the peer's acknowledgement.
• It can go negative for a short time. Clamp at zero.
• It stays high after a frame fails: a replaced connection, a socket error, a failed connect, or a stop of the writer. Those frames were accepted and never written.
• Only the coalescing writer (TCP, UDS, and QUIC) publishes frames_written. gRPC, NATS, ZMQ, and UCX have no such series, so the query returns no rows for them. Do not add or vector(0), because zero reads as "queue empty", not "not measured".
• The transport label is the TransportKey of the transport, not a fixed name. Select on the series, not on a name pattern.

Queueing on the receiver during attach

  sum(rate(velo_streaming_anchor_attach_rtt_seconds_sum{outcome="success"}[5m]))
/ sum(rate(velo_streaming_anchor_attach_rtt_seconds_count{outcome="success"}[5m]))
-
  sum(rate(velo_streaming_anchor_operation_duration_seconds_sum{operation="attach",outcome="success"}[5m]))
/ sum(rate(velo_streaming_anchor_operation_duration_seconds_count{operation="attach",outcome="success"}[5m]))

Both families are histograms, so the query compares mean durations from _sum and _count. The sum removes the labels that differ between sender and receiver.

The sender records the round trip. The receiver records the time inside the handler. The difference is an upper bound on the queueing at the receiver: it also includes the send path, both wire legs, the handler spawn, and the wake of the sender task.

• The query uses outcome="success" on both sides, because the error populations do not compare.
• The query sums away transport_scheme and instance. The two series come from different nodes, and the sender can record "unknown" for a scheme that the receiver named.
• SPSC and MPSC attaches share one RTT series, so the result is an average over both.

Batched streaming

For the mux families and how to find a stream that cannot keep up, see Stream saturation.

Stream saturation

If streaming traffic shows sender errors, missed deadlines, or a Dropped frame with no producer crash, use this runbook. The usual cause is saturation. The producer generates frames faster than the consumer drains them. Velo reports saturation through Prometheus counters and one log line. This page tells you which signals to read and what to do.

The signals differ between the per-stream path and the messenger mux. Read The per-stream cascade for streams with one TCP or gRPC connection each. Read Saturation under the mux for streams that negotiated messenger-mux-v2.

The per-stream cascade

A per-stream stream uses a chain of bounded flume channels around a TCP or gRPC byte stream:

flowchart TD
    P[producer: StreamSender::send] --> C1[connect-side channel, 4096]
    C1 --> T[TCP socket buffers and network]
    T --> C2[bind-side channel, 4096]
    C2 --> R[reader pump]
    R --> A[anchor frame_tx, 256: smallest, fills first]
    A --> Q[consumer: StreamAnchor::next]

When the consumer falls behind, the 256-deep anchor channel fills first. Pressure then moves back up the chain:

1. The anchor channel (256) fills.
2. The reader pump's try_send returns Full. velo_streaming_reader_pump_backpressure_total increments.
3. The bind-side channel (4,096) fills.
4. The server pump's try_send returns Full. velo_streaming_server_pump_backpressure_total increments.
5. The TCP receive buffer fills, and the kernel sends a zero-window ACK.
6. The producer's TCP write stalls.
7. The connect-side channel (4,096) fills.
8. StreamSender::send sees Full. velo_streaming_producer_send_backpressure_total increments, and the producer waits in send_async.
9. The sender's heartbeats use try_send, so a full channel drops them. After DETECTION_MULTIPLIER × heartbeat_interval of silence (3 × 5 s by default), the reader pump's watchdog fires. velo_streaming_heartbeat_watchdog_firings_total increments, and the session ends.

The consumer sees one of two results when the watchdog fires:

• The anchor channel had room. This is the common case, usually a producer that died. The consumer receives a Dropped frame, which StreamAnchor::next() returns as StreamError::SenderDropped.
• The anchor channel was full. The watchdog uses a non-blocking try_send, so that registry and cancel cleanup cannot deadlock. The Dropped frame is lost. The consumer drains the queued frames and then receives None (a clean end of stream). A tracing::warn! line with the local_id records the loss.

In both cases velo_streaming_heartbeat_watchdog_firings_total is the authoritative signal. Do not detect watchdog kills from StreamError::SenderDropped alone.

Counters

All four counters have no labels. They register in the Prometheus registry that you pass to Velo::builder().metrics(...).

A backpressure counter measures events, not latency, throughput or bytes. A high rate means that the system is at its capacity. A zero rate means that there is headroom.

Dashboard

Build at least three panels:

1. rate(velo_streaming_reader_pump_backpressure_total[1m]). A rising rate is the earliest warning on the per-stream path. For mux streams, use rate(velo_streaming_slot_credit_exhausted_total[1m]) instead, which the producer's node counts.
2. rate(velo_streaming_server_pump_backpressure_total[1m]) and rate(velo_streaming_producer_send_backpressure_total[1m]) beside the first panel. When these rise in sequence, the cascade is moving up.
3. increase(velo_streaming_heartbeat_watchdog_firings_total[5m]). Any non-zero value is a dead session. If the rate panels rose first, the cause was saturation. If they stayed flat, the producer crashed or the network failed.

The watchdog log line

When the watchdog fires on the per-stream path, the reader pump writes one warn line:

reader_pump: heartbeat watchdog fired, injecting Dropped (saturation indicator: see velo_streaming_*_backpressure_total)
  local_id=...
  anchor_frame_tx_len=256 anchor_frame_tx_cap=256
  transport_rx_len=4096 transport_rx_cap=4096
  heartbeat_deadline_ms=5000
  detection_multiplier=3

Read the channel depths:

• If anchor_frame_tx_len equals anchor_frame_tx_cap, the consumer side was saturated.
• If both depths are near zero, the silence came from upstream of the consumer. The cause is a producer crash, a network partition, or a backlog on the producer's egress.

On a single-sender mux stream, the stream watchdog writes a different line. An MPSC anchor's reader pump, on any transport, neither logs a firing nor counts it:

stream_watchdog: nothing arrived from a sender holding credit for the detection window, injecting Dropped
  local_id=...
  slot_buffer_len=...
  arrivals=...
  heartbeat_deadline_ms=5000
  detection_multiplier=3

This watchdog fires only when nothing was delivered to the slot for the whole detection window while its sender still held data credit or had records waiting behind a sequence gap. Heartbeats spend data credit, so a sender that held credit could have sent one. A consumer that falls behind leaves its sender without credit, and the watchdog exempts such a sender, so a firing on a mux stream is never a consumer that fell behind. Records still in the slot buffer when it fires are not delivered: the consumer reads SenderDropped next. The silence came from upstream: a producer crash, a network partition, or a backlog on the producer's egress or the peer link. On a mux peer link that carries thousands of streams, heartbeats wait in the same queue as data. A deep enough backlog there silences a live sender. In one measured UCX run, 1,722 streams on one congested peer were killed this way while their worker was healthy.

Mitigations

Apply these in order, from the smallest change to the largest:

1. Slow the producer. A tokio::time::sleep of 100 µs between sends, or tokio::task::yield_now().await, often ends saturation. The producer does not know the consumer's rate. Back off voluntarily, or use the producer-side backpressure counter as feedback.
2. Speed up the consumer. Move work out of the anchor.next().await loop into a separate task. The anchor consumer must only take each frame and hand it on.
3. Resize the MPSC anchor channel. MpscAnchorConfig::channel_capacity sets the depth of an MPSC anchor channel (256 by default). A larger channel absorbs bigger bursts at a memory cost per anchor. The SPSC anchor channel is fixed at 256.
4. Reduce the number of concurrent anchors on the per-stream path. Each anchor adds channel memory and a reader pump. If your application creates one anchor per work item, batch the work items into one anchor. For many streams to few peers, use the mux, which is on by default. See Tune batched streaming.

Saturation under the mux

A muxed stream has no socket of its own. It has credit. A consumer that stops draining stops returning credit, and the stream's egress parks. The producer then waits, as it waited on a full socket buffer.

The batcher pulls each slot's records from the slot inlet into a per-slot withheld queue, whether or not the slot has credit. The slot byte budget (1 MiB by default) bounds that queue. When the queue reaches the budget, the batcher stops pulling from that slot's inlet. The inlet (C+1 records deep) then fills, and StreamSender::send waits. When credit returns and the queue drops below the budget, the batcher pulls again.

flowchart TD
    P[producer: StreamSender::send] --> I[slot inlet, C+1]
    I -->|until the byte budget| W[batcher: withheld queue, slot byte budget]
    W -->|credit available| B[_stream_batch on the peer connection]
    B --> S[consumer slot buffer, C+1]
    S -->|read directly| Q[consumer: StreamAnchor::next]
    W -.->|budget reached: inlet paused| I

Producer backpressure

A producer faster than its consumer waits in send. The stream stays open, and nothing is dropped. This holds for a consumer that drains more slowly than the producer sends, and for a consumer that stopped draining.

finalize, detach and Drop are synchronous. When the inlet is full, the terminal waits in a task, so these calls do not block. The terminal then goes out after the records ahead of it.

An earlier design closed the slot when the queue passed the budget. That design also closed streams whose consumer was draining, because any producer faster than one credit round trip reached the budget within milliseconds.

The consumer reads the slot buffer itself, so credit returns only when it takes a record. A consumer that stops polling holds its sender to the credit window C, and then to the byte budget. The stream watchdog exempts a sender that holds no credit, so it never ends a stream whose consumer stopped polling with its window full. Such a stream stays open until the application drops the anchor. If the producer dies while it still holds credit, the watchdog ends the stream on time, even with records unread. If it dies holding no credit, the watchdog ends the stream once the consumer reads enough to return credit to it. velo_streaming_reader_pump_backpressure_total does not move for mux streams. Watch velo_streaming_slot_credit_exhausted_total, which the producer's node counts.

If the slot is fenced behind an unresolved OpenSlot or rendezvous admission, its records wait for that admission, and the producer waits at the byte budget. If a producer leaves while its slot is fenced, the consumer's Dropped waits for the admission too. If the admission fails, the failure is epoch death for the (peer, lane) of the slot. The slot is retired without the deferred Dropped, and the consumer falls back on the heartbeat watchdog.

The knob is MuxConfig::slot_byte_budget. A larger budget lets a producer run further ahead of its consumer, and costs that much memory per slot on the producer's node.

Mux counters

velo_streaming_producer_send_backpressure_total changes meaning under the mux. On the per-stream path it means that the 4,096-deep connect-side channel was full. Dashboards built on that meaning shift once streams negotiate the mux, which they do by default. Watch velo_streaming_slot_credit_exhausted_total for consumer-driven saturation.

The async_open_ack exposure

With MuxConfig::async_open_ack enabled, the OpenSlot fence withholds a slot's records from the first record, with or without credit. A producer that starts generating into a peer whose send queue is congested fills the byte budget before its own OpenSlot is admitted, and then waits in send. The cause is sender-side congestion, not the consumer.

peer_byte_budget bounds the receive side only. N concurrent opens into a stalled peer can hold N times the slot byte budget in egress memory. The default awaited open serializes new opens behind the same admission and bounds this to one wait at a time.

Write coalescing ratio

Two counters describe the write side on the per-stream path:

Their ratio is the coalescing ratio. The egress pump packs everything already queued on one stream into one flush. A stream whose producer runs ahead reports hundreds of frames per flush. A stream that emits one frame at a time reports about 1.0.

A flush is a unit of coalescing, not a syscall. write_all can loop over several writes. An oversized frame is written in segments and still counts as one flush. The kernel decides the TCP segmentation.

A ratio near 1.0 is not a fault. It means that one frame was queued each time the pump woke. It matters when many anchors stream to the same peer. Coalescing is per stream, so it cannot pack frames spread across streams. That case is what the mux solves. See Batched streaming. Under the mux, read velo_streaming_mux_records_per_batch{direction="sent"} instead.

Benchmarking

Use this chapter to measure the streaming response plane. It describes the four in-repo harnesses, the external serving rig, and the measurement rules that earlier mistakes made necessary. Response plane performance has the results. To compare two commits on one machine, see Review for performance.

Select the harness

The first two in-repo harnesses model the same serving shape. Anchor hosts stand in for frontends and own one response stream per request. Engines run continuous batching: each forward pass puts exactly one token on each active request's stream. Request dispatch is an in-process channel. The response path is real: anchors live on different workers from the engines, so every attach and every token crosses a loopback TCP socket.

Run batched_streaming

Run all commands from the examples/ directory:

cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 24 --max-batch 8 --tokens 40

The example has three anchor hosts. It fails the run if any request misses a token, sees an unexpected frame, or if the batcher's velo_streaming_mux_records_per_batch{direction="sent"} does not account for every token.

Keep --pass-delay-ms above zero for a per-stream comparison. With no gap, the engine runs ahead, several tokens queue on each stream, and per-stream coalescing packs them. That flatters the per-stream path and hides the effect under test.

To reproduce the flush-policy comparison in Batched streaming, run each configuration five times with --flush-policy auto and --flush-policy manual:

cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 96 --max-batch 32 --tokens 40 --flush-policy auto --initial-credit 256
cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 96 --max-batch 32 --tokens 40 --flush-policy manual --initial-credit 256
cargo run --release --all-features --example batched_streaming -- --engines 2 --requests 24 --max-batch 8 --tokens 40 --pass-delay-ms 0 --flush-policy auto --initial-credit 256

CI runs batched_streaming at the defaults with both flush policies.

Run response_plane_bench

Run all commands from the examples/ directory:

cargo run --release --all-features --example response_plane_bench -- --anchor-hosts 2 --engines 128 --requests 2000 --credit-sweep-interval-ms 200

CAUTION: Set --credit-sweep-interval-ms 200 to match velo's default. The harness default of 2 ms reproduces an earlier default and exists for the sweep A/B.

The harness reports TTFT and ITL as HDR histograms (p50, p95, p99). It also reports requests per second, tokens per wire write and process CPU. CPU is split into user and system time from /proc/self/stat. The consumer asserts the position of every token, so a run fails if sharing a batch ever reorders a stream.

Sweep --engines, not --anchor-hosts. The per-peer costs on a frontend scale with its ingress peers, and an anchor host's ingress peers are the engines that stream to it. --anchor-hosts moves the smaller side of the same product.

The harness agrees with batched_streaming. At --anchor-hosts 3 --engines 2 --requests 96 --max-batch 32 --tokens 40 --initial-credit 256, it reports 5.41 tokens per write where batched_streaming reports 5.38, both at a credit window of 256.

Limits of the in-process harness

• One process. The reported CPU belongs to the whole topology, not to one frontend. An A/B that changes one frontend-side setting is valid. A comparison with another system's per-frontend CPU is not.
• Loopback. It removes wire time, which exaggerates the syscall term in favor of the mux.
• Size. It runs out of memory above about 256 nodes in one process on a 374 GiB host. A 512-worker shape needs the external rig.
• Queueing. With --max-batch well below --requests, most of TTFT is the wait for a batch slot. Read TTFT and requests per second only as comparisons between arms.
• Raw percentiles. The harness reports percentiles over every recorded request after --warmup-requests. It does not separate steady state from the opening burst. See Steady-state and raw percentiles.

Run stream_lifecycle

Run all commands from the examples/ directory. The harness runs two nodes in one process over loopback TCP, on the messenger mux, or on one TCP connection per stream with --path per-stream. The producer opens short streams, finalizes each one, and then does what --mode says:

Wall time moves with the load on the machine. Compare builds and modes by the instructions of the process, divided by --streams, and take the median of several runs. On macOS:

cargo build --release --example stream_lifecycle
/usr/bin/time -l target/release/examples/stream_lifecycle --mode plain

On Linux, use perf stat -e instructions instead of /usr/bin/time -l. Batched streaming has the results for the mux.

On the per-stream path (--path per-stream), keep --streams to a few thousand and leave time between runs: each stream opens a connection, and the ephemeral ports run out. Provisional, on the same laptop and under the same conditions as the mux table, with --streams 4000 --warmup 400, the two modes interleaved, median of five: plain 525,586 instructions per stream (520,910 to 526,380) and retired 527,360 (526,161 to 529,770). The spreads overlap, so awaiting retired() costs nothing this run can resolve; the pump reports the retirement in both modes, and most of each figure is the connection each stream opens. The producer holds a stream for about 130 µs p50 and 500 µs p99 after finalize while it waits for the write. To be confirmed on dedicated hardware.

Run anchor_setup_bench

Run all commands from the examples/ directory. The harness has no network and no mux, so its numbers show the cost of the anchor path only. It runs two phases, each on its own runtime and manager:

1. Create --held SPSC anchors and hold them unattached. The harness reports the create time of each anchor, and the tokio tasks and the RSS that they add.
2. Run --tasks concurrent tasks. Each task runs --streams lifecycles: create an anchor, attach a sender on the same worker, finalize, and read the terminal. The harness reports streams per second and the p50, p99 and p99.9 of one lifecycle.

cargo run --release --example anchor_setup_bench

To compare two builds, run each build three times or more, interleave the runs, and take the median. The tail of the lifecycle depends on the thread count: a lock that every create takes shows at 8 threads and not at 1 or 2. Response plane performance has the results for the unattached timeout.

The external serving rig

The external rig runs velo as the response plane inside Dynamo's serving stack. It uses Dynamo's own HTTP frontend, its mocker engine and the aiperf load generator. The same rig runs Dynamo's own response planes, so only the response plane changes between arms. The rig scripts and the Dynamo adapter live outside this repository. The table below is the shape of the main comparison; other measurements state their own shape.

Steady-state and raw percentiles

A closed-loop load generator at concurrency N starts its measured phase with N requests at once. aiperf 0.10.0 ramps request rate, not concurrency. On the external rig, each rep therefore opens with a burst of 8,192 simultaneous requests. The 8,192 warm-up requests warm caches, but the measured phase still opens with its own burst.

The burst sets the raw p99. In six reps on the external rig, every request at or above the raw TTFT p99 started within the first 0.2 s of the measured phase. That tail was 760 to 930 ms for both velo and the comparison plane. Its excess over the median came mostly from the request plane and the mocker processes absorbing 1,024 simultaneous requests each, not from the response plane.

The rig's summary therefore reports steady-state percentiles beside the raw ones. Steady state is the set of requests that started 10 s or later into the measured phase, about 200,000 of 250,000 per rep. Read verdicts from the steady-state columns. Report the raw columns too, so that a reader sees the burst.

Measurement rules

Each rule below exists because a measurement without it was wrong.

1. Run on an exclusive node. A CPU A/B ran on a shared login node with about 11 other users. Its spread was up to 5.5 µs per token, against an effect of about 2 µs per token. The result was retracted. On an exclusive 144-core node, the total spread over 42 runs was 2.43 µs per token.
2. Build both arms from one tree on the node, and interleave them. Run the arms alternately, rep by rep, so that drift hits both. Normalize CPU per token or per request over identical work.
3. Pin the load generator away from the system under test. With aiperf unpinned on the frontend node, node A ran at 95% utilization. Its load average was 159 on 144 cores. velo's TTFT p50 read 992 ms. With the frontend and aiperf on disjoint cores, the same configuration read 98 ms. Pin etcd and nats-server to the load generator's cores too. Unpinned, they took 12% to 38% of the frontend's cores in profiled reps.
4. Read timestamps in the right zone. aiperf 0.10.0 writes naive local timestamps. Reading them as UTC put the CPU window after the run, on the idle tail. Every frontend CPU figure measured that way was about three times too low.
5. Compare reps at a matched backlog draw. On the external rig, one or more mocker processes fall behind during the opening burst and keep that backlog. The number of processes that hold the backlog (the "holders") sets throughput, ITL and end-to-end latency for every arm. Compare two arms only at the same holder count.
6. Do not fix the arm order. A matrix that always ran the same arm second put that arm in a degraded band in every rep. Its request-level numbers measured run position, not the setting.
7. Change one setting per arm. A pair that differed in two settings attributed a result to the wrong one.
8. Assert the transport per request. Fail any request whose negotiated key is not messenger-mux-v2. A silent fallback measures the per-stream path under a mux label.
9. Record the build. Write the velo commit and its dirty state into each rep's metadata. If the installed build and the checkout differ, fail the rep.
10. Report latency beside throughput. A change that adds latency can raise throughput by letting the consumer catch up. Watch TTFT in particular, because any windowed flush policy can make it worse.
11. Prove a regression test before you trust it. Revert the fix and make sure that the test fails. A test that passes with the fix reverted proves nothing.
12. Check that every process was scraped. The rig gave worker process p the metrics port base + p. With 16 processes the range covered port 9100, which node_exporter holds on the compute nodes. Process 10 served traffic with no /metrics, and every summed worker metric missed a sixteenth of the fleet. An empty scrape file must fail the rep.
13. Time the first record at both ends before you blame a stage for TTFT. Dynamo's worker time_to_first_response stops when the prologue is sent, not at the first token, so it cannot separate engine queueing from the response path. The rig adds histograms for the engine's first item and for its send on the worker, and for the prologue and the first data item on the frontend. With them, a 200 ms TTFT gap turned out to be the response path (the first token left the worker 10 ms after the request arrived) and not the mocker. See The credit window and a saturated frontend.
14. Sort reps by the environment before you compare arms. On the 2-node rig, reps fall into two states that move every arm, the comparison plane included. In the bad state, frontend CPU per request rose from about 11 to 16–20 ms and irq and softirq time on node A rose from 5–9 to 13–16 cores. Network softirq runs on the frontend's own pinned cores: with the frontend on 24 cores, it took 5 to 7 of them, in both planes. Three reps that mix the states average two regimes and settle nothing.

Response plane performance

This chapter reports how velo performs as the response plane of an LLM serving stack. It gives the result, the changes that produced it, the changes that did not help, and the mechanisms found on the way. Benchmarking describes the rig and the measurement rules. Read it before you compare any number here with a new one.

The response plane is the path that carries generated tokens from a worker back to the frontend. In these measurements velo runs the messenger mux over TCP inside Dynamo's serving stack. The comparison plane is Dynamo's multiplexed TCP response plane (Dynamo PR 11918, ported onto the same Dynamo base so that both planes share one stack). Both planes carry many streams over a few shared connections and batch their frames.

Result

Measured on 2026-09-10 on the external rig that Benchmarking describes: 512 mocker workers, concurrency 8,192 and 250,000 requests per rep. The frontend ran on one tokio runtime with 72 workers. Each of two matrices ran three reps per plane.

The velo configuration: mux over TCP, zero-RTT stream setup, write on admission (FlushPolicy::Auto with on_admission), reply_linger 1 ms, initial_credit 256 (the default when these were measured; now 32, see The credit window and a saturated frontend), async_open_ack off.

A holder is a mocker process that keeps part of the 8,192-request backlog after the opening burst. The holder count sets throughput and the tails for every plane, so reps compare only at a matched holder count. Steady state is the requests that started 10 s or later into the measured phase. Where two reps share a draw, both values are shown.

• At 1 to 3 holders, velo's first-token p50 is 2 to 7 ms ahead. At 5 to 6 holders, the two are level (55.5 against 55.1 ms).
• Steady-state p90 and p99 are level or ahead.
• Throughput is equal within the draw.
• Every rep in the two matrices had zero errors.
• Frontend CPU per request is 1.2 to 2.9 ms higher on velo. This is recorded, not a requirement.

Against the shipping per-request plane

Dynamo's shipping plane opens one TCP connection per response. Measured on 2026-09-05 with core pinning, before zero-RTT setup. The velo arm here is the mux with an attach round trip per stream. CPU is the corrected figure.

The per-request plane's accept loop is bistable. In the jammed state about 3,900 sockets sit connected but not accepted, and TTFT is the backlog divided by the accept rate. Its raw p99 of 1.7 s comes from that loop.

Dynamo's QUIC response plane (PR 11996) did not complete a clean run at this scale on this cluster. Six reps, measured before core pinning, had 4,947 to 84,525 errors each: HTTP 500s and streams with no content. Its low TTFT medians in those reps were survivorship, because dropped requests do not queue. A likely factor is net.core.rmem_max at 212,992 bytes, which clamps QUIC's UDP sockets about 30 times below what quinn requests. This is a result for this cluster, not a general verdict on QUIC.

What produced the result

The changes are listed in order of effect. Each one landed with a test that fails without it.

1. One tokio runtime in the frontend. This is the largest lever, and it is outside velo. The Dynamo frontend ran two 72-worker runtimes on 72 cores. The Python bridge built its own runtime lazily, because its initialization code sat in a branch that never ran. The velo node ran on the first runtime. The HTTP handlers and the per-stream reader pumps ran on the second. Every response record crossed between them through the injection-queue mutex of the second runtime. That lock took 2.4% of the cores plus 6.8% in lock contention. A worker blocked on that mutex does not service the time driver. Linger timers then stopped for seconds, and credit stopped. The 8,192 streams fell into a 6 to 8 s limit cycle. The fix initializes the bridge with Dynamo's runtime, a 24-line change. Thread count fell from 242 to 154.
2. Zero-RTT stream setup. The worker sends without waiting for the attach round trip. The worker-ingress-to-first-token segment fell from 78 to 48 ms at p50.
3. Credit on the next batch for every drained slot. Each drain names the slot it drained, and the next inbound batch reconciles it. Before this change, each inbound batch walked every slot of its peer. At about 1,000 slots per peer and 11 slots touched per batch, flume::Shared::len alone took 2.1% of the frontend's 72 cores, about 0.5 ms per request.
4. One timer per stream in the reader pump. The pump used to build a tokio::time::timeout per record. Each one took the time driver's lock twice. The Sleep subtree was 7.3% of the frontend's cores, plus 0.8% for the cancellation future. With one pinned timer, the timer subtree fell to 0.15%. The mux has since dropped the reader pump; its stream watchdog wakes once per window and does no timer work per record. See The reader pump leaves the mux data path.
5. Reply linger. Credit replies wait up to 1 ms to share a batch. With zero-RTT setup, the frontend's egress had carried one batch per credit reply: 1,027,872 outbound batches against 97,042 without zero-RTT. The linger cut frontend batches four to six times and saved 0.4 ms of CPU per request.
6. Control maps bounded by allocation. A 4,096-entry control cap was sized for about 1,024 slots per peer. One mocker process held 4,000 to 6,700 live slots. The cap refused credit grants and closes, and with async_open_ack it refused the answer that lifts a slot's fence. A fenced slot then waited for the 15 s watchdog, and the client saw an HTTP 500.
7. Drain-driven credit return with a visit floor. Measured on 2026-09-01 on an exclusive 144-core node with the in-process harness at 256 ingress peers, 42 interleaved runs, at a credit window of 256 with streams of 64 tokens, so no stream ran out of credit:

The hook costs about 2%. It is what makes the 200 ms sweep safe, because the sweep was the only path that returned credit to a quiet peer. The net gain is 4.5%, and the case for the change is correctness. Without a floor on doorbell visits, a stream drained through an 8-record window rang the doorbell 1,000 times in about 270 ms. The 2 ms floor spread the same 1,000 walks over about 3 s.

The Dynamo adapter also polls the anchor inline from the connection task instead of through a consumer task with a 64-deep mailbox. This removed about 0.2 ms of adapter CPU per request.

What is not a lever

These changes were measured or traced and did not help. They are recorded so that nobody tries them again without a new reason.

Where velo's extra CPU goes

A per-subtree partition of a perf profile of both frontends, with one runtime (one rep each, measured on 2026-09-06). Each stack line goes to exactly one bucket, and each plane sums to 100%. The values are milliseconds per request.

velo's anchor and adapter consumer (1.46) cost about what the comparison plane's reader and receiver cost (1.24). The excess is task hops. The reader pump is a channel-to-channel relay with no counterpart, and the lane, dispatch and transport are three stages where the comparison plane has one task. Each record also reaches the HTTP connection task as its own wake.

After the inline receiver, a second profile put the velo-only buckets at 3.20 ms per request. The buckets were: reader pump 1.01, anchor 0.85, ingress 0.51, adapter 0.49, TCP 0.18, dispatch 0.14, batcher 0.01. The reader pump and the anchor channel are the remaining addressable surplus. The pump cannot merge into the anchor channel (see the table above). Each other hop-chain change is optional and needs a same-matrix tail check. These two profiles predate the direct feed, which takes the reader pump off the mux data path. See The reader pump leaves the mux data path.

Before the fixes above, these symbols appeared only in velo's profile, as a share of the 72 frontend cores:

Removed, measured on the rig rather than in a profile: each bind used to spawn one task that slept for the 60-second accept window, and a claim did not cancel it. A frontend at about 2,000 requests per second held 146,000 live tokio tasks against about 25,000 on the QUIC plane. The accept window is now a deadline in one queue that the credit sweep drains, so a bind owns no task and a window closes up to one sweep interval late. The test a_finished_zero_rtt_stream_leaves_no_task_behind holds the task count to its baseline after streams end.

Removed, measured with anchor_setup_bench: each SPSC anchor used to spawn one task for its unattached timeout. Attach cancelled the task, but each stream still paid one spawn and one exit, and each unattached anchor held one task. An unattached deadline is now an entry in one queue per manager, and one task sweeps that queue. The sweep wakes at the earliest entry, so a timeout fires at its deadline, as before. The host is one aarch64 machine with 20 cores (10 Cortex-X925, 10 Cortex-A725). The unattached rows are the median of three interleaved runs at 8 tokio threads. The lifecycle rows are the median of five interleaved runs of 16M lifecycles (3.84M at 2 threads), 64 tasks. Such a run lasts about 10 s, so the sweep pops entries for all but its first second. CPU is the user and system time of the process, divided by the lifecycles.

At 2 threads the p99.99 ranges overlap: 127 to 251 µs with a task per anchor, and 133 to 216 µs with the queue. A run of 1.28M lifecycles ends before any entry comes due, so it measures only the push; the 16M run includes the sweep.

The queue does not remove an entry when its anchor attaches or ends. The sweep drops the entry when it comes due. A new entry comes due within 1 s, and a requeued entry within 60 s, so the queue holds about stream rate × 1 s of stale entries, at 24 bytes each. An entry queued at the deadline of its anchor stayed for the full timeout instead: 120 s by default, and 30 years for Duration::MAX.

Rejected: one lock for the whole queue. Every create takes a lock on the queue. With one lock, the lifecycle p99.9 at 8 threads was 110 µs, against 22 µs with a task per anchor. No entry came due inside that run, so the sweep did not cause the tail: creates waited on each other for the one lock. With 1 or 2 threads, the single lock was faster than the task at every percentile, so the cost was contention. The queue now has 16 parts, each with its own lock, and the id of an anchor selects its part.

The tests unattached_anchors_share_one_sweep_task and ended_anchors_leave_the_queue_within_the_first_look hold the task count and the queue length. MPSC anchors use a second queue of the same kind, and mpsc_anchors_share_one_sweep_task holds its task count.

Removed, measured with anchor_setup_bench and stream_lifecycle: each SPSC and MPSC sender spawned one heartbeat task that lived as long as its stream. Now one task per AnchorManager beats every remote sender, from the same kind of queue. A sender on the anchor's own worker does not beat at all, because no watchdog reads its beats. Same host and settings as above, median of three interleaved runs, against the deadline queue alone:

The two rows measure two different parts of the change. A same-worker sender now does not beat at all, so the same-worker row shows the cost of a heartbeat task per stream, not the cost of the sweep. Only the mux row exercises the sweep. Both arms of each row have the deadline queue. The same-worker runs are 1.28M lifecycles, which end before the anchor sweep pops anything, in both arms.

On the mux path the difference is inside the spread of the runs. The test senders_share_one_heartbeat_task holds the task count, and a_silent_remote_stream_outlives_the_watchdog_window holds an idle remote stream open for 10 heartbeat intervals.

These costs are counted from source, not measured:

• The per-record copy. Ingress copies each record body into a Vec (one allocation and one memcpy, estimated at 35 to 55 ns). Removing it needs Bytes from the slot buffer through to the consumer, and it is not done (see Batched streaming design). On the microbenchmark below, memcpy was 0.5% of frontend CPU. A typed Bytes consumer copies each payload a second time, in serde. Byte items remove that second copy.
• Wakes. flume fires a waker only for a parked receiver, so k records for one slot in one batch already cost one wake. With the direct feed, a record makes one hop, from the slot buffer to the consumer.

The reader pump leaves the mux data path

Measured on 2026-09-26 with a two-process microbenchmark shaped like response_plane_bench, on one Grace node: frontend 32 threads, 2,048 streams, 918 records of 160 B, jemalloc. The figure is frontend CPU per record, median of 3 runs. Each pair of rows is its own A/B. The direct-feed baseline is a separate run of the dirty-set build, so the rows do not chain.

In the profiles before these changes, the reader pump was 28% of frontend CPU at 12 peers. In the dirty set, only the drain that newly lists a slot touches the peer's shared wake flag. The direct feed removes a task wake, a channel send and receive, and a hook allocation per record. These are microbenchmark numbers, not rig numbers.

First-token mechanisms

A per-request join splits client TTFT into three segments. The join matches aiperf's request id, the frontend's log and the mocker's log, with clock skew below 1 ms. The three segments sum to the client's TTFT.

• A: client send to the frontend's "request received" log line (HTTP ingress).
• B: the frontend to the worker handler's start (the request plane).
• C: worker ingress to the first token at the client (mostly the response plane).

TTFT equals the wait for the first HTTP byte to within 0.3 ms at every percentile, in every plane. No plane writes a byte before the first token.

These mechanisms were found:

• A saturated load-generator node looked like a slow plane. Before core pinning, velo's TTFT p50 read about 1.1 s. The queues that velo owns held a small share of it: 26 to 36 ms in the node-global inbound queue, about 110 ms in the ordered lanes and 79 ms in the worker's egress queue. The rest was the time between a wake and a run on a node at 95% utilization. Pinning moved velo's p50 to 98 ms at the same throughput.
• The attach round trip was the largest term. Before zero-RTT setup, the worker waited for _anchor_attach before it started generating. Under load before pinning, that round trip averaged 390 to 524 ms. With pinning it averaged 13 to 22 ms. Zero-RTT setup removes it.
• The comparison plane has nothing on the first-record path that grows with load. The frontend mints the stream id at request registration. The prologue goes on an urgent lane that the writer drains before ordered data. The only shared queue that a first frame crosses is bounded by a 256 KiB byte budget per connection.
• Zero-RTT setup first moved the cost into the request path. With two frontend runtimes, segments A and B each grew by about 8 to 9 ms under zero-RTT setup, while C fell. The frontend sent ten times more batches, one per credit reply. The reply linger removed the extra batches but not the A and B growth, so the batches were a symptom of frontend contention. After the one-runtime fix, velo's p50 moved ahead of the comparison plane, and segment B contributes nothing to the steady-state tail.
• TTFT p50 followed the backlog draw. Before the one-runtime fix, velo's p50 was 54 to 55 ms at one holder, 61 to 62 at two, 73 to 76 at three and 83 at five. The comparison plane stayed at 48 to 49 ms. After the fix, velo's p50 is 39 to 45 ms at one to three holders.
• The steady-state tail is the hot mocker process and HTTP ingress, in both planes. In steady state, segment B contributes nothing to p99. Segment C adds 125 to 209 ms, and 59% to 89% of the p90-to-p99 band sits on one mocker process with 3,000 to 6,400 requests in flight. Segment A adds 39 to 112 ms. The raw p99 is the opening burst, described in Benchmarking.
• End-to-end and ITL tails measure one mocker process. Arrivals are equal across the 8 mocker processes. In-flight counts are not: seven sit at 150 to 200 and one at 3,000 to 7,000 for the whole run, in every plane. By Little's law, a request on the hot process stays about 15 s against 0.43 s elsewhere. End-to-end p99 and ITL p99 therefore measure which process holds the backlog, not the plane.
• A handler wait cost a hello round trip per request. The Dynamo adapter calls Velo::wait_for_handler before each request to make sure the worker can reach the frontend's _stream_stop handler. wait_for_handler refreshed the peer's handler list with a full _hello round trip every time, and under a saturated frontend that round trip waited in the frontend's messenger queues. It now returns at once when the handler list already names the handler. The test test_wait_for_handler_reuses_cache_and_refreshes_missing_handlers counts one handshake for repeated waits on a known handler, and a new one for a missing handler or a new instance. On the rig its effect on TTFT was within the rep-to-rep spread. A Dynamo adapter change also prepares each peer once instead of per request.
• velo's tail discipline is conditional. Before pinning, velo appeared to keep a much shorter end-to-end tail. That was a starved frontend limiting how many streams ran at once. With the frontend on 48 cores and the load generator on 96, velo held 8.1 s end-to-end p99 at 3,008 req/s against the comparison plane's 25.2 s at 2,460. On a 72/72 split, both planes posted about 11.5 s.

The credit window and a saturated frontend

Measured on 2026-09-26 and 2026-09-27. When the frontend is the bottleneck, every stream runs at its credit window, and the windows fill the shared path between a worker and the frontend: the worker's egress queue, both socket buffers and the frontend's reader. A new stream's first record queues behind all of it. The comparison plane has no per-request credit, and its worker-side send blocks, so its backlog stays in the producer.

Setup: the Dynamo mocker rig with 16 processes of 32 workers, concurrency 8,192, input length 1,024, output length 900, speedup 10 and 250,000 requests per repetition. The frontend was pinned to 24 or 32 cores. Each allocation ran one window against a window of 256, three repetitions per arm with the arms interleaved; ranges cover all three. The last column is a stream's first record (its prologue), from the worker handing it to the plane to the frontend receiving it, as a mean over requests.

Rows are grouped by allocation. In the 64 allocation, two of the three 256 repetitions and one of the 64 repetitions fell into the degraded environment state that Measurement rules describes. In their good repetitions, 64 read 98 ms of TTFT against 156 ms at 256, and its first record took 31 ms, the top of the range of 32. The data does not separate 64 from 32 beyond that.

The default window is 32. At 24 cores it brings velo's TTFT to the comparison plane's, with throughput within noise of 256 in the same allocation and ITL p99 no higher; ITL p50 is 0.1 to 0.4 ms higher, because streams need more grants. At 32 cores, 32 and 256 read the same TTFT p50, throughput and ITL, and the first record still waits less at 32. A window of 16 had the lowest TTFT at 24 cores, but it doubles the per-record credit-return cost ((drain_visit_floor + reply_linger) / initial_credit, about 190 µs at 16 against 94 µs at 32), and it was not measured at 32 cores.

At 32 the surplus waits in each stream's own withheld queue on the producer rather than in the shared path. In the first repetition of the 24-core allocation that ran 32 against 256, summed over the 16 worker processes, sender credit exhaustion rose from about 240,000 to about 1.2 million events; the worker's mean egress queue wait per frame fell from 9.5 to 1.2 ms, and the frontend's mean lane wait per batch from 1.8 to 0.7 ms. The per-stream backlog itself does not go away while the frontend is the bottleneck; it moves to where it delays only its own stream.

UCX transport instability

Measured on 2026-09-04, before core pinning: the velo mux over velo's UCX transport on InfiniBand (2 mlx5 adapters per node), against the same mux over TCP on 200G Ethernet. Three reps each, 512 workers, concurrency 8,192.

UCX did not beat TCP, and it did not hold its tail. The failure chain:

1. The KV router put about 5,760 concurrent streams on one of the eight mocker processes. This imbalance occurs with every transport.
2. All 64 workers in a process share one velo node and one mux peer link. All 1,722 failures were in that one process, spread evenly over its 64 workers.
3. Over UCX, that one link fell behind. On the hot process, the delay from worker completion to client completion grew to a median of 18.87 s. Over TCP, at the same or higher concurrency, it peaked at 1.0 to 2.75 s.
4. Heartbeats share the per-peer queue with data, so streams behind more than 15 s of backlog went silent at the frontend.
5. The reader-pump watchdog (3 × 5 s) killed 1,722 streams. Every log line showed anchor_frame_tx_len=0 and transport_rx_len=0. closed_slot drops were 1,722 × 255 exactly, so no record had arrived before each kill.
6. The worker was healthy the whole time. It completed all 27,702 of its requests.

The measured revision had no UCX backpressure edge beyond its shared ring:

• ucp_am_send_nbx never refuses. An exhausted endpoint returns a request pointer, not an error.
• inflight_ops counts posted operations but does not gate admission.
• The per-peer admission gate queues without a bound.
• All peers share one 1,024-entry ring into the progress thread. TCP gives each connection a 256-entry channel and a blocking write.

Version 0.14 bounds queued and outstanding sends per peer. A send permit is released on UCX completion or failure, and peer retirement cancels pending admission. The progress thread can still process control and endpoint cleanup when a data peer is full. This change requires a new saturation measurement; it does not change the historical results below.

Because admission never blocks, the on-admission flush policy never parks, and batches collapse. UCX carried 6.97 to 8.65 records per batch against 18 to 29 over TCP with the same settings. That is 2.6 to 4.1 times the message rate for the same records.

These alternatives were checked and refuted:

• Send-backpressure counters do not follow the failure. Their totals ran opposite to the instability across reps 2 and 3.
• The peer byte budget (8 MiB over 5,760 slots) does not depend on the transport. The TCP control carried 5,326 streams on the same budget and held p99 at 2.5 s.
• Batch collapse amplifies the failure but does not start it. The clean rep had fully collapsed batches too.
• UCX scheduling is not unfair. The failed peer carried eight times the load of the others.
• Spinning progress threads do not starve the CPU. There were 9 progress threads on 288 cores.

No tuning setting reaches this mechanism. The mux's stream watchdog, which replaced the reader pump's watchdog on single-sender streams, exempts a sender that holds no data credit, but here the senders held credit and their records were stuck upstream, so it would still have fired. The fix has two parts. The UCX transport needs a backpressure edge (gate admission on in-flight operations or on a per-peer ring share). Liveness needs a heartbeat path that data cannot block, or a watchdog that can tell a starved stream from a dead one. The UCX inbound path records frames like every other transport. Version 0.14 adds the bounded admission edge. Saturation measurements must still check whether data backlog delays stream heartbeats beyond the watchdog. RDMA performance covers the UCX transport outside the response plane.

Instrumentation cost

These instruments are on the hot path. Their costs are accepted without an A/B measurement:

• Egress writer instruments (velo_transport_egress_queue_wait_seconds, velo_transport_frames_written_total, velo_transport_write_duration_seconds, TCP, UDS and QUIC only). Per frame: one Instant::now() at send, then one elapsed() and one histogram observe at dequeue. Per write: a second observe and up to five counter compare-and-swap loops. If this cost shows in a measurement, observe once per write on the oldest frame instead of once per frame.
• Ordered-lane metrics for _stream_batch. Per batch: four Arc clones, one elapsed() with one histogram observe, and one gauge increment and decrement. This cost is the measurement of the lane wait itself, so no A/B applies.
• Batcher counters (velo_streaming_mux_records_sent_total, velo_streaming_mux_batcher_wakes_total). The collectors are pre-bound into arrays. Per record: one u16 increment. Per wake: one AtomicF64 compare-and-swap. velo_streaming_mux_staged_records already paid one gauge compare-and-swap per staged record.

The rule for new instruments on these paths: no label lookup per record. Pre-bind the collectors, and observe per batch where possible.

The instruments did not perturb the rig. Three instrumented reps and one control rep stayed inside the historical band. One outlier rep was a hot mocker process with 1,887 live slots, not the instruments.

Findings outside velo

• axum's graceful-shutdown watch costs 1.4 to 2.2 ms per request in both planes. It is one process-wide lock, polled again on every connection wake. It tracks the backlog draw, not the plane.
• Two tokio runtimes in the frontend. See item 1 of What produced the result. The fix belongs in Dynamo's Python bindings and helps every response plane.
• aiperf 0.10.0 can wait forever when a record references a dataset entry that fails to decode. It then writes no export. The rig patches it to finish after 15 report intervals with no progress, once all credits are complete and at least 95% of records are processed. The patch fired on runs that missed 2 to 397 of 250,000 records.
• Mixed model cards stop a mocker fleet silently. Packed workers with DYN_SYSTEM_PORT set publish a mix of self-hosted and fallback model cards. Dynamo's discovery controller parks them in a Conflict state with no log line, and the fleet never serves. Set DYN_SELF_HOST_METADATA=0 and leave DYN_SYSTEM_PORT unset.
• The socket buffer clamp. By default, velo requests 2 MiB socket buffers (see Socket buffers). This cluster's net.core.rmem_max and wmem_max are 212,992 bytes and cannot be raised, which clamps each buffer to about 208 KiB and disables autotuning. At 0.5 ms round-trip time, one connection carries at most about 400 MB/s. The rig's worst-case mux traffic is about 200 MB/s in total, so the clamp does not bind with 4 or more worker processes per node.

RDMA performance

This chapter records measured numbers for the rendezvous RDMA path, with the setup behind each number. For how the path works, see Rendezvous and RDMA. For the setup procedure, see Run rendezvous over RDMA.

Every two-node number here is an unaccelerated floor. The runs used rc_verbs on a plain verbs memory domain. At the time, velo's vendored UCX opened no mlx5 memory domain (see RDMA design). They are lower bounds on what the hardware does, not measurements of it. No run has used the accelerated rc_mlx5 lane yet. Re-derive rdma_min_bytes on that lane before you trust the 64 KiB default.

Two-node setup

UCX_PROTO_INFO=y on the live run named the device for every protocol range:

| 1476303..inf | (?) rendezvous zero-copy read from remote | rc_verbs/mlx5_2:1 50% on path0 and 50% on path1 |
|       0..107 | short                                     | rc_verbs/mlx5_2:1/path0                         |

Cold pair: one transfer per process pair

Each rep started a fresh owner process and a fresh consumer process, one on each node, with the rendezvous_rdma_two_proc example. There were 5 reps per cell. The time is the example's own velo.get() measurement: the acquire round trip, the transfer, and the copy out of registered memory. release is not included. The chunked baseline ran with VELO_RDMA_RENDEZVOUS_DISABLE=1.

• All 50 transfers matched byte for byte, including the size that is not granule-aligned. No corruption, truncation, or hang occurred.
• All 25 RDMA reps reported rdma/ok on both sides.
• Every RDMA number includes the one-time endpoint wireup. The flat 14 ms floor from 64 KiB to 1 MiB is that wireup, not the transfer.
• The raw results show a wall time near 30 s per rep. That was NFS negative-dentry caching on the shared home directory, which delayed the file exchange between the two processes. It is not a velo latency.

Warm pair: many transfers per process pair

The sweep harness (see How the sweep works) ran 10 transfers per size inside one process pair on the same two nodes. The table gives medians of reps 1 to 9. Rep 0 is excluded because it carries the one-time wireup.

• The first RDMA get() on a fresh peer pair took 14,145 µs. Later gets took about 108 µs.
• The first chunked transfer took 216 µs. It pays no comparable cost, because it uses the TCP control connection that already exists and never creates a UCX endpoint.
• All 100 sweep transfers took rdma/ok. All 100 transfers in the chunked run took chunked/kill_switch. Every payload matched.
• Warm RDMA is a fixed cost of about 105 µs plus a size term. The 1 MiB and 4 MiB points give about 9 GB/s marginal rate. Two points show the shape only and are not a bandwidth measurement.
• The rate is far below the 400 Gb/s link, because get() copies the payload out of registered memory into Bytes. get_pinned() and get_into() skip that copy. Neither was measured.

The fixed ~105 µs is the acquire round trip. Registration (next section) is not the bottleneck.

Threshold verdict for rdma_min_bytes

The default stays at 64 KiB. The threshold turned out not to be the lever.

1. Warm pair. RDMA beats chunked at every measured size, down to 4 KiB (1.4x). There is no crossover above 4 KiB on this fabric. The 64 KiB default is therefore conservative. It costs about 40 to 70 µs per transfer in the 4 to 64 KiB band. That cost is smaller than the extra pinned memory and lease traffic that small pinned slots add.
2. Cold pair. The 14 ms wireup reverses the result. Chunked wins by 41x at 64 KiB, 11x at 1 MiB, and 1.1x at 16 MiB. RDMA wins only at 256 MiB (3.4x). The cold crossover lies between 16 MiB and 256 MiB, near the low end. The two cells are 16x apart, so the data supports no tighter number.
3. The lever is the wireup. A lower threshold makes the cold case worse. A threshold inside the cold band discards the warm win, which is the common case for any pair that transfers more than once. UcxTransportBuilder::eager_endpoints exists because of this result. It moves the wireup to register(). It is off by default.

Two caveats limit the comparison:

• The chunked arm used the kill switch, and the kill switch also makes the owner stage in plain memory (pinned=false in every chunked line, pinned=true in every RDMA line). Staging mode therefore changes together with the path. The table answers "enable RDMA or not". It is not a clean sweep of rdma_min_bytes with pinned staging on both arms. The 4 KiB gap is about 40 µs either way, so the verdict likely holds.
• All numbers are from the rc_verbs floor. The warm latency and the 14 ms wireup can change on rc_mlx5.

Registration cost and raw GET latency

bench_rma ran two UCX workers in one process on one node. These are HCA-loopback numbers, not two-node numbers, and they do not compare with the tables above. The test header printed UCX_TLS=tcp, but the environment overrode it. The run used the same rc_verbs lane and port. Measured on 2026-08-29.

Registration costs 43 to 266 µs. It is not the bottleneck of a warm transfer, because the pool registers an arena once and many transfers use it. A registration per transfer adds this cost to every transfer, so pre-registered arenas stay the right design.

An earlier probe measured ucp_mem_map much higher on a different host: 161 to 805 µs for 4 KiB to 16 MiB, and 3.2 ms at 100 MiB. That probe ran on x86_64 (DGX B200, ConnectX-7 InfiniBand, MLNX OFED 25.07) with UCX 1.22.0 built on the node. The two results use different hosts, CPU architectures, driver stacks, and memory domains. They are not reconciled.

Packed rkey size

A packed rkey carries key material for each memory domain, and different domain types pack different amounts. The 20 B and 19 B results can therefore both be right. No run has measured a packed rkey under a DEVX domain in velo's own build. From the UCX source, each extra IB device adds 9 B.

The rkey_pack_canary test asserts at least 9 B, and the bound stays there. The 20 B value came from the verbs domain that the link-order defect forced. The value changes once the mlx5 domain opens.

Correctness on InfiniBand

On the same two nodes and lane:

• All 23 rendezvous_rdma integration tests passed, including a_malformed_descriptor_falls_back_to_chunked and a_failed_get_falls_back_chunked_exactly_once.
• get_cancelled_by_endpoint_replacement passed in 0.16 s. The caller got an answer, the region was released, and teardown balanced.
• These RMA tests also passed: rkey_pack_canary, preparse_accepts_real_packed_rkeys, map_get_roundtrip, get_zero_length, get_out_of_range, unmap_waits_for_inflight, shutdown_with_inflight_get, get_cancel_still_releases_the_region, peer_shutdown_during_get_answers_caller, and truncated_rkey_is_refused_before_ucx.
• Four teardown tests (map_get_roundtrip, unmap_waits_for_inflight, get_cancel_still_releases_the_region, peer_shutdown_during_get_answers_caller) log ucp_ep.c:2222 UCX ERROR ep ... has already been closed. Every assertion holds. This double close does not occur over TCP.
• unusable_rkey_is_refused_by_ucx_over_tcp aborted the process. See RDMA design.

Not measured: the abandon_rma_ops teardown path under a stopped owner, get_pinned() and get_into() throughput, concurrent transfers, and any soak on hardware.

Probe measurements that shaped the design

These numbers come from probes before the rendezvous code existed. They ran on x86_64 (DGX B200, ConnectX-7 InfiniBand, MLNX OFED 25.07) with UCX 1.22.0 and 1.19.0, pinned to mlx5_0:1. Measured on 2026-08-20.

• One-sided GET. A 1 MiB GET completed in 826 µs while the owner worker never ran its progress loop. The data matched. This held with UCP_ERR_HANDLING_MODE_PEER on rc_mlx5 with DEVX on, with UCX_IB_MLX5_DEVX=n, and on rc_verbs.
• Eager Active Messages. With UCP_AM_SEND_FLAG_EAGER on every send, the receiver saw eager data at every size up to 8 MiB and never a rendezvous receive. Without the flag, sends switched to rendezvous at 1 MiB and above. Eager 8 MiB reached 1.28 GB/s against 1.21 GB/s for rendezvous. The single-port loopback path capped at 1.43 GB/s, and UCX reached 83 to 98% of the raw-verbs ceiling on that path.
• Progress thread wakeup. ucp_worker_signal against a spinning poller took 0.83 µs at p50 and 1.28 µs at p99. Against a blocking poll() it took 7.3 µs at p50, with a 115 to 165 µs tail from CPU idle states. A bare eventfd shows the same tail, so the platform causes it, not UCX. One earlier aarch64 sample of 3.09 µs was a lucky blocking draw.
• UCX protocol boundaries for Active Messages on rc_mlx5, host memory, CX-7: short up to 2038 B (2030 B with the reply flag), bcopy 2039 to 8118 B, zero-copy 8119 to 8246 B, multi-fragment zero-copy up to 311,293 B, rendezvous from 311,294 B.

The spin-then-park loop of the UCX progress thread (UcxConfig::spin_us, 20 µs by default) follows from the wakeup numbers. A loaded submitter pushes onto the ring in about 100 ns and skips the signal. Only an idle worker pays the wakeup.

How the sweep works

The warm-pair table came from a measurement harness, rv_sweep. It is not a shipped example. It compiles against the current API as an example with required-features = ["ucx"], using only dependencies that examples/Cargo.toml already has. This section describes it so that it can be rebuilt.

Shape. It has the same two-process shape as rendezvous_rdma_two_proc. Each role builds a Velo with a TCP transport, a UCX transport added through add_ucx_transport, a Prometheus registry, and RdmaConfig { rendezvous: RdmaRendezvousConfig { rdma_min_bytes, .. }, .. }.

Flags. --role owner|consumer, --dir <shared directory>, --sizes <comma-separated bytes> (default 4096 to 1048576 in powers of two), --reps <n> (default 10), and --min-bytes <n> (default 65536, applied to both roles). The 4 MiB row needs a --sizes list longer than the default.

Owner.

1. For each size, stage --reps separate slots with register_data_pinned. A slot is single-use, because release drops its only reference. One slot per rep lets the pair measure a warm endpoint instead of a new wireup each time.
2. Record metadata(handle).pinned for each slot, and print rdma_registered_bytes().
3. Write owner.json: the owner's PeerInfo and a list of (size, handle as u128 decimal, pinned).
4. Wait for consumer.json, register the consumer as a peer, and wait for a done file.
5. Call graceful_shutdown(ShutdownPolicy::Timeout(30 s)).

Consumer.

1. Read owner.json, build the instance, register the owner, and write consumer.json.
2. Wait up to 120 s for the owner's _rv_acquire handler with wait_for_handler.
3. For each slot: read velo_rendezvous_rdma_path_total{reason="ok"}, time velo.get(handle), call release, and read the counter again. An increase means the RDMA path ran.
4. Verify the length and every byte against the pattern.
5. Print SWEEP size=<n> rep=<n> pinned=<bool> path=<rdma|chunked> get_us=<n>.
6. Print a summary of every non-zero path_total label, write done, and shut down.

Payload. Byte i is (i * 31 + (i >> 8)) as u8. The pattern does not repeat at the chunk size, so a chunk written at the wrong offset fails the comparison.

File exchange. Each card is written to a temporary name and renamed, which is atomic in one directory. The reader polls every 25 ms. Before each read, it calls read_dir on the parent directory. On NFS, a plain read of a name that the client cached as absent does not revalidate, and read_dir forces it. The rendezvous_rdma_two_proc example does not have this step.

Arms. The RDMA arm runs as is. The chunked arm in the table ran the same binary with VELO_RDMA_RENDEZVOUS_DISABLE=1.

Two changes make the next run cleaner:

• For a clean threshold sweep, run the chunked arm with --min-bytes above the largest size instead of the kill switch. Both arms then stage in pinned memory, and the owner answers below_min. Only the path differs.
• Add a get_pinned arm. get() includes a copy out of registered memory, and that copy dominates at large sizes.

Run it.

1. Build: cargo build --manifest-path examples/Cargo.toml --release --features ucx --example rv_sweep.
2. Allocate two --exclusive nodes on one subnet manager, each with an active InfiniBand port. Verify the port with ibv_devinfo.
3. Start the owner, then the consumer, with the same --dir and the same UCX_NET_DEVICES and UCX_TLS on both.
4. Verify the lane with UCX_PROTO_INFO=y, as in Verify the lane.
5. Take medians over reps 1 and later.

QUIC performance

This chapter records what the QUIC transport costs against TCP, the settings that change it, and one quinn defect that the transport works around. For the design, see Transports.

Settings

The examples read VELO_QUIC_LANES, VELO_QUIC_MAX_MTU, VELO_QUIC_STREAM_WINDOW, and VELO_QUIC_SERVER_ENDPOINTS, so a sweep can change them without new flags.

Packets above 6550 bytes are lost

quinn gives up to 10 packets to one GSO send, and it does not limit the size of the batch. A UDP datagram holds at most 65507 bytes. When a packet is larger than 6550 bytes, a full batch is too large, and the kernel returns EMSGSIZE. quinn-udp 0.5 treats EMSGSIZE as success, because it expects that error only from MTU probes. So the whole batch is lost with no error, and each flight waits for a loss-probe timeout.

Measured on loopback with 64 KiB messages, one at a time:

The transport lowers max_mtu to 6550 and logs a warning. The test large_frames_round_trip_at_a_jumbo_mtu requests 8952 and fails if 200 round trips of 64 KiB take 2 s or more. Without the limit they take about 9 s.

Loopback measurements

Measured on 2026-09-23 on one aarch64 workstation (20 cores, shared, load average near 8), over loopback, with the throughput example and 5,000 messages for each cell. Two reps for each configuration. These numbers show direction only. For a network between two nodes, see Two nodes. For the response plane on a cluster, see Response plane on a cluster.

• Small messages cost about twice the latency of TCP. Each direction passes through quinn's connection and endpoint driver tasks, so one message takes more task hops than on TCP.
• Large messages reach about 600 to 800 MiB/s on one connection. One quinn task does the packet work and the encryption for a connection. TCP moves 64 KiB segments on loopback, and the kernel does that work.
• max_mtu 6550 gives 25% to 30% more for large messages when the path carries it. It does not help small messages.

Settings that did not help

• A larger initial congestion window. 1 MiB instead of quinn's 14,720 bytes made no difference, at the default MTU or at 6550.
• Larger flow-control windows. A 16 MiB stream window with a 64 MiB send window made no difference to one-at-a-time traffic, and was slower with 64 in flight (9,300 to 9,600 against 10,900 to 11,200 msg/s).
• Charging Tokio's task budget per batch of frames. Dynamo's QUIC plane needed this because it polled quinn once per frame header and once per payload. The velo reader uses FramedRead, which decodes all buffered frames before it polls quinn again, and routes frames through flume, which does not charge the budget.
• LSE atomics. Building with -C target-feature=+lse changed neither transport by more than 5%, on loopback on a 144-core Grace node.

Two nodes

Measured on 2026-09-29 with the throughput example in its two-host mode. The server ran on one node and the client on another. Each node has 144 Grace aarch64 cores, and the process had 16 of them. The nodes connect through 200G Ethernet at MTU 1500, with net.core.rmem_max at 212,992. Each cell sent 20,000 messages. Three reps, with TCP and QUIC interleaved.

• Small messages cost about 25 µs more per round trip than on TCP, as on loopback. The network does not change this cost: it comes from the task hops inside quinn. With many messages in flight the gap closes, and pipelined QUIC matches TCP.
• One QUIC connection tops out at about 0.8 GB/s, the same ceiling as on loopback on the same node type. So the limit is the CPU of the connection, not the network. A profile of the 64 KiB case put about 15% of the samples in AES-GCM. The rest was quinn's packet work, the kernel's UDP path, and task wakeups. TCP moves about 3.9 GB/s, because the kernel and the NIC offloads share that work.
• A peer that needs more than about 0.8 GB/s needs more than one connection; see Lanes. The Dynamo response plane on the cluster carries far less than that per connection (see below).

Lanes

With lanes(n), each peer gets up to n connections, one for each lane used, each dialed from its own UDP socket. Each lane has its own quinn endpoint driver, shared by every peer dialed on that lane, and each connection has its own connection driver. The work runs on up to n cores.

The messenger sends its own traffic on lane 0, so lanes(n) does not change the throughput of ordinary messages. The batched-streaming mux places streams over the lanes (see Lanes), so streams do get faster. A caller can also send on a lane with send_message_on_lane.

Measured on 2026-09-29 across the same two nodes as above, 64 KiB messages pipelined, 20,000 messages per cell, two reps. A prototype, which is not in the tree, spread the messages round robin over the lanes, with one connection per lane, each from its own UDP socket. The receiving node had 32 server sockets in one SO_REUSEPORT group.

The UDP receive-buffer error count stayed below 50 in each run, so the gain comes from more cores, not from more socket buffers. 64 B pipelined messages do not change with the lane count.

Lanes on one server socket share its endpoint driver. So each server socket has its own port, and a dialer sends lane k to socket (offset + k) % n, which spreads its lanes evenly. The prototype instead put all server sockets on one port in a SO_REUSEPORT group, where the kernel hashes each connection to a socket at random, so some sockets carried three or four lanes and others none. The table below shows what that cost with 8 lanes. It ran in a separate job on another pair of nodes, so its 32-socket row differs from the table above by run-to-run spread.

With one port for each socket, the lanes of one dialer are placed evenly, so fewer sockets do. Measured on 2026-09-29 across two nodes, 64 KiB messages pipelined, 20,000 messages per cell, two reps, with messages spread round robin over the lanes by a measurement build:

With 8 lanes, 4 sockets carry two lanes each and reach the same rate as 8 sockets. One lane is unchanged.

A fixed bind_addr port P binds ports P to P + n - 1, one for each server socket. Peers spread over all of them, ordinary traffic included, so open all of them in a firewall. Transports on one host need fixed ports at least n apart.

During a rolling upgrade, an old peer can dial a new node, and a new peer can dial an old node. A peer without per-socket ports dials only the first port of a new node, so until it upgrades it shares that socket with the other old peers.

A TCP prototype with the same layout, one connection per lane, scaled too: 2.5, 4.6, 8.1 and 13.1 GB/s (10^9 bytes per second) with 1, 2, 4 and 8 lanes. One TCP connection is limited by its receiver: one reader task does the recvmsg copy, and by default the transport's fixed socket buffers cap the TCP window (see Socket buffers). TcpTransportBuilder::lanes(n) builds the same layout. See TCP lanes. These TCP numbers came from the prototype, not from that code.

Streams over lanes

Measured on 2026-09-30 across two nodes of the same type as above, with the throughput example in its stream mode (--modes stream). The client creates the anchors, and the server attaches to them and streams to them, so the client is the consumer and places the streams on lanes. Both sides set VELO_QUIC_LANES to the lane count and keep the default of 4 server sockets. Each cell sends 200,000 items in total, split evenly over the streams, after a warm-up round. Two reps. The TCP row is one connection in the same job, for reference.

16 KiB items, MiB/s:

64 B items, items per second:

• Streams scale with lanes as messages do. One lane stays at the one-connection limit, 500 to 780 MiB/s. Each doubling of lanes nearly doubles the rate, up to about 5,300 MiB/s at 8 lanes, which is 2.5 times TCP on one connection.
• 16 streams do not fill 8 lanes. With two streams on each lane, 8 lanes reach about 3,700 MiB/s. Each stream waits for credit every 32 records, which limits the rate of one stream. This was not profiled.
• Small items lose a little at few streams. 16 streams of 64 B are about 9% slower at 8 lanes than at 1. The likely cause is that each lane's batch then carries the records of only two streams. This was not profiled. At 256 streams, 8 lanes move twice as many items as 1.
• The streams used the lanes. One lane carries at most about 790 MiB/s, so 5,300 MiB/s at 8 lanes needs data on at least 7 lanes. The lane tests check the placement directly.

A second job on another pair of nodes gave 533 to 735 MiB/s at 1 lane and 3,721 to 5,150 MiB/s at 8 lanes, so the spread between node pairs is larger than between reps.

Later runs on three more pairs of nodes gave 3,237 to 5,402 MiB/s at 8 lanes and 64 streams (median 5,198, 11 reps), and 3,655 to 5,326 MiB/s at 256 streams (median 4,948). The first rep of a job is often the low one. A diagnostic build counted the bytes that each lane's batch handler received: over 8 lanes, the busiest lane carried at most 1.09 times the mean, at 64 and at 256 streams.

The same stream mode over TCP lanes (VELO_TCP_LANES), on one pair of nodes, 2 reps, 16 KiB items, MiB/s:

With 64 B items at 256 streams, TCP moved 783,000 to 857,000 items/s at 1 lane and 2,120,000 to 2,219,000 at 8 lanes. At 16 streams, 8 lanes were slower than 4, for both item sizes.

Batched streaming over QUIC

Measured on the same workstation with response_plane_bench (2 anchor hosts, 64 engines, 2,000 requests, 200 warm-up requests, --flush-policy auto), three reps.

In the two quieter reps, QUIC matched TCP on throughput and first-token latency (11,300 and 11,000 against 11,600 and 11,000 requests/s). It used 15% to 30% more CPU for each request. The mux batches many token records into each frame, so the per-message cost of QUIC matters less here than in the one-at-a-time tests. max_mtu 6550 was slower in all three reps with these small frames. Use it only for traffic with large messages.

Response plane on a cluster

Measured on 2026-09-23 with Dynamo's response plane on two nodes. Each node has 144 Grace aarch64 cores. The nodes connect through 200G Ethernet at MTU 1500. One node runs the frontend, etcd, NATS, and the load generator. The frontend has cores 0-71 and the load generator has cores 72-143. The other node runs 512 mock workers in 8 processes. Each run sends 250,000 requests at concurrency 8,192, with 1,024 input tokens and 256 output tokens. Each configuration ran three times, and the runs of the four configurations alternated.

The hosts have net.core.rmem_max and net.core.wmem_max at 212,992. The transport requests 8 MiB and 4 MiB for each socket, and the kernel gives 425,984 bytes. The transport logs one receive-buffer and one send-buffer clamp warning for each socket. The frontend used 32 server endpoints. The workers used the default of 4.

The configurations:

• Velo, TCP. The messenger mux over the TCP transport. The frontend mints the stream terms at registration (zero-RTT attach), and batches write when the transport admits them.
• Velo, QUIC. The same configuration over the QUIC transport.
• Multiplexed TCP. A multiplexed TCP response plane that does not use velo, for reference.
• Dynamo QUIC. Dynamo's own QUIC response plane on quinn 0.11.12, with 32 server endpoints and no batch interval.

A mock-worker process that holds a share of the 8,192 open requests answers late. Throughput and first-token latency change with the number of these processes in a run. The table therefore compares runs with the same number, shown as "Holders". Steady state is the requests that started 10 s or more after the first request. ITL is the mean inter-token latency of each request. CPU is frontend CPU time for each request. Drops is the RcvbufErrors count of the frontend host for the run.

TTFT p50 and p99 are steady-state values. One run of Velo over TCP had 7 holders and is not in the table. No run had client errors.

• QUIC shows no cost for velo here. At the same number of holders, velo over QUIC and velo over TCP agree on throughput, latency, and CPU. The differences are 1% to 5%, and one or two runs for each cell cannot resolve them. On loopback, QUIC used 15% to 30% more CPU for each request. On the cluster, most of the frontend CPU is outside the transport, and the mux puts many records in each frame.
• The clamped buffer loses about 0.1% of datagrams. With 32 endpoints, the velo frontend received 4.7 to 5.3 million datagrams in each run and dropped about 0.1% of them. The drops did not show in the tail latency.
• Velo over QUIC is ahead of Dynamo QUIC. With 2 holders, velo moved 2,742 requests/s against 2,328 to 2,496. It also had a lower first-token p99, a lower ITL p99, and 6% to 9% less CPU. The Dynamo frontend dropped 16 to 18 times as many datagrams with 2 holders, and 27 to 41 times as many with 1 holder. It received about 0.7 datagrams for each output token, against 0.07 to 0.08 for velo. With the batch interval at 0, Dynamo sends close to one packet for each token record. The velo mux puts many records in each frame. Dynamo's default interval of 5 ms was not measured.
• max_mtu 6550 was not measured. The path MTU of the cluster is 1500, so a larger max_mtu has no effect.

Workspace crates

The workspace has two runtime crates, two hub crates, and one FFI crate.

Why only two crates

There used to be nine crates: velo-messenger, velo-transports, velo-streaming, and more. Each had its own version. A 0.1.1 patch of one crate changed its dependency on another internal crate from 0.1 to 0.2. Cargo took 0.1.1 as compatible with 0.1.0, so downstream lockfiles got two copies of the messenger types. The result was a flood of E0277 and E0308 errors in downstream builds.

The fix was structural. All runtime code moved into lib/velo/src/. Do not add a new crate under lib/.

ucx-rs is one exception. It is a leaf FFI crate. It exports no types that velo or velo-ext share, so it cannot cause two copies of a shared type. It must be on crates.io before a velo release with the ucx feature can ship.

velo-hub and velo-hub-ext are the other exceptions. They do share types with velo and with each other, so they take the workspace version and require velo and each other with a caret on that version. See Versioning. A separate trait crate for hub extensions was rejected: an extension gets a Velo and returns axum routes, so that crate would still depend on velo and axum, and it would be one more crate that can cause two copies of a type.

What goes in velo-ext

• The traits: Transport, FrameTransport, PeerDiscovery, ServiceDiscovery, TransportObservability.
• The types in their signatures: WorkerId, InstanceId, PeerInfo, WorkerAddress, TransportKey, MessageType, TransportError, SendOutcome, ShutdownState, Direction, and similar.
• The channel types in their signatures: TransportAdapter, DataStreams, make_channels, AdmissionGate.

These stay out of velo-ext:

• Concrete transports and discovery backends.
• prometheus, tonic, axum, and other heavy dependencies.
• The runtime types: Messenger, Velo, AnchorManager, RendezvousManager.

The test for the boundary: cargo tree -p velo-ext | grep -c prometheus must print 0.

Module map of velo

Versioning

velo and velo-ext are published. Out-of-tree code compiles against velo-ext, so every change to it reaches every downstream implementation. The rules below are the structural fix for the bug class in Workspace crates. They are not optional.

Rules

1. Only velo and velo-ext publish. A new workspace crate gets publish = false. ucx-rs, velo-hub, and velo-hub-ext are the documented exceptions. velo-hub has publish = false until a real port of kvbm's hub passes kvbm's lifecycle tests against it, and velo-hub-ext publishes together with it.
2. velo-ext has an exact pin. [workspace.dependencies] has velo-ext = { path = "lib/velo-ext", version = "=X.Y.Z" }. A caret requirement lets a later "compatible" patch change downstream lockfiles without notice. Do not change = to a caret.
3. When velo-ext moves past its latest published version, velo must move past its own. Update the pin to match.
4. A new trait method in velo-ext needs a default implementation. A method with no default breaks every external implementation. A defaulted addition is a patch release (0.5.0 to 0.5.1). Before 1.0, Cargo reads 0.y as the breaking position.
5. A change to a signature, bound, or parameter of a trait method is breaking. Before 1.0, that is a minor bump (0.5 to 0.6), and velo must carry a bump over its latest published version too.
6. Removing a public item from velo-ext is breaking.
7. A bump is measured against the latest published version. Only versions on crates.io take part in Cargo's resolution. So main must carry a large enough bump over the latest published version for everything merged since that publish. Several breaking changes between two publishes share one bump. For example, if crates.io has 0.17.0 and main is at 0.18.0, a breaking change stays at 0.18.0. After the next publish, the first breaking change bumps again.

velo-hub

velo-hub-ext is the wire protocol of the hub, on velo-ext alone. velo-hub re-exports it as velo_hub::protocol, so its types are part of the velo-hub API. The rules below cover both crates.

The API of velo-hub uses velo types (Velo, PeerInfo, axum Router). A move of velo-hub from velo 0.21 to velo 0.22 changes these types, so it is a breaking change of velo-hub, but its own source does not change. cargo semver-checks cannot see that. These rules make it safe:

1. velo-hub and velo-hub-ext use version.workspace = true. A velo minor bump is always a velo-hub and velo-hub-ext minor bump, and velo-hub X.Y works with velo X.Y.
2. [workspace.dependencies] requires velo with a caret on the workspace version: velo = { path = "lib/velo", version = "0.22.0", default-features = false }. Before 1.0, a caret on 0.22.0 means >=0.22.0, <0.23.0, so it never admits a second minor. An exact = pin would also stop a velo patch from reaching users of velo-hub. velo-hub-ext has the same kind of line: velo-hub-ext = { path = "lib/velo-hub-ext", version = "0.22.0" }.
3. scripts/check-semver.sh fails if the major.minor of either requirement is not the workspace version. This check runs before the semver:skip escape and before the script stops for a change with no crate changes.
4. velo, velo-hub, and velo-hub-ext publish together, from one commit.
5. velo-hub re-exports velo, axum, and velo-hub-ext (as velo_hub::protocol). Hub crates downstream use these re-exports, and require velo-hub with a caret. CI also checks that velo-hub-ext depends on no velo runtime and no prometheus.

The CI gate

The Semver Check job runs scripts/check-semver.sh. For each crate that the change touches, the script reads the latest version of the crate from the crates.io index and runs cargo semver-checks check-release against it. It fails when a breaking change is not covered by the version bump over that published version. It skips a crate that was never published (the index answers 404). Any other answer from the index fails the gate: no connection, an error status, a page that is not an index entry, or a crate whose every version is yanked or a pre-release. The gate also fails a version below the latest published one, and a version with a pre-release suffix, even without a breaking change. Build metadata, such as the +ucx.1.22.0 of ucx-rs, is ignored. A change to the root Cargo.toml also selects every crate that takes its version from it. The check is cumulative, so a breaking change that reached main without a bump fails the next change that touches that crate. The semver:skip PR label exists for emergencies. Use it only when a reviewer agrees.

To run the gate locally:

bash scripts/check-semver.sh

Inside the workspace

While all callers of an item are in the workspace, the compiler finds them all. Change the interface, change the callers, and delete the old form. Do not keep a second entry point, a forwarding wrapper, or an alias for an old name.

Testing

Commands

cargo build --all-features
cargo test --all-features --all-targets
cargo clippy --all-features --no-deps --all-targets -- -D warnings
cargo fmt --check
cargo machete
bash scripts/check-semver.sh

Always use --all-features. Without it, the code behind zmq, grpc, quic, nats-*, etcd, ucx, and simulation does not compile, and its tests do not run.

Also test the core configuration. An all-features build can hide a missing dependency feature in a smaller build:

timeout 900 cargo test --locked -p velo --no-default-features --features test-helpers --all-targets
UCX_TLS=tcp timeout 900 cargo test --locked -p velo --no-default-features --features ucx,test-helpers --all-targets

The bytes serde feature is enabled explicitly for tests. Tests must not rely on an unrelated transport to enable it.

Run one integration test by its name. The name is <module>_<file>. For example, tests/transports/zmq_integration.rs is transports_zmq:

cargo test --features zmq --test transports_zmq

Put a time limit on each run

Run each suite under timeout: timeout 900 for the full workspace, and timeout 300 for one target. A test that hangs stops the whole runner, and it does not fail. The tests most likely to hang are those that change timeout and deadline code.

If the run exits with 124, suspect the newest test or the newest change. Run that test binary again with --test-threads=1 --nocapture and a short timeout to find the test that hangs.

Services and system packages

The ucx feature builds UCX with InfiniBand support. UCX configure only warns when the rdma-core headers are missing, and then it builds a TCP-only UCX. ucx-rs stops the build instead, so a TCP-only build can never ship by mistake. No RDMA hardware is needed for the tests. They run with UCX_TLS=tcp.

CI links with mold (RUSTFLAGS="-C linker=clang -C link-arg=-fuse-ld=mold"). With all features and the bundled libzmq, the default ld can run out of memory on CI runners.

CI jobs

Rules for tests

• Write the test first. A claim about a fault is proved by a test that fails for that fault. The same test then proves the fix.
• Prove the precondition. Do not wait for it. A test that sleeps and hopes that a state is reached fails under load and under coverage instrumentation. Wait on an observable signal, such as a counter or a hook, that shows the state.
• Put the hard lessons in tests. A comment does not stop a regression. A test does. The doc comment of the test can be long, because it carries the reason forward.
• Count a cost, not only a result. A performance fault passes every functional test. Count the allocations, timer arms, tasks or writes of a hot path, and fail the test when the count grows. See Review for performance.
• Keep an ignored test honest. If a failing test must land before its fix, mark it #[ignore = "<finding>: <why it fails>"]. An ignored test enforces nothing. Removing the ignore is the first step of the fix.

Known flaky tests

peer_batcher::tests::flush_policy::kicks_during_an_admission_park_neither_double_send_nor_reorder sometimes fails with left: 3.0, right: 2.0. The test assumes the timing of a full admission gate. It fails more often under coverage instrumentation. If a PR changes no file under lib/velo/src/streaming/ and this is the only failure, it is this flake. CI runs only on a new commit, so push a new commit to run it again.

Review for performance

A functional test cannot see a performance fault. The message still arrives, only later or at more CPU. This chapter says where velo pays its costs, which costs have shipped before, how to hold a cost in a test, and how to compare two commits on one machine.

The perf-review skill in .claude/skills/perf-review/ runs this review with agents. Run it before a PR that changes code on a hot path. thermonuclear-review runs the same regression check when a change touches a hot path.

Where the costs are paid

Rank a cost by how often it is paid, not by where the code sits.

The code site can mislead. Socket buffers are set once per connection, but when they were set after the first data, the TCP window collapsed for the life of the connection, and throughput fell 100 to 350 times. That is a per-byte cost. See Socket buffers sized after data was in flight.

For scale:

• One 1 KB message over TCP on loopback takes about 11.4 µs one way. Two cross-thread wakes are 84% of what the transport adds. See Where the per-message time goes.
• One mux record costs 5 to 9 µs of frontend CPU. See The reader pump leaves the mux data path.

So 100 ns per message is about 1% of either budget.

Costs that shipped

Each of these costs reached velo, and each fix came with a test. A change that brings one back fails its test, unless the change goes around the place that the test counts.

Some changes were measured and did not help. Read these lists before you propose an optimization:

• What is not a lever
• Settings that are not levers
• Settings that did not help

Hold a cost in a test

A comment does not stop a regression. Count the cost in a test, and make the count fail when the cost comes back.

• Allocations. crate::test_alloc::allocations_in(|| ...) counts the allocations of a closure. It works only in lib/velo unit tests, because it is that test binary's global allocator, and it counts only the calling thread.

• Builds of a future or a timer. Route every build through one function that bumps a counter in tests and compiles to nothing in release:

  #![allow(unused)]
  fn main() {
  #[cfg(test)]
  fn note_teardown_arm() {
      let _ = TEARDOWN_ARMS.try_with(|n| n.fetch_add(1, Ordering::Relaxed));
  }
  #[cfg(not(test))]
  #[inline(always)]
  fn note_teardown_arm() {}
  }

  The test counts only builds that go through the seam. New code that builds the same future somewhere else escapes the test, so a review must check for that. To prove a bypass, add the seam call back at the new site in a scratch copy and run the test: if it fails there, the change went around it.

• Tasks. Compare Handle::current().metrics().num_alive_tasks() with a baseline after the work ends.

• Writes and batches. Use the recording writer in transports/coalesce/tests.rs, or the ratios from observability::test_helpers::MetricSnapshot end to end.

Prove each test before you trust it: revert the fix, and check that the test fails. See rule 11 in Measurement rules.

Compare two commits on one machine

1. Build each arm in its own worktree and target directory. Use the same features for both. Do not share one CARGO_TARGET_DIR between two worktrees of this repo: cargo gives velo's build the same name in both and decides freshness by file times, so one arm can run the other arm's code with no warning. Check that each binary names its own worktree's source paths (strings <binary> | grep <worktree>).
2. Count before you time. A count of allocations, timer arms, tasks or writes is a fact on any machine. Next best is perf stat -e instructions:u divided by the work done. Wall time comes last.
3. Run a harness that reaches the changed code. tx_budget attributes the TCP and UDS send path layer by layer. throughput and ping_pong cover the messenger. response_plane_bench, batched_streaming and stream_lifecycle cover the mux, and they are the only harnesses that install metrics. See Benchmarking.
4. Interleave the arms. Pin both to the same cores, run at least five reps of each, and change which arm goes first.
5. Call a difference real only when the ranges do not overlap. Report latency beside throughput.
6. Check the memory kept per operation. Run each arm at N, 4N and 16N operations, at the same concurrency. Divide the RSS difference between N and 4N by 3N, and check that 4N to 16N grows at the same rate. Growth is not yet a leak. Tasks of streams that ended can pile up in a loop that never yields, and glibc keeps freed pages. Read num_alive_tasks() and RSS after malloc_trim(0) before you call it a leak. In anchor_setup_bench, two tasks per stream grew RSS to 24 GB at 16M lifecycles (a measured peak). In a run of 512k lifecycles, malloc_trim(0) after the run took RSS from 777 MiB to 6 MiB.
7. Label the result. A number from a shared or dev machine is directional. A number for this book needs an exclusive node and the Measurement rules.

Documentation

This book is the documentation of Velo. The source is in docs/src/. A GitHub Actions workflow builds it and publishes it to GitHub Pages on each push to main.

Where text goes

Rules

• agent-docs/ never merges to main. Use it for notes while work is in progress. Before the PR merges, move the lasting parts into this book and delete the directory from the branch.
• Keep only what lasts. Write how it works, why, the numbers with their setup, and the results that did not work. Do not copy plans, phase or workstream labels, PR numbers, or handoff notes. History is in git.
• Write in simple English. Use the simple-english skill for the book, README files, and comments.
• Keep comments short. A comment says why, or which fault the code avoids. It does not repeat the code.
• Update the book in the same PR. Each PR that changes behavior updates the chapters that describe it and the top-level README.md.

Build the book locally

Install the tools at the versions that CI uses:

cargo install --locked mdbook --version 0.4.52
cargo install --locked mdbook-mermaid --version 0.16.2
cargo install --locked mdbook-linkcheck --version 0.7.7

Build the book and check its links. This is the command that CI runs:

bash scripts/build-book.sh

A broken internal link fails the build. To preview while you edit, serve the book:

mdbook serve docs --open

Batched streaming design

This chapter is the design record for the messenger mux. It lists the rulings that constrain future changes, each with the alternative that was rejected and what that alternative costs. It also records the validation method, the failures found and their mechanisms, and the hazards that remain open. Batched streaming describes how the mux works. Read that chapter first.

Validation methodology

At two nodes and ten streams on loopback, the mux is the same speed as the per-stream path or slightly slower. The gain is proportional to X/Y (streams per peer) and appears only at scale. The method therefore uses independent lines of evidence, ordered by how few assumptions each needs.

V1: resource-ceiling arithmetic

The per-stream path uses one socket per remote stream: one file descriptor and about 2 MiB of requested socket buffer in each process, and 4 tasks across the two ends. At the default ulimit -n of 1,024, a process stops below 1,024 concurrent remote streams, less its other descriptors. This is a hard wall, not a slope. A small-scale null result cannot refute it, which is why it comes first. The intended test opens N streams and counts /proc/self/fd. That test is not built.

V2: analytical cost model

Let s be the syscall and TCP cost per write_all, e the encode cost, f the receive-side decode and handoff, w a channel-handoff wake and p the per-record framing cost:

per-stream:  X · (e + s + w_connect + w_recv + f)
mux:         X · (e + f + p) + Y · (s + w)

The model makes three predictions that a measurement can refute:

• Syscalls and memory improve for any X/Y above 1.
• CPU crosses over where X/Y exceeds (s + w) / (s + w_connect + w_recv - p). Because p (about 20 ns) is small next to s (1 to 3 µs), that ratio is near 1. The prediction is that the mux costs less CPU for X/Y above 2, and saves more than 20% of streaming CPU above X/Y of 8.
• Latency gets slightly worse, by one extra channel hop of 1 to 5 µs. Report this term beside every throughput number.

V3: microbenchmarks

The planned Criterion targets measure the model constants directly:

• encode_frame for one payload against one batch of N, for e, p and the 64 KiB coalescing cliff.
• Decode of N frames at once against N calls, for f.
• A channel round trip, for w.
• Loopback N writes against one batched write, for s.

These benchmarks are not built.

V4: deterministic simulation

SimFabric charges base_latency per transfer, so it captures the packing gain deterministically. Streaming over it needs a SimFrameTransport and a StreamConfig::Sim, which do not exist. The simulation models only the network term, and it favors the mux there. It models no syscalls, no scheduling, no socket memory and no descriptor exhaustion, which is where most of the real gain is. Its larger value is as a deterministic correctness harness for credit accounting, slot generations and connection loss under adversarial interleavings.

V5: loopback scale sweep

The plan sweeps X over {1, 8, 64, 256, 1024, 4096} and Y over {1, 2, 8, 32} at about 64-byte frames, with the same binary on both paths. It records wall time, CPU, peak RSS, peak descriptor count, the batching ratio and p50 and p99 per-frame latency. Loopback removes wire time, which favors the mux, so the sweep needs a two-host repeat. A variant pinned to 2 cores shows the task and syscall gain under CPU scarcity, which is the normal state of an inference server. response_plane_bench covers part of this sweep. See Benchmarking.

V6: falsification criteria

If one of these criteria holds, the design stops or changes:

• The batching ratio does not improve at X/Y of 64. Then sends are not clustered in time and the premise is wrong. This is the cheapest and most decisive test. The per-stream path measures 1.00 for the forward-pass shape, and the mux measures 2.19 to 5.38 in batched_streaming at a credit window of 256.
• p99 per-frame latency at X/Y of 1 is more than twice the per-stream figure. Then a deployment with that shape sets enabled: false. This criterion does not decide the default (see below).
• CPU does not fall at X/Y of 64. Then encode dominates, and the fix is a cheaper codec. On the external serving rig, the mux cost 8.09 ms of frontend CPU per request against 10.48 for one connection per request.
• Measurable head-of-line blocking under mixed fast and slow consumers. Then the credit design failed.
• velo_streaming_heartbeat_watchdog_firings_total rises after heartbeat consolidation. Then the consolidation lost real failure detection.
• Throughput improves because added latency lets the consumer catch up. This is the trap. Always report latency beside throughput, and watch time to first token. Any windowed or hinted flush policy can make it worse, which is why the default flush does not wait.

The mux is on by default. The latency criterion above does not decide the default, and it has not been measured since the default changed. What decides it is a failure of the per-stream path: on a cluster, it failed 80% to 92% of requests at about 2,500 new streams per second, because the workers could not get a local port. See Batched streaming. A deployment with one stream per peer that measures a latency cost sets enabled: false.

The change of default has these effects that an operator can see:

• A consumer with the mux mints zero-RTT tickets, and a producer without the mux cannot open them. Upgrade producers before the consumers that mint tickets.
• velo_streaming_producer_send_backpressure_total changes meaning. See Saturation.
• A producer that runs the slot byte budget ahead of its consumer waits in send. The stream is not killed. See Saturation.

Rulings

Ride the Messenger, not a connection per peer

The mux packs records into active messages on the Messenger's existing connections. A dedicated mux transport with its own connections needs four more parts:

• A handshake magic value to learn the sender's identity.
• A fan of connections per peer for encode parallelism.
• A four-task connection lifecycle.
• A solution for acceptor identity.

The Messenger envelope already carries the sender, and one batcher per peer spreads encode work.

The price is that streaming shares queues, framing and backpressure with control traffic. Order is an explicit protocol obligation, so every record carries a per-slot frame_seq. The 13-byte record header pays for this. Measure before you shrink it.

Ordered dispatch, not a reorder window

_stream_batch uses ordered per-sender dispatch. The deprecated VeloFrameTransport put a 4,096-deep reorder buffer over a dispatcher that spawned a task per inbound message. Under cross-stream contention the window overflowed and deadlocked the consumer. That transport is deleted.

Do not shard the lane or reorder batches from one peer. OpenSlot travels in its own batch, and a Data batch that overtakes it truncates the head of the stream silently. Per-slot frame_seq is enough for Data only after OpenSlot lands.

Credit against the mux buffer, never against the anchor channel

The anchor's frame_tx has writers other than the mux: the same-worker attach path, detach and finalize, the watchdog's injection and MPSC senders. A proof of "C credits against a C-deep channel" fails with a second writer. The mux therefore owns a C+1 buffer and a reader pump moves records into frame_tx.

This ruling keeps the reader pump. Merging the pump into the anchor channel removes one task hop per record (about 1 ms of frontend CPU per request on the serving rig), but it breaks the credit proof. Removing the pump for mux streams also removes the heartbeat watchdog and velo_streaming_reader_pump_backpressure_total.

Addendum, 2026-09-26: the pump is off the mux data path, by another route. The ruling against the merge stands, and so does its reason. What changed is who reads the mux buffer. The StreamAnchor now polls the slot buffer itself, ahead of its own frame_tx, and counts each record it takes on the slot's DrainSignal (lib/velo/src/streaming/control/feed.rs). The mux is still the buffer's only writer, so the credit proof is unchanged. The anchor channel carries only sentinels and the heartbeat that tells a parked consumer a new feed is installed.

The watchdog is not lost. A stream watchdog, one task per mux stream that holds no data, keeps the pump's other duties: it reaps an unclaimed bind, injects Dropped after DETECTION_MULTIPLIER dead windows, and keeps the pre-bind exemption. It wakes once per window or when the mux closes the bind. A window is live if the ingress delivered to the slot, or if the sender holds no data credit with nothing waiting behind a sequence gap. The backpressure counter is lost for mux streams: nothing forwards into the anchor channel, so it no longer moves. velo_streaming_slot_credit_exhausted_total, counted on the producer's node, is the consumer-saturation signal there.

What changes in behavior:

• A consumer that stops polling holds its sender to the credit window C. The pump counted each move into the 256-deep anchor channel as a drain, so the sender could run C + 1 + 256 records ahead.
• Detection lands between DETECTION_MULTIPLIER windows and one window more after the last arrival, because windows run on the watchdog's own clock. The pump detected exactly DETECTION_MULTIPLIER windows after the last forwarded frame.
• A window is exempt only while the sender holds no data credit, as the consumer node's credit account sees it, and nothing waits in the reorder hold: credit spent into the hold behind a gap that never closes would otherwise exempt a stuck stream for good. Arrivals count deliveries, not records parked in the hold. An earlier version exempted any non-empty buffer; a dead sender behind a consumer that was connected but slow then kept its anchor, slot and peer batcher until the consumer drained. A sender that still holds credit can heartbeat, so its silence is its own whether or not records wait unread. The account also covers a sender starved by the byte budget, whose credit the consumer node withholds while the buffer holds that budget with fewer than C records in it.
• Retiring a bind or removing the anchor withdraws the feed. The consumer drops a withdrawn feed before it reads that buffer again, so unread records in it are discarded.

Measured on one Grace node with a two-process microbenchmark (frontend 32 threads, 2,048 streams, 918 records of 160 B, jemalloc, median of 3). Frontend CPU per record fell from 5.87 to 5.33 µs at 12 peers (−9.2%) and from 7.71 to 7.38 µs at 48 peers (−4.3%). In the profiles before these changes, the reader pump was 28% of frontend CPU at 12 peers. Response plane performance has the setup and the dirty-set numbers.

Generations and epochs

Dense slot reuse without a generation delivers a stale record to the stream that now holds its index. A u8 generation is enough because the epoch scopes the whole table: batches from an old epoch are discarded as a whole.

The byte side throttles grants and does not refuse records

When frame credit and the byte cap disagree, the receiver withholds the next grant. Refusing a record whose frame credit was already granted breaks a stream for a peer that obeyed every rule. The ingress hold is the one exception, because the alternative is unbounded growth behind a gap that can stay open.

Drain every inlet up to the byte budget, and do not split control from data

The first design split the egress inlet into an unbounded control lane and a bounded data lane. FrameTransport::connect returns one flume::Sender<Vec<u8>>, and in that byte channel a terminal and a token look the same. A split needs a typed sink in the velo-ext trait, which is a breaking change to a published crate.

The batcher instead drains every inlet, with or without credit, into a per-slot withheld queue that the slot byte budget bounds. At the budget, the batcher stops pulling from that inlet, and the producer's send waits. A synchronous terminal send on a full inlet waits in a task, so it does not block its caller.

The first version killed the slot at the budget instead of pausing. That killed every producer faster than one credit round trip, including producers whose consumer was draining. A 10,000-record stream of 1 KiB records over loopback died after 1,128 records.

Ingress copies each record out of its batch

The ingress gets each batch as one Bytes, and each record knows its range in it. It still copies every record body into its own Vec for the slot buffer. Handing the consumer payload.slice(range) instead would remove that copy, and with byte items the consumer would then copy nothing. It is not done, for two reasons:

• The byte budgets would stop bounding memory. A slice keeps its whole batch in memory. A batch holds records of many streams, so a consumer that keeps one 100 B token keeps up to the batch size, and many slow consumers keep many batches. The slot and peer budgets count record bytes, so they would no longer bound what the node holds. Slicing only a record that fills most of its batch would bound this, but it is a second policy to keep correct.
• velo-ext fixes the slot buffer type. The mux implements FrameTransport::bind, which returns flume::Receiver<Vec<u8>>. A slot buffer of Bytes needs the trait path to keep its own Vec channel, and it ripples through the attach and pre-bind paths, the direct feed, the MPSC pump and the anchor.

The producer side has the same limit. Its frame goes through the slot inlet, the flume::Sender<Vec<u8>> that FrameTransport::connect returns, so it is built once and copied into the batch once. Writing it straight into the batch needs another inlet type for the mux and a StreamSender that knows which transport it has.

The cost of the ingress copy is estimated at 35 to 55 ns per record, in Response plane performance.

Control is coalesced state, bounded by allocation

Credit returns, closes and singleton resolutions reach the batcher as per-slot state, not as messages. A queue is unbounded exactly when a flush parks on admission, which is when the peer returns the most credit.

A size cap on these maps was the wrong bound. A 4,096-entry cap was sized for about 1,024 slots per peer. On the serving rig one peer held 4,000 to 6,700 slots, and the cap refused grants, closes and the answer that lifts a fenced slot. A refused grant is lost for good, because the receiver zeroes its ungranted count when it mints the CreditUpdate. Each map is now bounded by what distinguishes a real key from a bogus one:

• Control that a peer sends about this side's slots is accepted only for an index this batcher allocated, at its live generation. A retired generation is dropped silently, because a close-then-reopen race is ordinary.
• Replies for slots that this side's ingress admitted are never refused.
• Rejections of an OpenSlot that ingress never admitted travel on a reject lane capped at 8,192. A dropped rejection loses no credit, so only this lane can have a cap.

Admission is the backpressure, and a failed admission is epoch death

The batcher awaits each flush's admission, so a congested peer parks the batcher in FIFO order and not a runtime worker. FireResult erases the admission error to a string, so the batcher cannot match on a subset of errors. Treating every failure as epoch death is correct anyway. A batch that never reached the wire leaves a frame_seq gap in every slot it carried, and the mux does not retransmit.

Eager OpenSlot, and credit from the attach response

OpenSlot goes out when the stream attaches, so the accept window (ReservationConfig::prebind_timeout, 180 seconds by default) measures "time until the OpenSlot arrives" and not "time until the first token". Otherwise a long prefill expires the window and kills a healthy request.

The slot opens already holding the window from the attach response. Before negotiation, the receiver sent a CreditUpdate on OpenSlot, which cost a round trip per stream. Both together grant 2C against a C+1 buffer, so the change was a swap.

flush_batch is a call, and Manual has no timer

The rejected design was an RAII gate with a 5 ms watchdog that forced the gate open. A gate can be held across an .await, which made a deadlock possible. The watchdog turns a forgotten flush into the windowed policy. Manual is then deterministic only until something is slow, which is the worse failure. A deployment that wants a net configures Auto { max_linger }.

The one exception is a pending credit reply. Nothing on the replying side knows that a peer is owed credit, so Manual cannot leave the reply to the application. reply_linger carries the reply, and the batch around it, out after a bounded wait. A pass that contains a credit reply can therefore split across two writes under Manual.

The price of Manual is also known. A per-pass flush caps a batch at that pass's fan-out, where Auto can pack the next pass too. Measured, Manual is lower only when the producer outruns the batcher, and that surplus costs per-token latency.

Negotiate per attach and register both transports

The receiver answers messenger-mux-v2 only when the sender named it. resolve_transport fails on an unknown key, so a receiver that chooses the mux on its own breaks every older sender. A node with the mux registers both keys. The wire version lives in the key, so incompatible versions never pair. A zero window with the mux key is refused, because a fallback connects to a transport where nothing listens.

Credit return: the consumer posts, the reconcile decides

The first design had the reader pump call credit.release(1) after each handoff. The landed design has the drainer count on the slot's DrainSignal, list the slot in a per-peer dirty set and post the peer. A reconcile then swaps the count and releases it. For a mux stream the drainer is the consumer itself, which reads the slot buffer directly; for an MPSC anchor it is the MPSC pump.

• Why the consumer does not release. Releasing needs the peer mutex that the inbound batch path takes. Taking it per record trades a periodic cost for a worse per-record one. Two paths that each release an amount for one record also double-count, and the periodic sweep still exists. With the quantity on the slot, every visit is idempotent.
• Why wakes coalesce per peer. Posting per record replaces a periodic cost with a per-record one. A per-slot record threshold withholds credit for the first T records of every slot. It also still posts once per slot per threshold, so it is worse on latency and on volume.
• Why the wake flag drops before the walk. A drain that lands during a walk must post a fresh wake. The same property lets a fast consumer re-arm the flag at once, so the task walks back to back under the peer mutex. drain_visit_floor (2 ms, the old sweep interval) caps the visit rate per (peer, lane). A wake inside the floor is scheduled for when the peer comes due, and later drains coalesce into that visit.
• Why the arrival path reads the dirty set. Narrowing the per-batch reconcile to the slots a batch delivered into left every other slot to the doorbell and the sweep. Every stream on the serving rig sends about 4 records more than its then-default 256-record window, so each stream's tail needs one grant. On the rig, sender credit exhaustion rose from 13 to about 20,500 per process. Doorbell visits fell to about one per peer per 12 ms. The frontend's lane wait rose from 0.36 ms to 1.4 s per batch. Throughput halved to 1,516 req/s and TTFT p50 rose from 55 to 331 ms. The grant must ride the peer's next inbound batch.
• Why the count replaces the occupancy estimate. Reading frame_tx.len() takes the channel's lock. At about 1,000 slots per peer and 6 million batches per rep, that read was the largest velo-only cost on the frontend.
• Why the order is clear, then swap. A drain that lands between the two steps finds the listing down and lists the slot again. The next pass then finds either a zero count or the new drain, never a count with nothing to fetch it.
• Why a bitmap, not a lane. The dirty set was a bounded flume lane per peer plus a listed flag per slot. With about one record per stream per batch, nearly every drain found its slot unlisted and took the lane's mutex, which every consumer of the peer shares. It is now one lock-free bitmap per peer (lib/velo/src/streaming/messenger_mux/ingress/dirty.rs). Only the drain that newly lists a slot touches the peer's shared pending flag, and the drain wake lane is unbounded, so a wake is never refused. On the two-process microbenchmark, frontend CPU per record fell from 6.66 to 5.82 µs at 12 peers (−12.5%) and from 8.60 to 8.11 µs at 48 peers (−5.6%).

The doorbell's deferral queue holds at most one entry per peer. An earlier version queued a second entry when a periodic tick cleared a peer's wake and a drain re-armed it inside the floor. The queue then grew by one entry per tick, permanently. The floor is also clamped to one hour, because last + floor overflows for an absurd Duration.

The periodic sweep defaults to 200 ms and must be non-zero. tokio::time::interval panics on a zero period inside the spawned sweep task. The build returned Ok and the mux then ran with no credit backstop, no eviction and nobody reading the doorbell. The build now refuses a zero interval.

A grant threshold (grant only when half the window has drained) was built and measured. It cut credit updates 19-fold and did not change CPU. It made the inter-token p99 worse, because a slow reader exhausts the sender sooner when drained credit is held back. Do not build it again.

Zero-RTT: pre-bind is the synchronous twin of bind

MessengerMuxTransport::bind_on_lane calls the same open_bind body as FrameTransport::bind. bind is async only because the trait is. One body keeps the two paths from drifting.

• Rejected: bind on OpenSlot. Minting a ticket and binding when the OpenSlot arrives has a simpler lifecycle. It inverts a layer (ingress resolves the anchor's channel from the registry) and it loses the meaning of the accept window.
• No protocol version bump. The ticket rides the application's envelope as an optional field. A version bump breaks an old worker outright. An absent field makes the worker attach the ordinary way.
• The watchdog exemption is gated on WatchdogContext::prebound, not on the drain claim alone. The mux parks a DrainSignal for every bind. The signal reads as unclaimed until the peer's OpenSlot arrives, which is after the attach response returned. prebound tells a real pre-bind (no sender yet) from an ordinary attach whose OpenSlot is still in flight. It is an Arc<AtomicBool> so that PreBind::adopt can clear it at once when a sender attaches the long way.
• The exemption lets the accept window be the true backstop. Counting silence before a sender exists caps the wait at DETECTION_MULTIPLIER × heartbeat_interval (15 s at the default), whatever the application configured.
• Adoption does not restart the accept window. An adopted attach inherits what is left of the accept window. Whichever deadline comes first catches a sender that dies before its first record: the watchdog, measured from adoption, or the remaining accept window. The watchdog fires DETECTION_MULTIPLIER (3) heartbeat intervals after adoption. So with the 180-second default, a heartbeat interval of 60 s or more lets the accept window win. The window closes up to one credit_sweep_interval late, so at exactly a third of the window the two can land together.

async_open_ack fences only when admission is pending

The first version fenced every detached open. On an uncongested peer the admission is already Admitted synchronously, and per-target FIFO already orders the slot's later records behind the OpenSlot. The fence then only made the first record wait for a spawn, a control insert, a wake and a release pass. The open now fences only when admission_state() is not Admitted.

This skip applies to OpenSlot only. A rendezvous record always fences its slot, even when admitted synchronously. The receiver resolves rendezvous bytes in a detached task before dispatch, so the sender's admission order does not order the receiver's apply.

The batch gap meter keeps a high-water mark

A detached open and a later flush to the same peer are separate tasks, so their batch_seq values can reach the wire out of order. Per-slot order does not depend on batch_seq, so this inverts a counter, not a stream. The gap meter used wrapping_sub, and one inverted pair added 4,294,967,295 to velo_streaming_mux_batch_seq_gaps_total. The meter now keeps a high-water mark under RFC 1982 comparison. A batch behind the mark is not metered, so an inverted pair costs exactly one. A batch_seq per registration class removes that one. It is not built.

One timer per stream in the reader pump

This applies to reader_pump, which now serves only per-stream transports. The mux's stream watchdog has no per-record work at all: it wakes once per window on its own timer and reads the slot's arrival count.

The pump arms one pinned Sleep for its stream. A received frame pushes the deadline only when the deadline is within half a window. When the timer fires, the pump re-arms from the last frame and counts a miss only if the last frame is older than the deadline.

The first version re-armed only from the fired arm. Under traffic every stream fired once per window, and 8,192 streams opened in the same second fired in phase. The receive-arm push is what keeps a live stream's timer from firing.

The pump stamps the frame time after the forward to the anchor channel completes. A forward blocked on a full anchor channel is the pump working, not the sender going silent. Stamping on arrival charges that block to the sender and fires the watchdog one window early.

The select! is biased toward the receive arm. tokio::time::timeout, which this replaced, always polled the receive first. An unbiased select can let a ready timer win over a ready frame, and report a real terminal as a watchdog Dropped. The messenger's ordered-lane router uses the same one-timer shape and the same bias.

Heartbeats: the consolidation that is partly built

A per-stream heartbeat task does not detect a hung producer, because it runs on its own task. Under the mux, the health check of the batcher detects a dead peer on the sending side, and the mux learns of it through epoch death. On a node that only receives, a dead producer is still the stream watchdog's job. A connection that closes after admission is not reported to the mux, and the stream watchdog ends the slots that lost records with it. The other signal that a stream heartbeat carries is per-slot saturation. The design keeps the heartbeat per slot and makes it cheap:

• Suppression. A per-slot "last send" tick (one relaxed AtomicU64 store per frame). A slot that sent anything in the interval skips its heartbeat.
• Phase alignment. Each sender follows one peer-level tick (a tokio::sync::watch driven by one timer task) instead of its own interval. Heartbeats then coalesce into one batch instead of scattering across the interval.
• No per-sender task. The batcher walks its live slots on each tick and emits SlotHeartbeat for the idle ones.

N scattered heartbeats per interval become at most one batch, and N tasks become one. The receiver still sees a per-slot beat inside its per-slot deadline, so the stream watchdog and DETECTION_MULTIPLIER do not change. SlotHeartbeat must not join the reserved control class: a heartbeat dropped under saturation is the saturation signal. The third point is built without the batcher: one task per AnchorManager beats every remote sender, so N senders use one task. Suppression and phase alignment are not built, and heartbeats still travel as Data records. A sender on the anchor's own worker does not beat, because no watchdog reads its beats.

Why the mux surface is not in velo-ext

The mux is a user of the Messenger send path, not something a transport implements. An out-of-tree Transport gets multiplexing without knowing that the mux exists. Exposing a mux to out-of-tree FrameTransport implementors (for example open_mux and MuxChannel) is deferred for two reasons:

1. Coupled defaulted methods fail silently. The velo-ext rules require default implementations. A transport can then report supports_mux() == true and leave the acceptor a no-op, so senders open slots into nothing. If this surface lands, it must be one method, as_mux() -> Option<&dyn MuxTransport>, returning a separate trait with no defaults. The seam must carry a typed sink (Frame::{Data, Terminal, ...}), not a byte channel. A transport can then reserve capacity for records that must never queue behind data.
2. A byte pipe can be the wrong abstraction. An RDMA transport already multiplexes per queue pair. Forcing it to serialize into a byte stream so that velo can split it again is slower. Removing a published velo-ext item is a major version bump, so the wrong shape is expensive.

The additions this work made to velo-ext were shaped to last. Transport::max_message_size(target) -> Option<usize> has a default, and None means unknown, which costs the caller a conservative budget. SendOutcome::Pending(SendAdmission) changed a published enum, so it shipped once, as a coordinated velo-ext and velo release. The egress recorders on TransportObservability have no-op defaults.

The cost of deferral is bounded. Out-of-tree FrameTransports get no multiplexing, and nothing else breaks. A receiver offers messenger-mux-v2 only when the mux is installed and enabled, so a deployment without the mux degrades correctly.

Methods that the mux needs outside the trait (advertised_limits, connect_negotiated, take_drain_signal, prebind, release_bind, close_claimed_slot) are inherent on the concrete MessengerMuxTransport. AnchorManager already holds the concrete type, so none of them needs a trait change.

Failures and their mechanisms