Rendezvous and RDMA

Rendezvous moves a large payload by handle. One worker (the owner) stages the bytes and gets a compact DataHandle. The owner passes the handle to consumers by any means: a message field, an event, or a typed response. Each consumer then pulls the bytes from the owner. The consumer drives every transfer. The owner only answers.

The bytes move on one of two paths. The chunked path works everywhere and pulls the payload in 512 KiB messages. The RDMA path lets the consumer's NIC read the owner's memory with one UCX ucp_get_nbx call. The owner decides the path for each transfer, and a caller writes the same code for both.

The RDMA path needs the ucx feature on Linux. See Run rendezvous over RDMA for the build and the runtime setup. See RDMA performance for measured numbers, and RDMA design for why the design has this shape.

The handle API

A handle is a u128: the owner's WorkerId in the upper 64 bits and a local slot id in the lower 64. Serde writes it as {hi, lo} because MessagePack has no clean u128 encoding.

The owner registers seven handlers: _rv_metadata, _rv_acquire, _rv_pull, _rv_ref, _rv_detach, _rv_release, and _rv_lease_renew. The control plane uses JSON over ordinary messenger calls, so any transport carries it.

One transfer, both paths

_rv_acquire takes the read lock and chooses the path in the same round trip.

sequenceDiagram
    participant C as Consumer
    participant O as Owner
    participant NIC as Consumer NIC
    C->>O: _rv_acquire {handle, rdma: offer}
    alt owner answers RDMA
        O-->>C: Rdma {lease_id, descriptor, lease_timeout_ms}
        C->>NIC: ucp_get_nbx(addr, len, packed rkey)
        NIC->>O: one-sided read of owner memory
        loop every lease_timeout / 2 while the GET runs
            C--)O: _rv_lease_renew {handle, lease_id}
        end
        alt GET or decode fails
            C->>O: _rv_detach {lease_id}
            C->>O: _rv_acquire {handle, rdma: none}
            O-->>C: Ready {...}
            C->>O: _rv_pull (chunked)
        end
    else owner answers chunked
        O-->>C: Ready {lease_id, transfer_id, chunk_size, chunk_count}
        loop chunk_count times
            C->>O: _rv_pull {transfer_id, index}
            O-->>C: chunk bytes
        end
    end
    C->>O: _rv_detach or _rv_release {lease_id}

Both answers end in the same detach or release. The owner checks the registration again at every acquire, so no cross-node invalidation protocol exists.

When the owner answers RDMA

The owner answers AcquireResponse::Rdma only if every check below passes. The checks run in this order. Each refusal increments velo_rendezvous_rdma_path_total{path="chunked", reason=...} on the owner.

Check 7 comes after the facts about the configuration and the slot, so draining counts only the acquires that the drain changed. A heap-staged slot on a draining owner still counts as not_pinned. Check 8 catches an external region that was deregistered under the slot.

A lease answered with a descriptor always gets a deadline. A chunked lease gets one only when its consumer asks for it (see What a fetch that gives up leaves behind).

What the consumer offers

The consumer builds its offer from three local facts. It asks the owner nothing.

• This instance has an RDMA registry.
• The kill switch on this instance is off.
• The owner is registered on a transport whose key matches the registry backend.

The GET uses this consumer's own UCX endpoint to the owner. So "UCX is registered for that peer" is the whole condition. UCX does not have to be the primary transport. The expected deployment is a TCP control plane with UCX beside it.

The rule has a cold-start effect. A peer that is not registered yet translates to nothing. The first get to an unknown owner therefore offers nothing and pulls chunks. Later gets take the RDMA path.

Fallback is a routing decision

After the owner sends a descriptor, four failures can occur on the consumer. They map to three reason labels:

For each failure, the consumer detaches the lease, acquires again with no offer, and pulls chunks. This always works, because a pinned slot also serves the chunked path.

The fallback happens exactly once. The second acquire carries no offer, so an owner that answers it with another descriptor is broken. The consumer returns an error in that case. A retry loop turns one bad owner into an unbounded storm of round trips.

A pinned slot is never RDMA-only. Old consumers, consumers without UCX, consumers below the threshold, and consumers whose GET failed all read the same slot by chunks. The owner does not need to know in advance which kind of consumer it talks to.

Staging on the owner

register_data_pinned

This call copies the bytes into a pool buffer that the NIC can read. It returns a DataHandle, not a Result. If the pool cannot give memory, the call stages the bytes in plain memory and records why:

• The registered-bytes budget is spent: budget.
• The pool cannot allocate for another reason: pool_exhausted.
• The kill switch is on: kill_switch.
• The instance has no UCX transport: the data goes to plain memory.

Zero-length data always goes to plain memory, because a GET has nothing to transfer. The call always costs one copy. The zero-copy form is register_data_in_region over memory that the caller registered.

register_data_in_region

This call stages a range of memory that the caller already registered with register_external_memory or register_owned. The range is relative to the registered pointer, not to the page-rounded effective range. A caller therefore cannot name a byte that is inside the registration but outside its own allocation.

The call returns a Result. A fallback to plain memory copies bytes that the caller asked velo not to copy. The kill switch does not affect staging here. It only decides whether _rv_acquire answers with a descriptor.

The slot holds an in-flight guard on the region. RegionGuard::unregister therefore waits for the anchors inside the region before it unmaps. The slot takes the guard first and then reads the region state. The reverse order leaves a gap in which the whole gate, drain, and unmap sequence can run.

Chunked reads of an external slot never form a Rust reference into the region. A peer holding the key can write the region at any time. The read copies with ptr::copy_nonoverlapping into a fresh buffer, under the region's copy gate, 512 KiB at a time. The copy gate orders each read against the deregistration latch. A copy in progress delays the latch. A copy that starts after the latch sees the flag and refuses.

Transparent staging

The messenger stages any payload above 256 KiB through rendezvous and replaces it in the frame with a handle. The receiver resolves the handle before the handler runs, so handler code never sees the difference.

The transparent stager runs inside the synchronous send_message. It uses only arenas that the pool has already mapped. It never maps a new arena, because mapping is an ibv_reg_mr whose cost is linear in size. A send must not stall on that.

The result: a process whose only staging is transparent never maps an arena and never takes the RDMA path. One explicit register_data_pinned call maps the first arena. After that, transparent payloads use it. Growing the pool from a send on a background task was rejected. A caller cannot reason about a message send that causes a 64 MiB pin.

The fetch has a deadline

The receiver fetches the payload with its own calls to the sender (_rv_acquire, then _rv_pull). Each call holds one of the receiver's response slots, and the request keeps its drain guard, until the fetch ends. A fetch with no deadline ends only when the sender answers. So a sender that stops answering took one slot from the receiver for each staged request, after its own caller had given up.

The fetch now ends at the first of two bounds:

• The caller's. A sync or unary call with .timeout sends what is left of it with the handle, in the _rv_budget_ms header, rounded up from nanoseconds to whole milliseconds. The receiver starts the bound when it takes the message off its queue, so it does not end before the caller's own timer. How much later it ends depends on transit, queueing and scheduling, which nothing bounds. When it runs out, the receiver drops the fetch and sends no reply: the caller has its RequestTimeout already, and a reply that raced the caller's timer would replace that with a different error.
• The receiver's. VeloBuilder::staged_fetch_timeout bounds every fetch on this instance. It covers what carries no bound: fire-and-forget sends, calls without a timeout, and senders from before the header. When it runs out, a sync or unary caller gets an error reply that says the fetch did not finish in time. It does not apply to the items of a stream. A stream item over the threshold is a fire-and-forget batch, so a dropped fetch lost the item with no word to the stream. Its consumer got SenderDropped from the stream watchdog about 15 s later, while the producer was still attached.

The bound is a duration and not an instant, because the two nodes do not share a clock. It is its own header because a receiver from before it reads the _rv value as a bare number. Such a receiver ignores the new header.

The staging headers belong to the send that staged

The receiver takes _rv and _rv_budget_ms out of a staged message before its handler runs, so a handler never sees them. A handler that passes its headers on in a send of its own therefore passes neither. A forwarded handle made the next receiver fetch a payload that was not its message's. A forwarded budget made it give up at another call's deadline.

A staged send writes both headers itself, after the tracing propagator, so the ones on the wire are its own. A budget without a handle means nothing to a receiver, so an eager send needs no such care. An encoder that owned the budget key was rejected: it compared every header key of every send with _rv_budget_ms, and it left a forwarded _rv in place.

What a fetch that gives up leaves behind

The receiver tells the sender nothing when it gives up a fetch. A slow sender then grants a lease that nobody pulls or releases, and a payload whose fetch never got a lease stays staged. Two things end both on the sender:

• A time-to-live. A staged payload whose call has a .timeout gets that deadline as its RegisterOptions::ttl. Past it, the slot is freed with every lease on it, whether or not a consumer is still reading. The caller has given up by then.
• A chunked lease deadline. A fetch with a bound sends an _rv_lease_ms header with its _rv_acquire. The owner ends the lease when no _rv_pull renews it within that time. An owner from before the header ignores it. An acquire without the header, from an older consumer or from a public get, gets a lease with no deadline, as before. A get caller can still hold its lease after the last pull.

A sweep in the store enforces both, every 100 ms. It starts when a deadline or a time-to-live is set, and it exits when none is left, so an instance that stages nothing with a deadline arms no timer. A UCX instance stops it at shutdown, together with the RDMA reaper.

The lease deadline is a header, not a field of RvAcquireRequest, because a new public field breaks code that builds the request. A message from the receiver that abandons the fetch was rejected. Before _rv_acquire returns, the receiver has no lease to name, and a lost message still leaves the payload staged. What still stays staged is in the limitation.

The receiver's limit

A staged payload's size is its owner's word: the acquire answer states total_len, or a descriptor states its length. With no limit, the consumer allocates that size before it pulls a byte, so one peer can make another allocate any amount for one message. VeloBuilder::staged_payload_max sets the largest payload a node takes this way. The transparent resolver checks the stated size against it before it allocates anything, on the chunked path and on the RDMA path.

A payload over the limit is refused:

• The consumer releases the handle, so the owner frees the staged copy. Transparent handles are single-use, and a detached one would stay on the owner with nobody left to read it.
• A request that expects a reply fails with an error that names both sizes. Its handler never runs.
• A stream item on the messenger mux ends its own stream with a TransportError that names both sizes, and the producer is told to stop (Batched streaming). Other streams from the same peer go on.
• Any other fire-and-forget message is lost, and the refusal is logged.

The limit does not bound a payload under the staging threshold. Such a payload travels inline, in one transport frame, and the frame limit bounds it.

The size is checked after the acquire, from its answer, not by a _rv_metadata query first: that query would add a round trip to every staged payload to catch the rare one that is too large. The check is not after the allocation, because by then the memory is spent.

Reading on the consumer

• get GETs into a pool buffer on the consumer, then copies into Bytes so the pool space returns at once.
• get_pinned returns the pool buffer itself. On the chunked path it copies the chunks into a pool buffer, so the return type is the same on both paths.
• get_into with a PinnedWriter (from alloc_pinned_writer) GETs straight into the caller buffer. Other RendezvousWrite destinations get a GET into a pool buffer and one copy.

The GET runs on a spawned task that owns the destination reservation (a TransferHold). If the caller drops the get future, the task keeps running and the granules stay reserved until the backend reports completion. Without this, a dropped future returns the granules to the free list while the NIC still writes into them.

Leases and deadlines

The owner sees every chunk request, so it sees an abandoned chunked transfer stop. It does not see an RDMA GET, because the consumer's NIC issues it. Without a deadline, a consumer that crashes mid-GET holds the read lock and the refcount forever. So RDMA leases carry a deadline.

• lease_timeout defaults to 30 s. The owner sends it to the consumer as lease_timeout_ms.
• The owner reaper scans every lease_timeout / 2, with a 10 ms floor. A silent lease is force-released between one and one and a half timeouts after its last renewal.
• While a GET runs, the consumer sends _rv_lease_renew every lease_timeout / 2, with a 5 ms floor. A lost renewal is harmless, because the next one is half a deadline away.
• Renewal stops when the transfer ends. Holding an RDMA lease idle past its deadline is not supported. A caller that holds data across a long pause releases it and acquires again.

On the wire, 0 means "no deadline". A consumer that gets 0 starts no renewal ticker. A sub-millisecond lease_timeout therefore clamps to 1 ms at build time. Without the clamp, the owner arms a deadline that the consumer was told does not exist, and the lease never renews. The clamp makes small values defined, not useful. Renewals go out at most every 5 ms and the reaper scans at most every 10 ms, so values below about 100 ms reap live transfers.

The descriptor

The RDMA answer carries a velo-owned binary descriptor, not JSON:

backend:    u8      1 = ucx
version:    u8      1
flags:      u8      0
generation: u64le   owner's registration generation
addr:       u64le   owner-authored absolute address
len:        u64le   bytes to read
rkey_len:   u16le   length of the packed key that follows
rkey:       [u8; rkey_len]

The header is 29 bytes, then exactly rkey_len key bytes, then nothing.

The format is binary because the packed rkey goes to ucp_ep_rkey_unpack, which has no length parameter. A truncated or corrupt blob is an out-of-bounds read inside UCX, and no Rust wrapper can make that safe after the fact. The framing must therefore be exact and owned by velo.

Decoding refuses anything that it cannot account for byte for byte:

• an unknown backend or version
• a non-zero flags
• a zero len
• an rkey_len that disagrees with the bytes that follow
• any trailing byte

A refusal is not an error for the caller. It costs one extra round trip on the chunked path.

The key is bounded at 4096 bytes by the descriptor and at 1024 bytes by the UCX backend. The backend checks the bound at map time and again before unpack. Before any pointer reaches ucp_ep_rkey_unpack, the UCX backend also walks the two stages of the UCX 1.22 packed-rkey format. It refuses a blob unless UCX's own parse ends inside the blob, and it refuses an out-of-range mem_type. This pre-parse checks framing only. A stale key with perfect framing passes it. See RDMA design for why that matters.

generation is in the descriptor for diagnostics. The consumer does not send it back in detach or release.

Version skew

Two #[serde(default)] fields make a mixed-version deployment fall back to chunked instead of failing:

• RvAcquireRequest::rdma (the offer). An old owner ignores it and answers Ready. An old consumer omits it, and an owner that sees no offer never answers Rdma.
• AcquireResponse::Rdma::lease_timeout_ms. If a future owner omits it, the default 0 means no deadline, which is the old behavior.

The rule for new fields: every field is #[serde(default)], and every default means what the protocol did before the field existed. A default that means "the new behavior" makes an old peer's silence look like consent.

Memory registration

Velo registers memory in two ways. Both end in the same lifecycle.

The arena pool

Velo sets UCX_RCACHE_ENABLE=n and UCX_MEM_EVENTS=n by default, so UCX installs no malloc hooks in the process. With those settings, every ucp_mem_map is a fresh ibv_reg_mr whose cost is linear in size, and UCX caches nothing. Pre-registered arenas are therefore required, not an optimization.

• The pool registers a few large arenas. Each arena is one page-aligned allocation, one ucp_mem_map, and one packed rkey.
• The first arena is 64 MiB (initial_arena_bytes). Later arenas grow geometrically up to 1 GiB (max_arena_bytes).
• Inside an arena, offset-allocator hands out 4 KiB granules in O(1). Its float bins waste at most about 12.5%.
• A request of 64 MiB or more (dedicated_arena_min) gets its own arena. Without this, the 12.5% round-up costs 128 MiB on a 1 GiB object.
• The suballocator node pool is sized to the granule count, about 28 bytes per granule (0.7% of the arena). A smaller node pool makes a fragmented arena report "full" while it still has room. The pool then maps an arena that it does not need.
• The free token (the private Allocation inside a PinnedBuf) and the wire descriptor are different values. A peer with a descriptor can read the range. It cannot free it, and the descriptor does not keep it alive.

registered_bytes_budget (1 GiB by default) caps mapped bytes across the pool and external regions together. It counts what the kernel pins: whole pages, reconciled to the effective range that UCX reports. A mostly empty arena costs its full size. Over the budget, pinned staging falls back to plain memory. Budget exhaustion is never a hard error.

Pages outlive their registration. Arena memory is freed only after the backend confirms the unmap. A registry torn down without its shutdown sweep leaks its arenas and logs at error. A leak is the safer failure, because freeing pages that UCX still pins is the hazard this layer exists to prevent.

External registration

Velo::register_external_memory(ptr, len) registers memory that the caller owns and returns a RegionGuard. The function is unsafe. Until RegionGuard::deregistered() resolves, the caller guarantees all of these:

• ptr is valid for reads and writes of len bytes. Registering a read-only mapping is undefined behavior.
• ptr + len does not wrap.
• The allocation is not freed, moved, remapped, or reallocated.
• No Rust reference into the range exists. A peer can write at any time.

Velo::register_owned(Box<[u8]>) is the safe form. Velo holds the buffer until an unmap is confirmed. On failure, the buffer comes back inside RegisterOwnedError, because BudgetExceeded is a routine refusal and must not consume the allocation. RegionGuard::unregister_owned gives the buffer back after release.

The guard does not borrow. Its lifecycle:

• deregistered().await is the single release signal. It resolves on a confirmed unmap, or at the end of graceful_shutdown, whichever comes first. Only then can the caller free the memory.
• unregister(timeout) gates new use, drains in-flight work, and unmaps. It returns Deregistered::Drained or Deregistered::DrainTimedOut. Both mean the memory is released. DrainTimedOut means velo did not wait for in-flight work. An earlier Err(Timeout) shape mixed up "still mapped" with "unmapped early", and only the first means "do not free".
• Drop without unregister starts a background deregistration, logs at warn, and returns at once. It never blocks, because blocking in Drop inside a tokio runtime deadlocks the worker. The memory stays pinned until the background task finishes. An early drop is a liveness bug for the caller, not a soundness bug for anyone else.
• watch() returns a RegionWatch. It carries the same observers but does not own the release, because unregister consumes the guard.
• If the progress thread dies while the backend still reports registrations, deregistered() never resolves. Velo cannot prove the memory was released, so it leaks the memory on purpose.

Registered means remotely writable

UCP has no enforced protection field. Any holder of a region's key can write the region, not only read it. The GET-only shape of the rendezvous protocol is a convention, not an enforcement. Registration pins whole pages, so bytes next to the allocation share the pinning and the remote writability. RegionGuard::effective_range reports what was pinned.

Registering memory is therefore a trust decision about the peers that this instance talks to. Key material only goes to peers that this instance already talks to. A &mut [u8]-shaped API is an aliasing lie, so velo does not offer one.

Lifecycle pressure

Registrations are evicted by a byte budget, never by an inactivity timer. A timer that evicts a registration while a peer still holds its descriptor creates a stale key. On InfiniBand, a stale key aborts the process. No surveyed system (UCX rcache, libfabric, MPICH, NCCL) evicts registrations on a timer.

Two timers exist anyway, on objects that no peer can hold a reference to.

Arena reclamation

One periodic sweep runs on the lease reaper's task. A second timer needs its own ordering against the first at shutdown.

• Empty pooled arenas, only when arena_reclaim_after is set (off by default). An arena with no suballocation has no live descriptor into it, so an unmap is safe. retain_arena_bytes (64 MiB by default) keeps a warm floor mapped, so an idle-then-busy workload does not pay a fresh registration. The sweep considers the newest arenas first, so the small first arena survives as the floor.
• Empty dedicated arenas, always, with no timer and no floor. A dedicated arena serves one request and is never offered again, so after its buffer drops it only consumes budget. Without this, a workload that stages 64 MiB objects repeatedly exhausts a 1 GiB budget after 16 of them and falls back to chunked for good.

The sweep period is the smaller of lease_timeout / 2 and arena_reclaim_after / 2, with a 10 ms floor. An arena is unmapped between one and two arena_reclaim_after intervals after it becomes empty.

Endpoint idle reaper

UcxTransportBuilder::ep_idle_timeout(Some(d)) closes UCX endpoints that nothing used for d. It is off by default and experimental. Values below 500 ms rise to 500 ms, which is about 35 times the 14 ms that the first RDMA GET on a fresh two-process pair over CX-7 InfiniBand took, wireup included.

• "Used" means both directions. Our sends, GETs, pings, and eager wireup stamp the endpoint. Every frame the peer sends us stamps it too, of any kind. Each frame header starts with the sender's 8-byte incarnation, and the receiver stamps the endpoint to the peer with that incarnation. The receive callback passes senders to the main loop through 8 slots, one per sender. If more than 8 different peers send frames between two drains of those slots, the main loop stamps every endpoint instead, so no sender's stamp is lost. On a worker that hears from many peers at once, idle endpoints can therefore stay open longer.
• Why a tag and not UCX's reply endpoint: UCX tells the receive callback which endpoint a frame came from only when the sender sets the REPLY flag. Velo sets that flag on Messages and pings only. A stamp that used it missed Responses, Events, Acks, Pongs, and ShuttingDown echoes, so a peer that streamed Responses back to us had our endpoint reaped under the stream. Setting the flag on every frame was rejected: after a reap, the peer's REPLY-flagged frames are lost (see below), so with the flag the next Response to a new request was lost too (3 of 3 in a probe). Without the flag, Responses kept arriving.
• An endpoint is idle only when no send posted on it is in flight and none has completed for the timeout. The reaper counts the sends in flight on each endpoint, and an asynchronous completion of the last one restarts the idle clock. A send that completes inline, while it is posted, records no completion time; the stamp taken when it was posted stands in for it. Replies on the endpoint that UCX hands the receive callback (Pongs and ShuttingDown echoes) are not counted. The inbound frame that caused each reply refreshes the endpoint in the same loop pass, before the reaper scans.
• The reaper never closes an endpoint while an RDMA operation to that peer is outstanding. When the operation completes, the idle clock starts again, as it does when a send completes.
• The scan runs every half timeout, and at least once per second. On an idle worker a due scan runs at most 100 ms late (the progress thread's park timeout). An endpoint closes between one timeout and one timeout plus one scan period plus 100 ms after its last use. A long pass on a busy worker delays the close further.
• The progress thread reads the reaper's clock a few times per pass, not once per use: at the start of the pass, after ucp_worker_progress returns, and after each ucp_ep_create. An inbound frame is stamped after it arrives, which keeps the endpoint open a little longer. A send can be stamped up to one command drain before it is posted, which can shorten the timeout by that much. A send whose asynchronous completion empties its endpoint's count also reads the clock once, in stamp_ep_drained, to record when it completed. With the reaper off, the progress thread does not read the reaper's clock per pass, and a send pays only two None checks, plus one clock read per ucp_ep_create.
• Idle time starts when ucp_ep_create returns, not before the call. In the in-process test harness over TCP on a GB200 node, with several UCX workers per process, one ucp_ep_create call took 110 to 150 ms. Across 31 creates in one process during the parallel UCX tests, the median was 630 ms, which is longer than the floor. The cause is not isolated. An endpoint with no send on it, such as an eager one, has only this stamp to keep it open.
• A first send between fresh workers also waits while the peer sets up its own endpoint back to us. With the CPUs oversubscribed, a fresh pair's first frame took 360 to 420 ms. With many UCX workers in one process, the peer's progress loop that contained that step took 526 to 573 ms. The count of sends in flight keeps the endpoint open for that time. The count costs no measurable time per send: on a GB200 node, in a release build with 6 alternating runs, the reaper-on send cost moved +1% to +3%, inside a noise floor of at least ±6%.
• The next use wires up a new endpoint with no error. The peer stays registered.

Closing an endpoint costs the peer. UCX pairs endpoints by remote worker. The peer's own endpoint back to us rides the same connection. After a reap, the peer's next Message to us is admitted and silently lost, with no error at either end (measured). By the mechanism inferred below, its next ping gets no Pong (not measured), so its check_health returns Timeout (or ConnectionFailed if keepalive fails the endpoint while the probe is waiting). Every Message the peer sends after that is lost too, until keepalive fails its endpoint: retrying does not help, and neither does a fresh endpoint on our side (measured by hand; the in-tree test asserts only the first loss). Pings are expected to be lost the same way (inferred, not measured). Its Responses and Events still arrive (measured: 8 of 8 arrived in every recorded run). Acks carry no REPLY flag either, so the same is expected, but it was not measured. The likely reason, inferred from the UCX source and a measurement but not proven: a REPLY-flagged frame carries an endpoint id for the reply path, and UCX drops a frame whose endpoint id no longer resolves. UCX keepalive (about 20 s) then declares the peer's endpoint failed, and the frame after that arrives. The cost per reaped endpoint is that the peer's Messages and pings to us are lost, silently, until keepalive (about 20 s) fails its endpoint. The first loss is a Message with no error, or a ping that times out. The disruption self-heals.

The reaper is meant for peers idle in both directions. That case still carries the cost above, whichever side speaks first, when the peer keeps its endpoint to us: if we speak first, our send arrives with no error, and the peer still loses its next Messages to us, and, by the same inferred mechanism, its pings. If the peer also runs the reaper and has closed its own endpoint, its next send creates a fresh one; that case is not measured. Send-only fan-out costs nothing, because no peer uses a path back. The reaper is unsafe for peers that this instance only health-probes. Each probe creates an endpoint, and each reap costs the peer its next Messages and pings to us, if it sends any before keepalive fails its endpoint.

The sender tag changes the wire format. The UCX blob version is 2. A version-1 node and a version-2 node refuse each other at register(), so all UCX peers must be upgraded together. See Version compatibility.

Eager endpoint wireup

UcxTransportBuilder::eager_endpoints(true) creates the endpoint at register() instead of at first use. It is off by default. The first GET on a fresh peer pair costs about 14 ms of UCX wireup, against about 108 µs for a warm GET (measured on 2026-08-29, see RDMA performance). Eager wireup moves that cost off the first transfer. The hint is fire-and-forget: register() does not wait, and a dropped hint falls back to lazy wireup.

With both knobs on, a peer that is registered but never used is wired up once and closed about one timeout (see the close window above) after ucp_ep_create returns. The UCX wireup itself finishes later, in the background.

Shutdown ordering

An RDMA GET is invisible to the owner's in-flight counters. An unmap after transport teardown can remove memory under a transfer that nobody on the owner can see. graceful_shutdown therefore runs the RDMA sweep before messenger teardown, under one shared deadline.

sequenceDiagram
    participant App
    participant Velo
    participant RV as RendezvousManager
    participant Reg as RdmaRegistry
    participant UCX as UCX progress thread
    App->>Velo: graceful_shutdown(policy)
    Velo->>Velo: begin_drain (messenger gate)
    Velo->>RV: stop lease reaper, demote pinned slots to heap
    Velo->>Reg: 1. Gate: refuse new registrations and allocations
    Velo->>Reg: 2. Drain: wait for in-progress registrations and region in-flight counts
    Velo->>Reg: 3. Deregister: unmap every region and arena
    Reg->>UCX: unmap (regions before endpoints)
    Velo->>Velo: messenger graceful_shutdown (remaining budget)
    UCX->>UCX: teardown force-unmaps anything left
    Velo->>Reg: latch deregistered() for every region

• Demotion copies, it does not drop. Each pinned slot is copied to the heap. A chunked pull admitted before the gate still finishes. A peer mid-GET holds a descriptor for the old address. That GET fails at its own end on InfiniBand and is silently lost over UCX_TLS=tcp.
• The gate is an ordering, not a flag. Admission takes an in-flight guard first and then reads the gate with SeqCst, as TransportAdapter::admit_message does. A plain token check followed by a map is a check-then-act race. A registration can land after step 3 and leave pinned memory with no tracking entry.
• The drain is bounded. When the budget runs out, velo warns and force-unmaps. The same straggler rule applies.
• RdmaConfig::shutdown_timeout (30 s) bounds the sweep when the policy is WaitForever. RdmaConfig::drop_dereg_timeout (30 s) bounds a background deregistration from a dropped guard.

See Shutdown and drain for the messenger half.

Configuration

Pass RdmaConfig with VeloBuilder::rdma_config. It has no effect without VeloBuilder::add_ucx_transport. A UCX transport added with add_transport gives full messaging and no RDMA registry.

The defaults keep the order rdma_min_bytes <= DEFAULT_THRESHOLD < DEFAULT_CHUNK_SIZE (64 KiB, 256 KiB, 512 KiB). A transparently staged payload is then large enough for the RDMA path from its first byte over the threshold. If rdma_min_bytes is above the transparent threshold, a band of payloads sits in pinned memory and never uses it.

The kill switch

VELO_RDMA_RENDEZVOUS_DISABLE=1 forces RdmaRendezvousConfig::enabled off. Velo reads it once, in VeloBuilder::build, so one process cannot answer half its acquires one way and half the other. Only 1, true, yes, and on (any case) count. Any other value leaves the path on. A switch that fires on a typo silently costs performance, but a missed typo shows in the metric.

The switch acts on both roles. An owner with it on never answers Rdma and stages new pooled slots in plain memory. A consumer with it on never sends an offer. Either side alone turns the path off, so a rollback works one node at a time. Pinned slots keep answering the chunked path, so no staged data becomes unreachable.

Metrics

Both sides record into path_total, at different points:

• The owner records at staging time, when register_data_pinned falls back (kill_switch, budget, pool_exhausted). It also records at each acquire (no_offer, not_configured, kill_switch, not_pinned, below_min, draining, ok).
• The consumer records when it builds its offer (not_configured, kill_switch, no_offer), after a failure (decode_error, pool_exhausted, get_failed), and when a GET completes (ok).

The series counts decisions, not transfers. With the owner's kill switch on, one slot adds kill_switch once at staging and again at each acquire. A fallback acquire carries no offer, so the owner also counts no_offer for a consumer whose GET failed. The owner counts ok when it sends a descriptor, before the GET runs. The consumer's ok is the proof that a GET completed.

This series is the only honest answer to "did the RDMA path run", because the chunked path also returns correct bytes. See Metrics reference.

Not yet built

• PUT in the other direction. The owner writes into a consumer-supplied buffer, flushes, and sends a completion message. The current decoder refuses any non-zero flags, and RdmaOffer carries backend names but no version. A direction flag is therefore not additive: every current peer refuses it and falls back to chunked during a mixed-version window. Either a version field in the offer ships one release ahead, or the rollout accepts chunked fallback everywhere in that window.
• A goodbye message before an idle close. It removes the Messages and pings that a peer loses after the endpoint reaper closes our endpoint to it.
• A deadline for every chunked lease. A chunked lease has a deadline only when its consumer asks for one. A consumer's LeaseGuard releases a chunked lease on every error path except one. That path is a release that cannot spawn because the runtime is shutting down. Nothing on the owner reclaims that lease, unless its slot has a time-to-live.
• A consumer-side unpacked-rkey cache. It removes both halves of the stale-key safety argument at once. An rkey outlives its operation, and the owner no longer revalidates each transfer. It is a safety change, not a profile-driven optimization.
• Register-in-place for large owned buffers. It needs an explicit API. Registering arbitrary caller Bytes by default is unsound: provenance is unknown and UCX pins past the caller's range.
• Metrics: an arena-utilization gauge, and a Prometheus series for endpoints closed by the idle reaper (the count exists only as an internal atomic).
• GPU memory, scatter-gather anchors, and an RDMA surface in velo-ext for out-of-tree transports.