Run rendezvous over RDMA

This guide builds velo with the ucx feature and wires the UCX transport for the rendezvous RDMA path. It then verifies that the path runs on the NIC you expect. For how the path works, see Rendezvous and RDMA.

What you need

• Linux on x86_64 or aarch64. On other platforms the ucx feature builds but adds no transport.
• A C compiler and GNU make. The vendored UCX tarball ships a generated configure, so autotools and cmake are not necessary for this crate.
• The rdma-core development headers. The build looks for three files: infiniband/verbs.h, infiniband/mlx5dv.h, and rdma/rdma_cma.h. On Debian and Ubuntu, libibverbs-dev and librdmacm-dev supply them.
• For real RDMA: an InfiniBand or RoCE port in state PORT_ACTIVE, and a memlock limit large enough for the registered-bytes budget.

The headers are necessary even for a build that only runs over UCX_TLS=tcp. The UCX tests and CI run that way with no RDMA hardware.

Build

1. Install the headers:

   sudo apt-get install -y libibverbs-dev librdmacm-dev
   

2. Turn on the feature in your Cargo.toml:

   velo = { version = "...", features = ["ucx"] }
   

3. Build:

   cargo build --features ucx
   

The first build compiles UCX 1.22.0 from the tarball inside crates/ucx-rs. This takes about 25 s on a 20-core machine (about 73 CPU-seconds). Later builds reuse it in about 0.3 s. A stamp file records the exact configure arguments and toolchain variables, so a change to either rebuilds UCX.

The result links UCX statically. At run time, no UCX shared objects, plugin directories, or LD_LIBRARY_PATH settings exist. The only dynamic dependencies are libibverbs.so.1, libmlx5.so.1, and librdmacm.so.1.

Why the build refuses a TCP-only UCX

If the verbs headers are missing, UCX's own configure prints a warning and builds a UCX with only TCP and shared memory. That build passes CI and then runs production traffic over TCP. ucx-rs refuses instead, with a panic that names the missing header.

For the same reason, ucx-rs passes --with-devx. Upstream defaults DEVX to check, which silently drops the accelerated mlx5 memory domain.

Build variables

The velo feature ucx turns on the ucx-rs default features: ib, rdmacm, and mt. You cannot turn off ib through velo.

Wire the transport

Add UCX with add_ucx_transport, not add_transport. Only add_ucx_transport creates the RDMA registry. Keep a TCP transport beside it for the control plane.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use velo::transports::tcp::TcpTransportBuilder;
use velo::transports::ucx::UcxTransportBuilder;
use velo::{RdmaConfig, Velo};

let tcp = Arc::new(
    TcpTransportBuilder::new()
        .from_listener(std::net::TcpListener::bind("0.0.0.0:0")?)?
        .build()?,
);
// Eager wireup moves the ~14 ms first-GET endpoint cost to register().
let ucx = Arc::new(UcxTransportBuilder::new().eager_endpoints(true).build()?);

let velo = Velo::builder()
    .add_transport(tcp)
    .add_ucx_transport(ucx)
    .rdma_config(RdmaConfig::default())
    .build()
    .await?;

// Owner: stage in registered memory. This call also maps the first arena.
let handle = velo.register_data_pinned(&payload).await;
}

The consumer calls get, get_pinned, or get_into as usual. Register the peers before the first get. A get to an owner that is not registered yet takes the chunked path.

If the only staging in a process is transparent (large messenger payloads), call register_data_pinned once at startup. The transparent stager never maps an arena, so without that call it never uses the RDMA path.

Set the runtime environment

Velo reads the UCX variables from the environment. The environment wins over the builder. UcxTransportBuilder::tls and net_devices apply only when UCX_TLS or UCX_NET_DEVICES is not set.

NOTE: No hardware run has used velo's build on rc_mlx5 yet. All two-node results so far are on rc_verbs (see RDMA performance). On rc_mlx5, do all steps in Verify the lane.

Velo sets UCX_RCACHE_ENABLE=n and UCX_MEM_EVENTS=n unless you set them. With these values, UCX patches no libc functions in the process. Keep them. Velo registers memory explicitly and never uses the UCX registration cache.

Do not set UCX_PROTO_ENABLE=n. It is a common workaround for protocol-v2 faults, and it silently disables all RMA. A multi-rail GET needs UCX_MAX_RMA_RAILS of 2 or more. The default is 1.

Prepare each node

1. Find the port:

   ibv_devinfo
   

2. Pick a port with state: PORT_ACTIVE and link_layer: InfiniBand. An active Ethernet port on the same card carries RoCE and looks like a success.

3. Set UCX_NET_DEVICES to that port, for example mlx5_2:1.

4. Set UCX_TLS. Include a ud transport beside the rc transport.

5. Raise the memlock limit:

   ulimit -l unlimited
   

6. If the IB partition gives this node partial pkey membership (0x7fff), set UCX_IB_PKEY explicitly. UCX_IB_PKEY=auto needs full membership, and raw verbs works where UCX fails.

Some systems ship a memlock limit of 8 MiB. With that limit, ibv_reg_mr fails at 32 MiB, and the first 64 MiB arena cannot map.

Verify the lane

A correct transfer does not prove that RDMA ran. The chunked path also returns correct bytes. Do all three steps.

1. Run with UCX_LOG_LEVEL=debug and find the memory domain line:

   UCX_LOG_LEVEL=debug ./your-binary 2>&1 | grep 'md open by'
   

   On a mlx5 NIC, the line must say uct_ib_mlx5_devx_md_ops. If it says uct_ib_verbs_md_ops, the accelerated transports have no devices.

2. Run with UCX_PROTO_INFO=y. Find the row rendezvous zero-copy read from remote. It must name the transport and device that you set, for example rc_mlx5/mlx5_2:1.

3. Read velo_rendezvous_rdma_path_total on the consumer. The series {path="rdma", reason="ok"} must increase with each transfer. Any other reason names why the path was not used. See Rendezvous and RDMA.

Do not use the system ucx_info to verify velo's build. The system UCX loads its transports as plugins and is a different library. On the same NIC, the system UCX can report rc_mlx5 while velo's static build does not.

Do not trust a configuration header that a program prints. The environment overrides the builder, so a header that says UCX_TLS=tcp can describe a run on InfiniBand. The UCX_PROTO_INFO table is the truth.

Version compatibility

Two velo nodes can talk over UCX only if they use the same UCX wire version. Each node publishes that version in its UCX address, and register() refuses a peer on a different one.

The current version is 2, from velo 0.15.0. Version 2 starts every frame header with the sender's 8-byte incarnation, so that the endpoint idle reaper can tell which peer sent each frame. A version-1 node would misread those 8 bytes, so the two versions refuse each other instead.

Upgrade all UCX peers together. During a rolling upgrade, an old node and a new node cannot reach each other over UCX. register() returns InvalidEndpoint, and the refusing node logs a warning: ucx: rejecting peer blob: unsupported ucx blob version 1 (expected 2).

Run the two-process example

The example stages 8 MiB on an owner and pulls it on a consumer, with a TCP control plane and UCX beside it. The consumer exits non-zero if the transfer did not take the RDMA path.

On one host, over TCP:

UCX_TLS=tcp cargo run --manifest-path examples/Cargo.toml --release --features ucx --example rendezvous_rdma_two_proc

With no --role, the example starts both roles as child processes.

On two nodes:

1. Build on both nodes:

   cargo build --manifest-path examples/Cargo.toml --release --features ucx --example rendezvous_rdma_two_proc
   

2. Pick a directory that both nodes can read and write, for example /shared/rv.

3. Start the owner on node A:

   UCX_NET_DEVICES=mlx5_2:1 UCX_TLS=rc_mlx5,ud_mlx5,self \
     ./examples/target/release/examples/rendezvous_rdma_two_proc --role owner --dir /shared/rv
   

4. Start the consumer on node B, with the same variables and --role consumer.

5. Read the velo_rendezvous_rdma_path_total lines that both sides print at the end.

Use --size <bytes> to change the payload. The two processes find each other through JSON files in --dir. On NFS, negative-dentry caching can delay that exchange by about 30 s. That delay is not a velo latency.

CI runs the single-host form with UCX_TLS=tcp on every pull request.

Run the tests

The UCX suites set tls("tcp") in code. The environment wins, so the same suites run on InfiniBand when UCX_TLS and UCX_NET_DEVICES are set.

# Rendezvous RDMA path, end to end
timeout 300 cargo test -p velo --features ucx,test-helpers --test rendezvous_rdma

# UCX transport and RMA plumbing
timeout 300 cargo test -p velo --features ucx,test-helpers --lib transports::ucx

# Registration layer
timeout 300 cargo test -p velo --features ucx,test-helpers --lib rendezvous::rdma

# The mlx5 memory domain sits ahead of the verbs one
timeout 300 cargo test -p ucx-rs --test ctor_order

# Registration and GET cost (prints numbers, asserts nothing)
timeout 300 cargo test -p velo --features ucx,test-helpers --lib bench_rma -- --ignored --nocapture

CAUTION: Do not run unusable_rkey_is_refused_by_ucx_over_tcp on InfiniBand. It aborts the process with SIGABRT there. Over TCP it passes.

Skip it on an InfiniBand run:

timeout 300 cargo test -p velo --features ucx,test-helpers --lib transports::ucx -- --skip unusable_rkey_is_refused_by_ucx_over_tcp

Common failures