Metrics reference

This page lists every Prometheus metric family that VeloMetrics registers. The description column is the help text from lib/velo/src/observability.rs. A histogram exposes _bucket, _sum, and _count series. For how to read the families together, see Observability. For the batched streaming families, see Stream saturation.

Transport

Metric	Type	Labels	Description
`velo_transport_frames_total`	counter	`transport`, `direction`, `message_type`, `outcome`	Transport frames observed by Velo.
`velo_transport_frame_bytes_total`	counter	`transport`, `direction`, `message_type`	Logical frame bytes observed by Velo transports.
`velo_transport_frames_written_total`	counter	`transport`, `message_type`	Frames a transport's per-connection writer handed to its send buffer (the kernel socket buffer for TCP and UDS, quinn's user-space buffer for QUIC), counted once the write returned. Subtract this from velo_transport_frames_total{direction="outbound",outcome="accepted"} on the same transport to get that transport's egress queue depth. Published only by the coalescing writer of the TCP, UDS and QUIC transports — the `transport` label is whatever TransportKey the transport was built with, not a fixed name, so select on this series' presence rather than on a transport-name pattern; see the book's Observability chapter for the identity's limits.
`velo_transport_egress_queue_wait_seconds`	histogram	`transport`	Time one outbound frame spent between the transport's send attempt and its connection's writer taking it off the send queue. Stamped before admission, so it covers the admission gate's pending queue as well as the bounded per-connection channel behind it. Observed once per frame, by the writer, at the dequeue — including a frame whose write then failed, so this count is at least velo_transport_frames_written_total and equals it only on a connection that never faulted. Published only by the coalescing writer of the TCP, UDS and QUIC transports — select on this series' presence, not on the `transport` label's value.
`velo_transport_write_duration_seconds`	histogram	`transport`	Wall time of one write on a connection's writer. One observation per write, and a write carries as many frames as the writer had queued — so this count is at most velo_transport_frames_written_total and equals it only when nothing ever coalesced. Large values mean the socket's send buffer is full and the wire or the receiver is the constraint; small values beside a large velo_transport_egress_queue_wait_seconds mean the writer is starved or the queue is simply long. Published only by the coalescing writer of the TCP, UDS and QUIC transports — select on this series' presence, not on the `transport` label's value.
`velo_transport_rejections_total`	counter	`transport`, `reason`	Transport-level rejections and drops observed by Velo.
`velo_transport_send_backpressure_total`	counter	`transport`	Sends that found the bounded channel for a target (and lane) full and were queued in its admission gate.
`velo_transport_registered_peers`	gauge	`transport`	Registered peer count per transport.
`velo_transport_active_connections`	gauge	`transport`	Active connection count per transport.

Messenger

Metric	Type	Labels	Description
`velo_messenger_handler_requests_total`	counter	`handler`, `response_type`, `outcome`	Messenger handler requests observed by Velo.
`velo_messenger_handler_duration_seconds`	histogram	`handler`, `response_type`, `outcome`	Messenger handler execution time.
`velo_messenger_handler_request_bytes_total`	counter	`handler`, `response_type`	Messenger handler request payload bytes.
`velo_messenger_handler_response_bytes_total`	counter	`handler`, `response_type`, `outcome`	Messenger handler response payload bytes.
`velo_messenger_handler_in_flight`	gauge	`handler`	Messenger handlers currently executing.
`velo_messenger_ordered_lanes`	gauge	`handler`	Live ordering lanes held by ordered-dispatch handlers.
`velo_messenger_ordered_lane_depth`	gauge	`handler`	Messages enqueued on ordering lanes but not yet handled.
`velo_messenger_ordered_lane_wait_seconds`	histogram	`handler`	Time a message waited on an ordering lane before its handler started.
`velo_messenger_ordered_lanes_created_total`	counter	`handler`	Ordering lanes created. Subtracting the live lane gauge yields lanes reaped.
`velo_messenger_dispatch_failures_total`	counter	`stage`, `reason`	Messenger dispatch and response path failures.
`velo_messenger_client_resolution_total`	counter	`path`, `outcome`	Messenger client routing path outcomes.
`velo_messenger_pending_responses`	gauge	—	Pending messenger response awaiters.
`velo_messenger_response_slot_exhausted_total`	counter	—	Response slot acquisitions that hit the per-worker capacity ceiling.
`velo_messenger_inbound_dequeued_total`	counter	—	Messages the dispatch loop took off the messenger's inbound queue. Subtract this from velo_transport_frames_total{direction="inbound",message_type="message",outcome="accepted"} to get that queue's depth — there is deliberately no gauge for it, because a sampled channel length reads the wrong number under exactly the load that makes the depth interesting. Sum the frame counter across transports before subtracting: it is per transport and this one is per instance. The identity holds while the instance is live — messages abandoned when a Timeout shutdown tears the dispatch loop down are never counted here, so from teardown onward the derived depth reads high by the abandoned count.

Streaming

Metric	Type	Labels	Description
`velo_streaming_anchor_operations_total`	counter	`operation`, `outcome`, `transport_scheme`	Streaming control-plane operations observed by Velo.
`velo_streaming_anchor_operation_duration_seconds`	histogram	`operation`, `outcome`, `transport_scheme`	Streaming control-plane operation duration.
`velo_streaming_anchor_attach_rtt_seconds`	histogram	`outcome`, `transport_scheme`	Wall time a remote anchor attach spent in flight, measured by the sender across the round trip to either the _anchor_attach or _mpsc_anchor_attach handler — the two attach kinds share this one series, so a subtraction against it is a population average, not a per-kind reading. Its excess over velo_streaming_anchor_operation_duration_seconds{operation="attach"} on the receiver, which both handlers record into as well, bounds that receiver's ingest queueing from above; the sender's own send path, both wire legs, the receiver's handler-spawn delay and the sender task's wake latency are inside the bracket too. The two are not directly comparable on their labels: this series' transport_scheme is the sender's own advertised-vs-answered view, which folds to "unknown" on an ordinary mixed-deployment negotiation rather than naming the key the receiver actually used, and the two histograms live in different registries on different nodes, so a comparison must sum both labels away rather than match on them, and covers only the outcome="success" population.
`velo_streaming_active_anchors`	gauge	—	Anchors currently present in the streaming registry.
`velo_streaming_backpressure_total`	counter	`transport_scheme`	Streaming backpressure events observed by Velo.
`velo_streaming_reader_pump_backpressure_total`	counter	—	Reader-pump forwards that hit the per-anchor frame channel's Full branch and fell through to send_async. Leading indicator of consumer-side saturation: this channel (bounded(256)) is the first to fill in the saturation cascade. Per-stream transports only: a messenger-mux stream has no reader pump (its consumer reads the slot buffer directly), so this never moves for it; watch velo_streaming_slot_credit_exhausted_total there, which the producer's node counts.
`velo_streaming_server_pump_backpressure_total`	counter	—	Server-side stream pump forwards that hit the bind-side frame channel's Full branch (bounded(4096)) and fell through to send_async. Ticks once the per-anchor channel is saturated and the transport-level channel begins to fill.
`velo_streaming_producer_send_backpressure_total`	counter	—	StreamSender::send calls that found the connect-side channel full and waited in send_async. The channel is 4096 deep on the per-stream path and C+1 deep under the mux, where it fills when the slot reaches its byte cap or the batcher waits on admission.
`velo_streaming_heartbeat_watchdog_firings_total`	counter	—	Heartbeat-watchdog firings: a session went DETECTION_MULTIPLIER × heartbeat_deadline with no data or heartbeat frames and was force-cleaned with a Dropped sentinel. Fired by the reader pump on a per-stream transport, where it is typically the lagging indicator of the cascade surfaced by the *_backpressure_total counters above. Fired by the stream watchdog on a single-sender messenger-mux stream, and only when nothing was delivered to the slot while its sender held data credit or had records waiting behind a sequence gap, so there it means silence upstream of the consumer (a dead or stalled producer, or a backlog on its egress or the peer link), never a consumer that fell behind. An MPSC anchor's reader pump, on any transport, does not record this counter.
`velo_streaming_unclaimed_bind_reaped_total`	counter	—	Reap of a mux bind, by its stream watchdog or its consumer (a zero-RTT pre-bind, an ordinary attach whose peer never sent its OpenSlot, or an attach that adopted a pre-bind) whose accept window closed with no sender having claimed it. Distinct from the heartbeat-watchdog firing above: this is the accept window catching what the watchdog either cannot see yet (no sender exists) or would catch too late (the watchdog's threshold exceeds the accept window's remaining span, at heartbeat_interval >= 20s, give or take one credit_sweep_interval, since the accept window closes on the sweep tick).
`velo_streaming_egress_flushes_total`	counter	—	Batches handed to the socket by the per-stream egress pump. Divide velo_streaming_frames_written_total by this for the coalescing ratio: 1.0 means every frame went out on its own, higher means frames are being packed together. This counts batches, not syscalls or TCP segments — write_all may loop over several underlying writes, a frame too large to pack is written segmented and still counts as one, and segmentation is the kernel's decision. Only batches that reached the wire are counted.
`velo_streaming_frames_written_total`	counter	—	Logical stream frames written to the wire. See velo_streaming_egress_flushes_total for the coalescing ratio.
`velo_streaming_slot_credit_exhausted_total`	counter	—	Times a slot ran out of data credit and its egress parked. Per-slot starvation, not peer-wide: other slots on the same batcher keep flowing, which is the property multiplexing has to buy back after giving up one socket per stream.

Batched streaming (mux)

Metric	Type	Labels	Description
`velo_streaming_mux_live_slots`	gauge	—	Slots the messenger mux currently holds state for, counting both sides: an egress slot allocated by connect() and an ingress slot created by an OpenSlot record each add one. Must return to zero once every stream has torn down — a leaked slot leaks credit and byte budget for the life of the epoch, and unlike a leaked socket it is invisible to lsof.
`velo_streaming_mux_reader_stall_total`	counter	—	Records the applier could not try_send into a slot buffer that credit had already reserved space in. Should always be zero: a slot never has more than C frames outstanding against a C+1-deep buffer, so a non-zero value means the credit invariant is broken. A bug, not a tuning signal.
`velo_streaming_mux_generation_mismatch_total`	counter	—	Records dropped because the slot index was live but at a different generation. This is dense slot reuse being caught: without the generation tag these records would have been delivered into whichever stream now occupies the index.
`velo_streaming_mux_records_dropped_total`	counter	`reason`	Mux records discarded before delivery, by reason. stale_epoch and generation are the protocol working as designed across a reconnect or a slot reuse; unknown_slot and closed_slot mean a sender is talking about a stream this node does not have; lane_mismatch counts a batch whose header lane is not the lane of the handler it arrived on, which is a sender fault; slot_collision counts an `OpenSlot` at an index whose slot has not been closed, which is a sender fault too, or one whose slot id came round while the old slot with that id still waits for a rendezvous record.
`velo_streaming_mux_batches_total`	counter	`direction`	_stream_batch active messages packed (sent) or decoded (received) by the mux.
`velo_streaming_mux_records_per_batch`	histogram	`direction`	Records carried by one _stream_batch, by direction. This is the multiplexing win expressed directly: 1.0 means the mux is paying protocol overhead for nothing. Every node both sends and receives batches — credit rides back on them — so an unlabelled sum here would mix a node's packing with its peers'.
`velo_streaming_mux_rendezvous_singletons_total`	counter	—	Records larger than the effective eager budget, sent alone in their batch so the messenger stages them through rendezvous. Each one pays a round trip and fences its own slot until the admission resolves; a rising rate means frames are outgrowing the target's eager budget.
`velo_streaming_mux_held_records`	gauge	—	Records sitting in per-slot ahead-of-sequence holds, waiting for a frame_seq gap to close. Non-zero only while a rendezvous singleton is outstanding, since that is the one path that can overtake the ordered lane.
`velo_streaming_mux_withheld_records`	gauge	—	Records the egress batcher has pulled from producer inlets but may not send yet — a slot out of credit, or fencing a singleton (a rendezvous transfer, or an OpenSlot under MuxConfig::async_open_ack) whose admission has not resolved. Bounded per slot by the slot byte cap: at the cap the slot stops pulling from its inlet, and the producer's send waits.
`velo_streaming_mux_staged_records`	gauge	—	Records packed into batches the egress batchers have open but have not written. Transient under `FlushPolicy::Auto` with `on_admission` set, where every wake that stages anything besides credit replies ends in a write and a batch holding only credit replies is held for up to `MuxConfig::reply_linger`. Under `FlushPolicy::Manual`, or `Auto { on_admission: false, max_linger: None }` — the two policies with no timer of their own — a pending credit reply still holds the whole batch it is in for that same bound, data included, so a plateau beyond it is a producer that stopped calling `flush_batch` (under `Auto { max_linger: Some(_) }` instead, the same plateau means that window's own timer stalled). Bounded by the batch clamps, so it costs latency rather than memory.
`velo_streaming_mux_control_refused_total`	counter	—	Coalesced control entries a peer batcher refused, for either of two reasons: a grant or close named a slot index this side never allocated, or a burst of `OpenSlot`s this side never admitted filled the bounded lane that carries their rejections. Either way the peer's own accounting is not at risk: a real held-slot credit or close reply is never refused. Not counted here: an ordinary stale-generation race, where a grant or close names an index this side allocated but at a generation that is not the one currently live there — that is dropped silently rather than signalled, since it is expected traffic rather than a hostile peer.
`velo_streaming_mux_hold_overflow_total`	counter	—	Slots closed because their ahead-of-sequence hold exceeded the credit or byte budget backing it. Scoped to the one slot: the peer's ordering lane and its other slots are untouched.
`velo_streaming_mux_epoch_deaths_total`	counter	—	Peer epochs that died, each failing every live slot beneath it exactly once. Driven by a batch send that was never admitted, which is the only ordered per-target congestion and liveness signal the mux has.
`velo_streaming_mux_credit_reposted_total`	counter	—	Ingress credit that was minted into a batch the epoch then threw away, and handed back to the control state so a later batch re-advertises it. `take_pending_grant` zeroes the slot's `ungranted` at mint time, so without this the sender's window would shrink by the delta for the life of the slot.
`velo_streaming_mux_credit_lost_total`	counter	—	Ingress credit from a discarded batch that reached neither the batcher's own control state nor the batcher that took the peer over. The sender's window for that slot is short by this much until the slot closes. Expected to stay at zero.
`velo_streaming_mux_batch_seq_gaps_total`	counter	—	Batches missing between the expected and received batch_seq within one epoch. Reported and moved past — the mux does not retransmit — so this is the count of batches whose records the per-slot frame_seq machinery had to recover from. A batch that arrives after its successor was already counted as missing is not counted again: the meter keeps a high-water mark, so a reordered pair costs one, never a wrapped u32.
`velo_streaming_mux_drain_visits_total`	counter	—	Credit reconciles the sweep task ran on one (peer, lane) because a consumer drained, counted once per walk actually performed. Wakes deferred by MuxConfig::drain_visit_floor are not counted until the walk they coalesced into runs, and the periodic sweep's own walks are not counted at all, so this divided by elapsed time is the doorbell's real visit rate and is bounded above by 1/drain_visit_floor per (peer, lane).
`velo_streaming_mux_records_sent_total`	counter	`record_type`	Mux records a batcher packed for its peer, by record type — this series only ever describes what a node sends, so it carries no direction label of its own. Compared with velo_streaming_mux_batches_total{direction="sent"} it says what those outbound batches are made of, which is the question a batch count alone cannot answer: a rise in small batches is either data arriving one record at a time or control flushed as it comes.
`velo_streaming_mux_batcher_wakes_total`	counter	`source`	Wakes of the per-peer batcher tasks, by what woke them: a producer's open, coalesced control, a queued record, a departed producer, or the linger timer. This attributes what woke the task, not what it wrote: one wake can still write more than one batch (a size clamp or an oversized record flushes inline), and a batch held open only by `MuxConfig::reply_linger` writes on the first end-of-wake check after the window elapses — a `linger` wake only when the batcher was otherwise idle at the deadline, a `control` or `frame` wake when one arrives first (measured 21-30% linger on the tier-3 frontend).

Rendezvous and RDMA

Metric	Type	Labels	Description
`velo_rendezvous_operations_total`	counter	`operation`, `outcome`	Rendezvous operations observed by Velo.
`velo_rendezvous_operation_duration_seconds`	histogram	`operation`, `outcome`	Duration of rendezvous operations.
`velo_rendezvous_bytes_total`	counter	`operation`	Total bytes transferred via rendezvous.
`velo_rendezvous_active_slots`	gauge	—	Number of data slots currently staged in the rendezvous registry.
`velo_rdma_registered_bytes`	gauge	—	Bytes currently registered for RDMA (arena pool plus external regions).
`velo_rdma_registrations_total`	counter	`kind`	RDMA memory registrations performed by Velo.
`velo_rdma_live_regions`	gauge	—	Memory regions the RDMA backend currently holds registered.
`velo_rendezvous_rdma_path_total`	counter	`path`, `reason`	Rendezvous path decisions, by the path chosen and the reason. Owner and consumer both record, so one transfer can count more than once.
`velo_rendezvous_rdma_get_duration_seconds`	histogram	—	Duration of the RDMA GET inside a rendezvous get, excluding the acquire round trip.
`velo_rendezvous_rdma_leases_reaped_total`	counter	—	RDMA leases force-released by the owner-side reaper after their deadline passed.