fi_rxm - Man Page

The RxM provider (ofi_rxm) is an utility provider that supports FI_EP_RDM type endpoint emulated over FI_EP_MSG type endpoint(s) of an underlying core provider. FI_EP_RDM endpoints have a reliable datagram interface and RxM emulates this by hiding the connection management of underlying FI_EP_MSG endpoints from the user. Additionally, RxM can hide memory registration requirement from a core provider like verbs if the apps don’t support it.

The RxM provider currently supports FI_MSG, FI_TAGGED, FI_RMA and FI_ATOMIC capabilities.

Endpoint types

The provider supports only FI_EP_RDM.

Endpoint capabilities

The following data transfer interface is supported: FI_MSG, FI_TAGGED, FI_RMA, FI_ATOMIC.

Progress

The RxM provider supports both FI_PROGRESS_MANUAL and FI_PROGRESS_AUTO. Manual progress in general has better connection scale-up and lower CPU utilization since there’s no separate auto-progress thread.

Addressing Formats

FI_SOCKADDR, FI_SOCKADDR_IN

Memory Region

FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, FI_MR_PROV_KEY MR mode bits would be required from the app in case the core provider requires it.

When using RxM provider, some limitations from the underlying MSG provider could also show up. Please refer to the corresponding MSG provider man pages to find about those limitations.

The ofi_rxm provider checks for the following environment variables.

FI_OFI_RXM_BUFFER_SIZE

Defines the transmit buffer size / inject size. Messages of size less than or equal to this would be transmitted via an eager protocol and messages greater in size would be transmitted via a rendezvous or SAR (Segmentation And Reassembly) protocol. Transmit data would be copied up to this size (default: ~16k).

FI_OFI_RXM_COMP_PER_PROGRESS

Defines the maximum number of MSG provider CQ entries (default: 1) that would be read per progress (RxM CQ read).

FI_OFI_RXM_ENABLE_DYN_RBUF

Enables support for dynamic receive buffering, if available by the message endpoint provider. This feature allows direct placement of received message data into application buffers, bypassing RxM bounce buffers. This feature targets providers that provide internal network buffering, such as the tcp provider. (default: false)

FI_OFI_RXM_SAR_LIMIT

Set this environment variable to control the RxM SAR (Segmentation And Reassembly) protocol. Messages of size greater than this (default: 128 Kb) would be transmitted via rendezvous protocol.

FI_OFI_RXM_USE_SRX

Set this to 1 to use shared receive context from MSG provider, or 0 to disable using shared receive context. Shared receive contexts reduce overall memory usage, but may increase in message latency. If not set, verbs will not use shared receive contexts by default, but the tcp provider will.

FI_OFI_RXM_TX_SIZE

Defines default TX context size (default: 1024)

FI_OFI_RXM_RX_SIZE

Defines default RX context size (default: 1024)

FI_OFI_RXM_MSG_TX_SIZE

Defines FI_EP_MSG TX size that would be requested (default: 128).

FI_OFI_RXM_MSG_RX_SIZE

Defines FI_EP_MSG RX size that would be requested (default: 128).

FI_UNIVERSE_SIZE

Defines the expected number of ranks / peers an endpoint would communicate with (default: 256).

FI_OFI_RXM_CM_PROGRESS_INTERVAL

Defines the duration of time in microseconds between calls to RxM CM progression functions when using manual progress. Higher values may provide less noise for calls to fi_cq read functions, but may increase connection setup time (default: 10000)

FI_OFI_RXM_CQ_EQ_FAIRNESS

Defines the maximum number of message provider CQ entries that can be consecutively read across progress calls without checking to see if the CM progress interval has been reached (default: 128)

FI_OFI_RXM_DETECT_HMEM_IFACE

Set this to 1 to allow automatic detection of HMEM iface of user buffers when such information is not supplied. This feature allows such buffers be copied or registered (e.g. in Rendezvous) internally by RxM. Note that no extra memory registration is performed with this option. (default: false)

The data transfer API may return -FI_EAGAIN during on-demand connection setup of the core provider FI_MSG_EP. See fi_msg(3) for a detailed description of handling FI_EAGAIN.

If an RxM endpoint is expected to communicate with more peers than the default value of FI_UNIVERSE_SIZE (256) CQ overruns can happen. To avoid this set a higher value for FI_UNIVERSE_SIZE. CQ overrun can make a MSG endpoint unusable.

At higher # of ranks, there may be connection errors due to a node running out of memory. The workaround is to use shared receive contexts for the MSG provider (FI_OFI_RXM_USE_SRX=1) or reduce eager message size (FI_OFI_RXM_BUFFER_SIZE) and MSG provider TX/RX queue sizes (FI_OFI_RXM_MSG_TX_SIZE / FI_OFI_RXM_MSG_RX_SIZE).

fabric(7), fi_provider(7), fi_getinfo(3)

OpenFabrics.

fi_provider(7), fi_verbs(7).