What is the RDMA Protocol?
The RDMA protocol involves the following high-level steps for the exchange of data from one memory location at one server to a memory location on another server:
1. Connection Establishment
– Two devices establish a connection using RDMA-capable hardware (e.g., NICs supporting RDMA).
– Both devices create queue pairs (send queue, receive queue and completion queue).
2. Memory Registration
– The sending and receiving devices register their memory regions with the RDMA hardware.
– Registered memory regions get pinned, meaning they cannot be moved or swapped by the OS during RDMA operations.
3. Exchange of Memory Keys
– Devices exchange memory keys (e.g., Remote Memory Address and access permissions) that allow direct access to the registered memory regions.
4. Data Transfer
– One device directly reads from or writes to the remote device’s memory, bypassing the CPU on both ends.
– Transfers occur with minimal latency since there is no involvement of intermediate buffers or kernel context switching.
5. Completion Notification
– RDMA hardware informs the application when the operation is completed, ensuring reliability and consistency.
6. Connection Teardown
– Once data exchange is complete, the RDMA connection is closed, and memory regions are deregistered.
The different types of packets exchanged between the two devices are encoded in the Base Transport Header (BTH). Since the data exchange is described in bullet 4 above, this is where performance testing becomes especially important. Data can be exchanged as either IPv4 or IPv6 and as Reliable Connection (RC), Reliable Datagram (RD), Unreliable Connection (UC) and Unreliable Datagram (UD). Furthermore, every data exchange consists of a SEND FIRST frame, several SEND MIDDLE frames and a SEND LAST frame.
This is shown in Figure 2 below, for the case of a RC SEND operation. The figure also illustrates how the BTH Opcode encodes the three types of RC SEND frames.
