RDMA-Based RPC

This is a quite simple RDMA-Based RPC implementation used in DPUKV.

Prerequisite

• Build tool: meson, ninja

• Dependent library: libibverbs, librdmacm, libevent, pthread

• Dependent Environment: RDMA-enabled NIC, hugepages

If not, use Soft-RoCE instead. Follow the commands bellow.

modprobe rdma_rxe
# bind a normal NIC
sudo rdma link add rxe_0 type rxe netdev <your NIC name>
# use `ip addr` to get the name
ibv_devices # you will see a virtualized NIC named rxe_0

How to use

Build

It is better to use hugepages for NIC because of IOMMU. You can use option 'use_hugepage' to use anonymous hugepages as buffer.

meson build -Duse_hugepage=enabled
meson compile -C build

Play

# start the server
./build/app/server <NIC ip> <port>

# start the client
./build/app/client <server ip> <server port>

Design

Connection Management

Register the rdma_cm_event_channel's fd into event loop.

RPC Context FSM

One RPC Procedure

sequenceDiagram
    participant Client
    participant Server
    Server->>Server: Pre-post Recv for Meta
    Client->>Server: Post Send, Send Meta
    Client->>Client: Pre-post Recv for Response
    Server->>Client: Post Read, Read Request
    Server->>Server: Call Handler
    Server->>Client: Post Write, Write Response

Loading

Client Side Context

stateDiagram
    Vacant --> SendingBufferMeta: Fill with request and send meta.
    SendingBufferMeta --> WaitingForResponse: Completion event of sending meta.
    WaitingForResponse --> Vacant: Completion event of remote writing response.

Loading

Server Side Context

stateDiagram
    Vacant --> WaitingForBufferMeta: Initialize and wait for meta.
    WaitingForBufferMeta --> ReadingRequest: Receive meta and read request.
    WaitingForBufferMeta --> FilledWithRequest: Receive meta and immediate request.
    ReadingRequest --> FilledWithRequest: Completion event of reading request.
    FilledWithRequest --> FilledWithResponse: Push into queue for handler thread.
    FilledWithResponse --> WritingResponse: Fill with resposne and write response.
    WritingResponse --> WaitingForBufferMeta: Completion event of writing response.

Loading

To-do

• basic connection management
• add send and recv verbs
• add read and write verbs
• add ring for multi-client-calls
• add background handler
• add memory pool
• add raw api
• use one poller for all connections
• accelerate small rpc
• split request and replay

Bug

• when serving multiple connections, server may go wrong.

Reference

• Note
• RDMA Aware Networks Programming User Manual