Commit 8700e3e7 authored Jun 16, 2016 by Moni Shoua Committed by Doug Ledford Aug 04, 2016

Soft RoCE driver

Soft RoCE (RXE) - The software RoCE driver

ib_rxe implements the RDMA transport and registers to the RDMA core
device as a kernel verbs provider. It also implements the packet IO
layer. On the other hand ib_rxe registers to the Linux netdev stack
as a udp encapsulating protocol, in that case RDMA, for sending and
receiving packets over any Ethernet device.  This yields a RDMA
transport over the UDP/Ethernet network layer forming a RoCEv2
compatible device.

The configuration procedure of the Soft RoCE drivers requires
binding to any existing Ethernet network device. This is done with
/sys interface.

A userspace Soft RoCE library (librxe) provides user applications
the ability to run with Soft RoCE devices.  The use of rxe verbs ins
user space requires the inclusion of librxe as a device specifics
plug-in to libibverbs. librxe is packaged separately.

Architecture:

     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
+---------------------------------------------------------------+
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Soft RoCE resources:

[1[ https://github.com/SoftRoCE/librxe-dev librxe - source code in
Github
[2] https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home - Soft RoCE
Wiki page
[3] https://github.com/SoftRoCE/librxe-dev

 - Soft RoCE userspace library

Signed-off-by: Kamal Heib <kamalh@mellanox.com>
Signed-off-by: Amir Vadai <amirv@mellanox.com>
Signed-off-by: Moni Shoua <monis@mellanox.com>
Reviewed-by: Haggai Eran <haggaie@mellanox.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>

parent 33688abb

MAINTAINERS

+9 −0

Original line number	Diff line number	Diff line
		@@ -7444,6 +7444,15 @@ W: http://www.mellanox.com
		Q: http://patchwork.ozlabs.org/project/netdev/list/
		F: drivers/net/ethernet/mellanox/mlxsw/

		SOFT-ROCE DRIVER (rxe)
		M: Moni Shoua <monis@mellanox.com>
		L: linux-rdma@vger.kernel.org
		S: Supported
		W: https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
		Q: http://patchwork.kernel.org/project/linux-rdma/list/
		F: drivers/infiniband/hw/rxe/
		F: include/uapi/rdma/rdma_user_rxe.h

		MEMBARRIER SUPPORT
		M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
		M: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

drivers/infiniband/Kconfig

+1 −0

Original line number	Diff line number	Diff line
		@@ -84,6 +84,7 @@ source "drivers/infiniband/ulp/iser/Kconfig"
		source "drivers/infiniband/ulp/isert/Kconfig"

		source "drivers/infiniband/sw/rdmavt/Kconfig"
		source "drivers/infiniband/sw/rxe/Kconfig"

		source "drivers/infiniband/hw/hfi1/Kconfig"

drivers/infiniband/sw/Makefile

+1 −0

Original line number	Diff line number	Diff line
		obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt/
		obj-$(CONFIG_RDMA_RXE) += rxe/

drivers/infiniband/sw/rxe/Kconfig

0 → 100644

+24 −0

Original line number	Diff line number	Diff line
		config RDMA_RXE
		tristate "Software RDMA over Ethernet (RoCE) driver"
		depends on INET && PCI && INFINIBAND
		depends on NET_UDP_TUNNEL
		---help---
		This driver implements the InfiniBand RDMA transport over
		the Linux network stack. It enables a system with a
		standard Ethernet adapter to interoperate with a RoCE
		adapter or with another system running the RXE driver.
		Documentation on InfiniBand and RoCE can be downloaded at
		www.infinibandta.org and www.openfabrics.org. (See also
		siw which is a similar software driver for iWARP.)

		The driver is split into two layers, one interfaces with the
		Linux RDMA stack and implements a kernel or user space
		verbs API. The user space verbs API requires a support
		library named librxe which is loaded by the generic user
		space verbs API, libibverbs. The other layer interfaces
		with the Linux network stack at layer 3.

		To configure and work with soft-RoCE driver please use the
		following wiki page under "configure Soft-RoCE (RXE)" section:

		https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

drivers/infiniband/sw/rxe/Makefile

0 → 100644

+24 −0

Original line number	Diff line number	Diff line
		obj-$(CONFIG_RDMA_RXE) += rdma_rxe.o

		rdma_rxe-y := \
		rxe.o \
		rxe_comp.o \
		rxe_req.o \
		rxe_resp.o \
		rxe_recv.o \
		rxe_pool.o \
		rxe_queue.o \
		rxe_verbs.o \
		rxe_av.o \
		rxe_srq.o \
		rxe_qp.o \
		rxe_cq.o \
		rxe_mr.o \
		rxe_dma.o \
		rxe_opcode.o \
		rxe_mmap.o \
		rxe_icrc.o \
		rxe_mcast.o \
		rxe_task.o \
		rxe_net.o \
		rxe_sysfs.o

Admin message