svcrdma: Use rdma_rw API in RPC reply path (9a6a180b) · Commits · 戴 / test

The current svcrdma sendto code path posts one RDMA Write WR at a time. Each of these Writes typically carries a small number of pages (for instance, up to 30 pages for mlx4 devices). That means a 1MB NFS READ reply requires 9 ib_post_send() calls for the Write WRs, and one for the Send WR carrying the actual RPC Reply message. Instead, use the new rdma_rw API. The details of Write WR chain construction and memory registration are taken care of in the RDMA core. svcrdma can focus on the details of the RPC-over-RDMA protocol. This gives three main benefits: 1. All Write WRs for one RDMA segment are posted in a single chain. As few as one ib_post_send() for each Write chunk. 2. The Write path can now use FRWR to register the Write buffers. If the device's maximum page list depth is large, this means a single Write WR is needed for each RPC's Write chunk data. 3. The new code introduces support for RPCs that carry both a Write list and a Reply chunk. This combination can be used for an NFSv4 READ where the data payload is large, and thus is removed from the Payload Stream, but the Payload Stream is still larger than the inline threshold. Signed-off-by:

Chuck Lever <chuck.lever@oracle.com> Signed-off-by:

J. Bruce Fields <bfields@redhat.com>

include/linux/sunrpc/svc_rdma.h

+0 −1

Original line number	Diff line number	Diff line
		@@ -212,7 +212,6 @@ extern int svc_rdma_xdr_decode_req(struct xdr_buf *);
		extern int svc_rdma_xdr_encode_error(struct svcxprt_rdma *,
		struct rpcrdma_msg *,
		enum rpcrdma_errcode, __be32 *);
		extern void svc_rdma_xdr_encode_write_list(struct rpcrdma_msg *, int);
		extern void svc_rdma_xdr_encode_reply_array(struct rpcrdma_write_array *, int);
		extern void svc_rdma_xdr_encode_array_chunk(struct rpcrdma_write_array *, int,
		__be32, __be64, u32);

net/sunrpc/xprtrdma/svc_rdma_backchannel.c

+3 −3

Original line number	Diff line number	Diff line
		@@ -90,9 +90,9 @@ out_notfound:
		* Caller holds the connection's mutex and has already marshaled
		* the RPC/RDMA request.
		*
		* This is similar to svc_rdma_reply, but takes an rpc_rqst
		* instead, does not support chunks, and avoids blocking memory
		* allocation.
		* This is similar to svc_rdma_send_reply_msg, but takes a struct
		* rpc_rqst instead, does not support chunks, and avoids blocking
		* memory allocation.
		*
		* XXX: There is still an opportunity to block in svc_rdma_send()
		* if there are no SQ entries to post the Send. This may occur if

net/sunrpc/xprtrdma/svc_rdma_sendto.c

+345 −351

File changed.

Preview size limit exceeded, changes collapsed.

net/sunrpc/xprtrdma/svc_rdma_transport.c

+2 −0

Original line number	Diff line number	Diff line
		@@ -1053,6 +1053,8 @@ static struct svc_xprt svc_rdma_accept(struct svc_xprt xprt)
		memset(&qp_attr, 0, sizeof qp_attr);
		qp_attr.event_handler = qp_event_handler;
		qp_attr.qp_context = &newxprt->sc_xprt;
		qp_attr.port_num = newxprt->sc_cm_id->port_num;
		qp_attr.cap.max_rdma_ctxs = newxprt->sc_max_requests;
		qp_attr.cap.max_send_wr = newxprt->sc_sq_depth;
		qp_attr.cap.max_recv_wr = newxprt->sc_rq_depth;
		qp_attr.cap.max_send_sge = newxprt->sc_max_sge;

Admin message