xprtrdma: Reduce max_frwr_depth (a7886849) · Commits · 戴 / test

Some devices advertise a large max_fast_reg_page_list_len capability, but perform optimally when MRs are significantly smaller than that depth -- probably when the MR itself is no larger than a page. By default, the RDMA R/W core API uses max_sge_rd as the maximum page depth for MRs. For some devices, the value of max_sge_rd is 1, which is also not optimal. Thus, when max_sge_rd is larger than 1, use that value. Otherwise use the value of the max_fast_reg_page_list_len attribute. I've tested this with CX-3 Pro, FastLinq, and CX-5 devices. It reproducibly improves the throughput of large I/Os by several percent. Signed-off-by:

Chuck Lever <chuck.lever@oracle.com> Signed-off-by:

Anna Schumaker <Anna.Schumaker@Netapp.com>

net/sunrpc/xprtrdma/frwr_ops.c

+11 −4

Original line number	Diff line number	Diff line
		@@ -193,10 +193,17 @@ frwr_op_open(struct rpcrdma_ia ia, struct rpcrdma_ep ep,
		if (attrs->device_cap_flags & IB_DEVICE_SG_GAPS_REG)
		ia->ri_mrtype = IB_MR_TYPE_SG_GAPS;

		ia->ri_max_frwr_depth =
		min_t(unsigned int, RPCRDMA_MAX_DATA_SEGS,
		attrs->max_fast_reg_page_list_len);
		dprintk("RPC: %s: device's max FR page list len = %u\n",
		/* Quirk: Some devices advertise a large max_fast_reg_page_list_len
		* capability, but perform optimally when the MRs are not larger
		* than a page.
		*/
		if (attrs->max_sge_rd > 1)
		ia->ri_max_frwr_depth = attrs->max_sge_rd;
		else
		ia->ri_max_frwr_depth = attrs->max_fast_reg_page_list_len;
		if (ia->ri_max_frwr_depth > RPCRDMA_MAX_DATA_SEGS)
		ia->ri_max_frwr_depth = RPCRDMA_MAX_DATA_SEGS;
		dprintk("RPC: %s: max FR page list depth = %u\n",
		__func__, ia->ri_max_frwr_depth);

		/* Add room for frwr register and invalidate WRs.

Admin message