rcu: Prevent rcu_barrier() from starting needless grace periods (f92c734f) · Commits · 戴 / test

Currently rcu_barrier() uses call_rcu() to enqueue new callbacks on each CPU with a non-empty callback list. This works, but means that rcu_barrier() forces grace periods that are not otherwise needed. The key point is that rcu_barrier() never needs to wait for a grace period, but instead only for all pre-existing callbacks to be invoked. This means that rcu_barrier()'s new callbacks should be placed in the callback-list segment containing the last pre-existing callback. This commit makes this change using the new rcu_segcblist_entrain() function. Signed-off-by:

Paul E. McKenney <paulmck@linux.vnet.ibm.com>

include/trace/events/rcu.h

+1 −0

Original line number	Diff line number	Diff line
		@@ -742,6 +742,7 @@ TRACE_EVENT(rcu_torture_read,
		* "OnlineQ": _rcu_barrier() found online CPU with callbacks.
		* "OnlineNQ": _rcu_barrier() found online CPU, no callbacks.
		* "IRQ": An rcu_barrier_callback() callback posted on remote CPU.
		* "IRQNQ": An rcu_barrier_callback() callback found no callbacks.
		* "CB": An rcu_barrier_callback() invoked a callback, not the last.
		* "LastCB": An rcu_barrier_callback() invoked the last callback.
		* "Inc2": _rcu_barrier() piggyback check counter incremented.

kernel/rcu/tree.c

+8 −2

Original line number	Diff line number	Diff line
		@@ -3578,8 +3578,14 @@ static void rcu_barrier_func(void *type)
		struct rcu_data *rdp = raw_cpu_ptr(rsp->rda);

		_rcu_barrier_trace(rsp, "IRQ", -1, rsp->barrier_sequence);
		rdp->barrier_head.func = rcu_barrier_callback;
		debug_rcu_head_queue(&rdp->barrier_head);
		if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head, 0)) {
		atomic_inc(&rsp->barrier_cpu_count);
		rsp->call(&rdp->barrier_head, rcu_barrier_callback);
		} else {
		debug_rcu_head_unqueue(&rdp->barrier_head);
		_rcu_barrier_trace(rsp, "IRQNQ", -1, rsp->barrier_sequence);
		}
		}

		/*

Admin message