Commit b32e9eb6 authored Nov 12, 2009 by Paul E. McKenney Committed by Ingo Molnar Nov 13, 2009

rcu: Accelerate callback processing on CPUs not detecting GP end



An earlier fix for a race resulted in a situation where the CPUs
other than the CPU that detected the end of the grace period would
not process their callbacks until the next grace period started.

This means that these other CPUs would unnecessarily demand that an
extra grace period be started.

This patch eliminates this extra grace period and speeds callback
processing by propagating rsp->completed to the rcu_node structures
in the case where the CPU detecting the end of the grace period
sees no reason to start a new grace period.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: laijs@cn.fujitsu.com
Cc: dipankar@in.ibm.com
Cc: mathieu.desnoyers@polymtl.ca
Cc: josh@joshtriplett.org
Cc: dvhltc@us.ibm.com
Cc: niv@us.ibm.com
Cc: peterz@infradead.org
Cc: rostedt@goodmis.org
Cc: Valdis.Kletnieks@vt.edu
Cc: dhowells@redhat.com
LKML-Reference: <1258094104417-git-send-email->
Signed-off-by: Ingo Molnar <mingo@elte.hu>

parent 0e0fc1c2

kernel/rcutree.c

+17 −1

Original line number	Diff line number	Diff line
		@@ -676,9 +676,25 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
		struct rcu_node *rnp = rcu_get_root(rsp);

		if (!cpu_needs_another_gp(rsp, rdp)) {
		if (rnp->completed == rsp->completed) {
		spin_unlock_irqrestore(&rnp->lock, flags);
		return;
		}
		spin_unlock(&rnp->lock); /* irqs remain disabled. */

		/*
		* Propagate new ->completed value to rcu_node structures
		* so that other CPUs don't have to wait until the start
		* of the next grace period to process their callbacks.
		*/
		rcu_for_each_node_breadth_first(rsp, rnp) {
		spin_lock(&rnp->lock); /* irqs already disabled. */
		rnp->completed = rsp->completed;
		spin_unlock(&rnp->lock); /* irqs remain disabled. */
		}
		local_irq_restore(flags);
		return;
		}

		/* Advance to a new grace period and initialize state. */
		rsp->gpnum++;

Admin message