Commit a2165e41 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Don't funnel-lock above leaf node if GP in progress 



The old grace-period start code would acquire only the leaf's rcu_node
structure's ->lock if that structure believed that a grace period was
in progress.  The new code advances to the leaf's parent in this case,
needlessly acquiring then leaf's parent's ->lock.  This commit therefore
checks the grace-period state after marking the leaf with the need for
the specified grace period, and if the leaf believes that a grace period
is in progress, takes an early exit.

Reported-by: default avatarJoel Fernandes <joel@joelfernandes.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
[ paulmck: Add "Startedleaf" tracing as suggested by Joel Fernandes. ]
parent e1333462

include/trace/events/rcu.h

+2 −2
Original line number Diff line number Diff line
@@ -91,8 +91,8 @@ TRACE_EVENT(rcu_grace_period, 
 *

 * "Startleaf": Request a grace period based on leaf-node data.

 * "Prestarted": Someone beat us to the request

 * "Startedleaf": Leaf-node start proved sufficient.

 * "Startedleafroot": Leaf-node start proved sufficient after checking root.

 * "Startedleaf": Leaf node marked for future GP.

 * "Startedleafroot": All nodes from leaf to root marked for future GP.

 * "Startedroot": Requested a nocb grace period based on root-node data.

 * "NoGPkthread": The RCU grace-period kthread has not yet started.

 * "StartWait": Start waiting for the requested grace period.

kernel/rcu/tree.c

+9 −0
Original line number Diff line number Diff line
@@ -1590,6 +1590,15 @@ static bool rcu_start_this_gp(struct rcu_node *rnp, struct rcu_data *rdp, 
			goto unlock_out;

		}

		rnp_root->gp_seq_needed = c;

		if (rcu_seq_state(rcu_seq_current(&rnp->gp_seq))) {

			/*

			 * We just marked the leaf, and a grace period

			 * is in progress, which means that rcu_gp_cleanup()

			 * will see the marking.  Bail to reduce contention.

			 */

			trace_rcu_this_gp(rnp, rdp, c, TPS("Startedleaf"));

			goto unlock_out;

		}

		if (rnp_root != rnp && rnp_root->parent != NULL)

			raw_spin_unlock_rcu_node(rnp_root);

		if (!rnp_root->parent)

CRA Git | Maintained and supported by SUSTech CRA and CCSE