rcu: Make expedited grace period use direct call on last leaf (5257514d) · Commits · 戴 / test

During expedited grace-period initialization, a work item is scheduled for each leaf rcu_node structure. However, that initialization code is itself (normally) executing from a workqueue, so one of the leaf rcu_node structures could just as well be handled by that pre-existing workqueue, and with less overhead. This commit therefore uses a shiny new rcu_is_leaf_node() macro to execute the last leaf rcu_node structure's initialization directly from the pre-existing workqueue. Signed-off-by:

Paul E. McKenney <paulmck@linux.vnet.ibm.com>

kernel/rcu/rcu.h

+3 −0

Original line number	Diff line number	Diff line
		@@ -276,6 +276,9 @@ static inline void rcu_init_levelspread(int levelspread, const int levelcnt)
		/* Is this rcu_node a leaf? */
		#define rcu_is_leaf_node(rnp) ((rnp)->level == rcu_num_lvls - 1)

		/* Is this rcu_node the last leaf? */
		#define rcu_is_last_leaf_node(rsp, rnp) ((rnp) == &(rsp)->node[rcu_num_nodes - 1])

		/*
		* Do a full breadth-first scan of the rcu_node structures for the
		* specified rcu_state structure.

kernel/rcu/tree_exp.h

+3 −2

Original line number	Diff line number	Diff line
		@@ -486,8 +486,9 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,
		rnp->rew.rew_func = func;
		rnp->rew.rew_rsp = rsp;
		if (!READ_ONCE(rcu_par_gp_wq) \|\|
		rcu_scheduler_active != RCU_SCHEDULER_RUNNING) {
		/* No workqueues yet. */
		rcu_scheduler_active != RCU_SCHEDULER_RUNNING \|\|
		rcu_is_last_leaf_node(rsp, rnp)) {
		/* No workqueues yet or last leaf, do direct call. */
		sync_rcu_exp_select_node_cpus(&rnp->rew.rew_work);
		continue;
		}

Admin message