rcu: Avoid misordering in __call_rcu_nocb_enqueue() (39953dfd) · Commits · 戴 / test

The NOCB leader wakeup ordering depends on the store to the header happening before the check for the leader already being awake. However, because atomic_long_add() does not return a value, it does not provide ordering guarantees, the incorrect comment in wake_nocb_leader() notwithstanding. This commit therefore adds a smp_mb__after_atomic() after the final atomic_long_add() to provide the needed ordering guarantee. Reported-by:

Amit Shah <amit.shah@redhat.com> Signed-off-by:

Paul E. McKenney <paulmck@linux.vnet.ibm.com> Tested-by:

Paul Gortmaker <paul.gortmaker@windriver.com>

kernel/rcu/tree_plugin.h

+2 −1

Original line number	Diff line number	Diff line
		@@ -2042,7 +2042,7 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool force)
		if (!ACCESS_ONCE(rdp_leader->nocb_kthread))
		return;
		if (ACCESS_ONCE(rdp_leader->nocb_leader_sleep) \|\| force) {
		/* Prior xchg orders against prior callback enqueue. */
		/* Prior smp_mb__after_atomic() orders against prior enqueue. */
		ACCESS_ONCE(rdp_leader->nocb_leader_sleep) = false;
		wake_up(&rdp_leader->nocb_wq);
		}
		@@ -2071,6 +2071,7 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,
		ACCESS_ONCE(*old_rhpp) = rhp;
		atomic_long_add(rhcount, &rdp->nocb_q_count);
		atomic_long_add(rhcount_lazy, &rdp->nocb_q_count_lazy);
		smp_mb__after_atomic(); /* Store old_rhpp before _wake test. /

		/* If we are not being polled and there is a kthread, awaken it ... */
		t = ACCESS_ONCE(rdp->nocb_kthread);

Admin message