Commit 1a2f5d57 authored Aug 06, 2020 by Paul E. McKenney

rcu: Attempt QS when CPU discovers GP for strict GPs

A given CPU normally notes a new grace period during one RCU_SOFTIRQ,
but avoids reporting the corresponding quiescent state until some later
RCU_SOFTIRQ. This leisurly approach improves efficiency by increasing
the number of update requests served by each grace period, but is not
what is needed for kernels built with CONFIG_RCU_STRICT_GRACE_PERIOD=y.

This commit therefore adds a new rcu_strict_gp_check_qs() function
which, in CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, simply enters and
immediately exist an RCU read-side critical section. If the CPU is
in a quiescent state, the rcu_read_unlock() will attempt to report an
immediate quiescent state. This rcu_strict_gp_check_qs() function is
invoked from note_gp_changes(), so that a CPU just noticing a new grace
period might immediately report a quiescent state for that grace period.

Reported-by Jann Horn <jannh@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

parent 44bad5b3

kernel/rcu/tree.c

+14 −0

Original line number	Diff line number	Diff line
		@@ -1574,6 +1574,19 @@ static void __maybe_unused rcu_advance_cbs_nowake(struct rcu_node *rnp,
		raw_spin_unlock_rcu_node(rnp);
		}

		/*
		* In CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, attempt to generate a
		* quiescent state. This is intended to be invoked when the CPU notices
		* a new grace period.
		*/
		static void rcu_strict_gp_check_qs(void)
		{
		if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD)) {
		rcu_read_lock();
		rcu_read_unlock();
		}
		}

		/*
		* Update CPU-local rcu_data state to record the beginnings and ends of
		* grace periods. The caller must hold the ->lock of the leaf rcu_node
		@@ -1644,6 +1657,7 @@ static void note_gp_changes(struct rcu_data *rdp)
		}
		needwake = __note_gp_changes(rnp, rdp);
		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
		rcu_strict_gp_check_qs();
		if (needwake)
		rcu_gp_kthread_wake();
		}

Admin message