rcu: Clear ->core_needs_qs at GP end or self-reported QS (b5ea0370) · Commits · 戴 / test

The rcu_data structure's ->core_needs_qs field does not necessarily get cleared in a timely fashion after the corresponding CPUs' quiescent state has been reported. From a functional viewpoint, no harm done, but this can result in excessive invocation of RCU core processing, as witnessed by the kernel test robot, which saw greatly increased softirq overhead. This commit therefore restores the rcu_report_qs_rdp() function's clearing of this field, but only when running on the corresponding CPU. Cases where some other CPU reports the quiescent state (for example, on behalf of an idle CPU) are handled by setting this field appropriately within the __note_gp_changes() function's end-of-grace-period checks. This handling is carried out regardless of whether the end of a grace period actually happened, thus handling the case where a CPU goes non-idle after a quiescent state is reported on its behalf, but before the grace period ends. This fix also avoids cross-CPU updates to ->core_needs_qs, While in the area, this commit changes the __note_gp_changes() need_gp variable's name to need_qs because it is a quiescent state that is needed from the CPU in question. Fixes: ("rcu: Confine ->core_needs_qs accesses to the corresponding CPU") Reported-by:

kernel test robot <rong.a.chen@intel.com> Signed-off-by:

Paul E. McKenney <paulmck@kernel.org>

kernel/rcu/tree.c

+9 −4

Original line number	Diff line number	Diff line
		@@ -1386,7 +1386,7 @@ static void __maybe_unused rcu_advance_cbs_nowake(struct rcu_node *rnp,
		static bool __note_gp_changes(struct rcu_node rnp, struct rcu_data rdp)
		{
		bool ret = false;
		bool need_gp;
		bool need_qs;
		const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
		rcu_segcblist_is_offloaded(&rdp->cblist);

		@@ -1400,10 +1400,13 @@ static bool __note_gp_changes(struct rcu_node rnp, struct rcu_data rdp)
		unlikely(READ_ONCE(rdp->gpwrap))) {
		if (!offloaded)
		ret = rcu_advance_cbs(rnp, rdp); /* Advance CBs. */
		rdp->core_needs_qs = false;
		trace_rcu_grace_period(rcu_state.name, rdp->gp_seq, TPS("cpuend"));
		} else {
		if (!offloaded)
		ret = rcu_accelerate_cbs(rnp, rdp); /* Recent CBs. */
		if (rdp->core_needs_qs)
		rdp->core_needs_qs = !!(rnp->qsmask & rdp->grpmask);
		}

		/* Now handle the beginnings of any new-to-this-CPU grace periods. */
		@@ -1415,9 +1418,9 @@ static bool __note_gp_changes(struct rcu_node rnp, struct rcu_data rdp)
		* go looking for one.
		*/
		trace_rcu_grace_period(rcu_state.name, rnp->gp_seq, TPS("cpustart"));
		need_gp = !!(rnp->qsmask & rdp->grpmask);
		rdp->cpu_no_qs.b.norm = need_gp;
		rdp->core_needs_qs = need_gp;
		need_qs = !!(rnp->qsmask & rdp->grpmask);
		rdp->cpu_no_qs.b.norm = need_qs;
		rdp->core_needs_qs = need_qs;
		zero_cpu_stall_ticks(rdp);
		}
		rdp->gp_seq = rnp->gp_seq; /* Remember new grace-period state. */
		@@ -1987,6 +1990,8 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
		return;
		}
		mask = rdp->grpmask;
		if (rdp->cpu == smp_processor_id())
		rdp->core_needs_qs = false;
		if ((rnp->qsmask & mask) == 0) {
		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
		} else {

Admin message