rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu (f69b17d7) · Commits · 戴 / test

cpu hotplug may happen asynchronously, some rcu callbacks are maybe still on dead cpu, rcu_barrier() also needs to wait for these rcu callbacks to complete, so we must ensure callbacks in dead cpu are migrated to online cpu. Paul E. McKenney's review: Good stuff, Lai!!! Simpler than any of the approaches that I was considering, and, better yet, independent of the underlying RCU implementation!!! I was initially worried that wake_up() might wake only one of two possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code, but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue. I was also worried about the fact that different RCU implementations have different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but this is OK as well because we just get an extra (harmless) callback in the case that they map together (for example, Classic RCU has call_rcu_sched() mapping to call_rcu()). Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock, and any stray callbacks that arrive (for example, from irq handlers running on the dying CPU) either are ahead of the CPU_DYING callbacks on the one hand (and thus accounted for), or happened after the rcu_barrier() started on the other (and thus don't need to be accounted for). Signed-off-by:

Lai Jiangshan <laijs@cn.fujitsu.com> Reviewed-by:

Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> LKML-Reference: <49C36476.1010400@cn.fujitsu.com> Signed-off-by:

Ingo Molnar <mingo@elte.hu>

kernel/rcupdate.c

+44 −0

Original line number	Diff line number	Diff line
		@@ -122,6 +122,8 @@ static void rcu_barrier_func(void *type)
		}
		}

		static inline void wait_migrated_callbacks(void);

		/*
		* Orchestrate the specified type of RCU barrier, waiting for all
		* RCU callbacks of the specified type to complete.
		@@ -147,6 +149,7 @@ static void _rcu_barrier(enum rcu_barrier type)
		complete(&rcu_barrier_completion);
		wait_for_completion(&rcu_barrier_completion);
		mutex_unlock(&rcu_barrier_mutex);
		wait_migrated_callbacks();
		}

		/**
		@@ -176,9 +179,50 @@ void rcu_barrier_sched(void)
		}
		EXPORT_SYMBOL_GPL(rcu_barrier_sched);

		static atomic_t rcu_migrate_type_count = ATOMIC_INIT(0);
		static struct rcu_head rcu_migrate_head[3];
		static DECLARE_WAIT_QUEUE_HEAD(rcu_migrate_wq);

		static void rcu_migrate_callback(struct rcu_head *notused)
		{
		if (atomic_dec_and_test(&rcu_migrate_type_count))
		wake_up(&rcu_migrate_wq);
		}

		static inline void wait_migrated_callbacks(void)
		{
		wait_event(rcu_migrate_wq, !atomic_read(&rcu_migrate_type_count));
		}

		static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self,
		unsigned long action, void *hcpu)
		{
		if (action == CPU_DYING) {
		/*
		* preempt_disable() in on_each_cpu() prevents stop_machine(),
		* so when "on_each_cpu(rcu_barrier_func, (void *)type, 1);"
		* returns, all online cpus have queued rcu_barrier_func(),
		* and the dead cpu(if it exist) queues rcu_migrate_callback()s.
		*
		* These callbacks ensure _rcu_barrier() waits for all
		* RCU callbacks of the specified type to complete.
		*/
		atomic_set(&rcu_migrate_type_count, 3);
		call_rcu_bh(rcu_migrate_head, rcu_migrate_callback);
		call_rcu_sched(rcu_migrate_head + 1, rcu_migrate_callback);
		call_rcu(rcu_migrate_head + 2, rcu_migrate_callback);
		} else if (action == CPU_POST_DEAD) {
		/* rcu_migrate_head is protected by cpu_add_remove_lock */
		wait_migrated_callbacks();
		}

		return NOTIFY_OK;
		}

		void __init rcu_init(void)
		{
		__rcu_init();
		hotcpu_notifier(rcu_barrier_cpu_hotplug, 0);
		}

		void rcu_scheduler_starting(void)

Admin message