sched: Cache task_struct::flags in sched_submit_work() (c1cecf88) · Commits · 戴 / test

sched_submit_work() is considered to be a hot path. The preempt_disable() instruction is a compiler barrier and forces the compiler to load task_struct::flags for the second comparison. By using a local variable, the compiler can load the value once and keep it in a register for the second comparison. Verified on x86-64 with gcc-10. Signed-off-by:

Sebastian Andrzej Siewior <bigeasy@linutronix.de> Signed-off-by:

Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20200819200025.lqvmyefqnbok5i4f@linutronix.de

kernel/sched/core.c

+5 −2

Original line number	Diff line number	Diff line
		@@ -4551,9 +4551,12 @@ void __noreturn do_task_dead(void)

		static inline void sched_submit_work(struct task_struct *tsk)
		{
		unsigned int task_flags;

		if (!tsk->state)
		return;

		task_flags = tsk->flags;
		/*
		* If a worker went to sleep, notify and ask workqueue whether
		* it wants to wake up a task to maintain concurrency.
		@@ -4562,9 +4565,9 @@ static inline void sched_submit_work(struct task_struct *tsk)
		* in the possible wakeup of a kworker and because wq_worker_sleeping()
		* requires it.
		*/
		if (tsk->flags & (PF_WQ_WORKER \| PF_IO_WORKER)) {
		if (task_flags & (PF_WQ_WORKER \| PF_IO_WORKER)) {
		preempt_disable();
		if (tsk->flags & PF_WQ_WORKER)
		if (task_flags & PF_WQ_WORKER)
		wq_worker_sleeping(tsk);
		else
		io_wq_worker_sleeping(tsk);

Admin message