Commit 5462bc3a authored by Rik van Riel's avatar Rik van Riel Committed by Peter Zijlstra
Browse files

x86/mm/tlb: Always use lazy TLB mode 



On most workloads, the number of context switches far exceeds the
number of TLB flushes sent. Optimizing the context switches, by always
using lazy TLB mode, speeds up those workloads.

This patch results in about a 1% reduction in CPU use on a two socket
Broadwell system running a memcache like workload.

Cc: npiggin@gmail.com
Cc: efault@gmx.de
Cc: will.deacon@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: kernel-team@fb.com
Cc: hpa@zytor.com
Cc: luto@kernel.org
Tested-by: default avatarSong Liu <songliubraving@fb.com>
Signed-off-by: default avatarRik van Riel <riel@surriel.com>
(cherry picked from commit 95b0e635)
Acked-by: default avatarDave Hansen <dave.hansen@intel.com>
Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20180716190337.26133-7-riel@surriel.com
parent a31acd3e

arch/x86/include/asm/tlbflush.h

+0 −16
Original line number Diff line number Diff line
@@ -148,22 +148,6 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) 
#define __flush_tlb_one_user(addr) __native_flush_tlb_one_user(addr)

#endif



static inline bool tlb_defer_switch_to_init_mm(void)

{

	/*

	 * If we have PCID, then switching to init_mm is reasonably

	 * fast.  If we don't have PCID, then switching to init_mm is

	 * quite slow, so we try to defer it in the hopes that we can

	 * avoid it entirely.  The latter approach runs the risk of

	 * receiving otherwise unnecessary IPIs.

	 *

	 * This choice is just a heuristic.  The tlb code can handle this

	 * function returning true or false regardless of whether we have

	 * PCID.

	 */

	return !static_cpu_has(X86_FEATURE_PCID);

}



struct tlb_context {

	u64 ctx_id;

	u64 tlb_gen;

arch/x86/mm/tlb.c

+1 −14
Original line number Diff line number Diff line
@@ -368,20 +368,7 @@ void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) 
	if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm)

		return;



	if (tlb_defer_switch_to_init_mm()) {

		/*

		 * There's a significant optimization that may be possible

		 * here.  We have accurate enough TLB flush tracking that we

		 * don't need to maintain coherence of TLB per se when we're

		 * lazy.  We do, however, need to maintain coherence of

		 * paging-structure caches.  We could, in principle, leave our

		 * old mm loaded and only switch to init_mm when

		 * tlb_remove_page() happens.

		 */

	this_cpu_write(cpu_tlbstate.is_lazy, true);

	} else {

		switch_mm(NULL, &init_mm, NULL);

	}

}



/*

CRA Git | Maintained and supported by SUSTech CRA and CCSE