Revert "Revert "mm, thp: restore node-local hugepage allocations"" (ac79f78d) · Commits · 戴 / test

This reverts commit . The commit references the original intended semantic for MADV_HUGEPAGE which has subsequently taken on three unique purposes: - enables or disables thp for a range of memory depending on the system's config (is thp "enabled" set to "always" or "madvise"), - determines the synchronous compaction behavior for thp allocations at fault (is thp "defrag" set to "always", "defer+madvise", or "madvise"), and - reverts a previous MADV_NOHUGEPAGE (there is no madvise mode to only clear previous hugepage advice). These are the three purposes that currently exist in 5.2 and over the past several years that userspace has been written around. Adding a NUMA locality preference adds a fourth dimension to an already conflated advice mode. Based on the semantic that MADV_HUGEPAGE has provided over the past several years, there exist workloads that use the tunable based on these principles: specifically that the allocation should attempt to defragment a local node before falling back. It is agreed that remote hugepages typically (but not always) have a better access latency than remote native pages, although on Naples this is at parity for intersocket. The revert commit that this patch reverts allows hugepage allocation to immediately allocate remotely when local memory is fragmented. This is contrary to the semantic of MADV_HUGEPAGE over the past several years: that is, memory compaction should be attempted locally before falling back. The performance degradation of remote hugepages over local hugepages on Rome, for example, is 53.5% increased access latency. For this reason, the goal is to revert back to the 5.2 and previous behavior that would attempt local defragmentation before falling back. With the patch that is reverted by this patch, we see performance degradations at the tail because the allocator happily allocates the remote hugepage rather than even attempting to make a local hugepage available. zone_reclaim_mode is not a solution to this problem since it does not only impact hugepage allocations but rather changes the memory allocation strategy for *all* page allocations. Signed-off-by:

David Rientjes <rientjes@google.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Stefan Priebe - Profihost AG <s.priebe@profihost.ag> Cc: "Kirill A. Shutemov" <kirill@shutemov.name> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by:

Linus Torvalds <torvalds@linux-foundation.org>

include/linux/mempolicy.h

+0 −2

Original line number	Diff line number	Diff line
		@@ -139,8 +139,6 @@ struct mempolicy mpol_shared_policy_lookup(struct shared_policy sp,
		struct mempolicy get_task_policy(struct task_struct p);
		struct mempolicy __get_vma_policy(struct vm_area_struct vma,
		unsigned long addr);
		struct mempolicy get_vma_policy(struct vm_area_struct vma,
		unsigned long addr);
		bool vma_policy_mof(struct vm_area_struct *vma);

		extern void numa_default_policy(void);

mm/huge_memory.c

+16 −26

Original line number	Diff line number	Diff line
		@@ -648,37 +648,27 @@ release:
		static inline gfp_t alloc_hugepage_direct_gfpmask(struct vm_area_struct *vma, unsigned long addr)
		{
		const bool vma_madvised = !!(vma->vm_flags & VM_HUGEPAGE);
		gfp_t this_node = 0;

		#ifdef CONFIG_NUMA
		struct mempolicy *pol;
		/*
		* __GFP_THISNODE is used only when __GFP_DIRECT_RECLAIM is not
		* specified, to express a general desire to stay on the current
		* node for optimistic allocation attempts. If the defrag mode
		* and/or madvise hint requires the direct reclaim then we prefer
		* to fallback to other node rather than node reclaim because that
		* can lead to excessive reclaim even though there is free memory
		* on other nodes. We expect that NUMA preferences are specified
		* by memory policies.
		*/
		pol = get_vma_policy(vma, addr);
		if (pol->mode != MPOL_BIND)
		this_node = __GFP_THISNODE;
		mpol_cond_put(pol);
		#endif
		const gfp_t gfp_mask = GFP_TRANSHUGE_LIGHT \| __GFP_THISNODE;

		/* Always do synchronous compaction */
		if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags))
		return GFP_TRANSHUGE \| (vma_madvised ? 0 : __GFP_NORETRY);
		return GFP_TRANSHUGE \| __GFP_THISNODE \|
		(vma_madvised ? 0 : __GFP_NORETRY);

		/* Kick kcompactd and fail quickly */
		if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags))
		return GFP_TRANSHUGE_LIGHT \| __GFP_KSWAPD_RECLAIM \| this_node;
		return gfp_mask \| __GFP_KSWAPD_RECLAIM;

		/* Synchronous compaction if madvised, otherwise kick kcompactd */
		if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags))
		return GFP_TRANSHUGE_LIGHT \| (vma_madvised ? __GFP_DIRECT_RECLAIM :
		__GFP_KSWAPD_RECLAIM \| this_node);
		return gfp_mask \| (vma_madvised ? __GFP_DIRECT_RECLAIM :
		__GFP_KSWAPD_RECLAIM);

		/* Only do synchronous compaction if madvised */
		if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags))
		return GFP_TRANSHUGE_LIGHT \| (vma_madvised ? __GFP_DIRECT_RECLAIM :
		this_node);
		return GFP_TRANSHUGE_LIGHT \| this_node;
		return gfp_mask \| (vma_madvised ? __GFP_DIRECT_RECLAIM : 0);

		return gfp_mask;
		}

		/* Caller must hold page table lock. */

mm/mempolicy.c

+1 −1

Original line number	Diff line number	Diff line
		@@ -1734,7 +1734,7 @@ struct mempolicy __get_vma_policy(struct vm_area_struct vma,
		* freeing by another task. It is the caller's responsibility to free the
		* extra reference for shared policies.
		*/
		struct mempolicy get_vma_policy(struct vm_area_struct vma,
		static struct mempolicy get_vma_policy(struct vm_area_struct vma,
		unsigned long addr)
		{
		struct mempolicy *pol = __get_vma_policy(vma, addr);

Admin message