mm/memory.c: fix a huge pud insertion race during faulting (625110b5) · Commits · 戴 / test

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer. Race: Thread 1: Thread 2: Comment create_huge_pud() Fallback - not taken. create_huge_pud() Taken. pmd_alloc() Returns an invalid pointer. This will result in user-visible huge page data corruption. Note that this was caught during a code audit rather than a real experienced problem. It looks to me like the only implementation that currently creates huge pud pagetable entries is dev_dax_huge_fault() which doesn't appear to care much about private (COW) mappings or write-tracking which is, I believe, a prerequisite for create_huge_pud() falling back on thread 1, but not in thread 2. Link: http://lkml.kernel.org/r/20191115115808.21181-2-thomas_os@shipmail.org Fixes: ("mm, x86: add support for PUD-sized transparent hugepages") Signed-off-by:

Thomas Hellstrom <thellstrom@vmware.com> Acked-by:

Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Matthew Wilcox <willy@infradead.org> Cc: <stable@vger.kernel.org> Signed-off-by:

Andrew Morton <akpm@linux-foundation.org> Signed-off-by:

Linus Torvalds <torvalds@linux-foundation.org>

include/asm-generic/pgtable.h

+25 −0

Original line number	Diff line number	Diff line
		@@ -938,6 +938,31 @@ static inline int pud_trans_huge(pud_t pud)
		}
		#endif

		/* See pmd_none_or_trans_huge_or_clear_bad for discussion. */
		static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud)
		{
		pud_t pudval = READ_ONCE(*pud);

		if (pud_none(pudval) \|\| pud_trans_huge(pudval) \|\| pud_devmap(pudval))
		return 1;
		if (unlikely(pud_bad(pudval))) {
		pud_clear_bad(pud);
		return 1;
		}
		return 0;
		}

		/* See pmd_trans_unstable for discussion. */
		static inline int pud_trans_unstable(pud_t *pud)
		{
		#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \
		defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)
		return pud_none_or_trans_huge_or_dev_or_clear_bad(pud);
		#else
		return 0;
		#endif
		}

		#ifndef pmd_read_atomic
		static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
		{

mm/memory.c

+6 −0

Original line number	Diff line number	Diff line
		@@ -4010,6 +4010,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
		vmf.pud = pud_alloc(mm, p4d, address);
		if (!vmf.pud)
		return VM_FAULT_OOM;
		retry_pud:
		if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) {
		ret = create_huge_pud(&vmf);
		if (!(ret & VM_FAULT_FALLBACK))
		@@ -4036,6 +4037,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
		vmf.pmd = pmd_alloc(mm, vmf.pud, address);
		if (!vmf.pmd)
		return VM_FAULT_OOM;

		/* Huge pud page fault raced with pmd_alloc? */
		if (pud_trans_unstable(vmf.pud))
		goto retry_pud;

		if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) {
		ret = create_huge_pmd(&vmf);
		if (!(ret & VM_FAULT_FALLBACK))

Admin message