Commit 70b50f94 authored by Andrea Arcangeli's avatar Andrea Arcangeli Committed by Linus Torvalds
Browse files

mm: thp: tail page refcounting fix 



Michel while working on the working set estimation code, noticed that
calling get_page_unless_zero() on a random pfn_to_page(random_pfn)
wasn't safe, if the pfn ended up being a tail page of a transparent
hugepage under splitting by __split_huge_page_refcount().

He then found the problem could also theoretically materialize with
page_cache_get_speculative() during the speculative radix tree lookups
that uses get_page_unless_zero() in SMP if the radix tree page is freed
and reallocated and get_user_pages is called on it before
page_cache_get_speculative has a chance to call get_page_unless_zero().

So the best way to fix the problem is to keep page_tail->_count zero at
all times.  This will guarantee that get_page_unless_zero() can never
succeed on any tail page.  page_tail->_mapcount is guaranteed zero and
is unused for all tail pages of a compound page, so we can simply
account the tail page references there and transfer them to
tail_page->_count in __split_huge_page_refcount() (in addition to the
head_page->_mapcount).

While debugging this s/_count/_mapcount/ change I also noticed get_page is
called by direct-io.c on pages returned by get_user_pages.  That wasn't
entirely safe because the two atomic_inc in get_page weren't atomic.  As
opposed to other get_user_page users like secondary-MMU page fault to
establish the shadow pagetables would never call any superflous get_page
after get_user_page returns.  It's safer to make get_page universally safe
for tail pages and to use get_page_foll() within follow_page (inside
get_user_pages()).  get_page_foll() is safe to do the refcounting for tail
pages without taking any locks because it is run within PT lock protected
critical sections (PT lock for pte and page_table_lock for
pmd_trans_huge).

The standard get_page() as invoked by direct-io instead will now take
the compound_lock but still only for tail pages.  The direct-io paths
are usually I/O bound and the compound_lock is per THP so very
finegrined, so there's no risk of scalability issues with it.  A simple
direct-io benchmarks with all lockdep prove locking and spinlock
debugging infrastructure enabled shows identical performance and no
overhead.  So it's worth it.  Ideally direct-io should stop calling
get_page() on pages returned by get_user_pages().  The spinlock in
get_page() is already optimized away for no-THP builds but doing
get_page() on tail pages returned by GUP is generally a rare operation
and usually only run in I/O paths.

This new refcounting on page_tail->_mapcount in addition to avoiding new
RCU critical sections will also allow the working set estimation code to
work without any further complexity associated to the tail page
refcounting with THP.

Signed-off-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
Reported-by: default avatarMichel Lespinasse <walken@google.com>
Reviewed-by: default avatarMichel Lespinasse <walken@google.com>
Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hugh Dickins <hughd@google.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: <stable@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 994c0e99

arch/powerpc/mm/gup.c

+3 −2
Original line number Diff line number Diff line
@@ -22,8 +22,9 @@ static inline void get_huge_page_tail(struct page *page) 
	 * __split_huge_page_refcount() cannot run

	 * from under us.

	 */

	VM_BUG_ON(atomic_read(&page->_count) < 0);

	atomic_inc(&page->_count);

	VM_BUG_ON(page_mapcount(page) < 0);

	VM_BUG_ON(atomic_read(&page->_count) != 0);

	atomic_inc(&page->_mapcount);

}



/*

arch/x86/mm/gup.c

+3 −2
Original line number Diff line number Diff line
@@ -114,8 +114,9 @@ static inline void get_huge_page_tail(struct page *page) 
	 * __split_huge_page_refcount() cannot run

	 * from under us.

	 */

	VM_BUG_ON(atomic_read(&page->_count) < 0);

	atomic_inc(&page->_count);

	VM_BUG_ON(page_mapcount(page) < 0);

	VM_BUG_ON(atomic_read(&page->_count) != 0);

	atomic_inc(&page->_mapcount);

}



static noinline int gup_huge_pmd(pmd_t pmd, unsigned long addr,

include/linux/mm.h

+22 −34
Original line number Diff line number Diff line
@@ -356,36 +356,39 @@ static inline struct page *compound_head(struct page *page) 
	return page;

}



/*

 * The atomic page->_mapcount, starts from -1: so that transitions

 * both from it and to it can be tracked, using atomic_inc_and_test

 * and atomic_add_negative(-1).

 */

static inline void reset_page_mapcount(struct page *page)

{

	atomic_set(&(page)->_mapcount, -1);

}



static inline int page_mapcount(struct page *page)

{

	return atomic_read(&(page)->_mapcount) + 1;

}



static inline int page_count(struct page *page)

{

	return atomic_read(&compound_head(page)->_count);

}



extern bool __get_page_tail(struct page *page);



static inline void get_page(struct page *page)

{

	if (unlikely(PageTail(page)))

		if (likely(__get_page_tail(page)))

			return;

	/*

	 * Getting a normal page or the head of a compound page

	 * requires to already have an elevated page->_count. Only if

	 * we're getting a tail page, the elevated page->_count is

	 * required only in the head page, so for tail pages the

	 * bugcheck only verifies that the page->_count isn't

	 * negative.

	 * requires to already have an elevated page->_count.

	 */

	VM_BUG_ON(atomic_read(&page->_count) < !PageTail(page));

	VM_BUG_ON(atomic_read(&page->_count) <= 0);

	atomic_inc(&page->_count);

	/*

	 * Getting a tail page will elevate both the head and tail

	 * page->_count(s).

	 */

	if (unlikely(PageTail(page))) {

		/*

		 * This is safe only because

		 * __split_huge_page_refcount can't run under

		 * get_page().

		 */

		VM_BUG_ON(atomic_read(&page->first_page->_count) <= 0);

		atomic_inc(&page->first_page->_count);

	}

}



static inline struct page *virt_to_head_page(const void *x)
@@ -803,21 +806,6 @@ static inline pgoff_t page_index(struct page *page) 
	return page->index;

}



/*

 * The atomic page->_mapcount, like _count, starts from -1:

 * so that transitions both from it and to it can be tracked,

 * using atomic_inc_and_test and atomic_add_negative(-1).

 */

static inline void reset_page_mapcount(struct page *page)

{

	atomic_set(&(page)->_mapcount, -1);

}



static inline int page_mapcount(struct page *page)

{

	return atomic_read(&(page)->_mapcount) + 1;

}



/*

 * Return true if this page is mapped into pagetables.

 */

include/linux/mm_types.h

+17 −4
Original line number Diff line number Diff line
@@ -62,10 +62,23 @@ struct page { 
			struct {



				union {

					atomic_t _mapcount;	/* Count of ptes mapped in mms,

							 * to show when page is mapped

							 * & limit reverse map searches.

					/*

					 * Count of ptes mapped in

					 * mms, to show when page is

					 * mapped & limit reverse map

					 * searches.

					 *

					 * Used also for tail pages

					 * refcounting instead of

					 * _count. Tail pages cannot

					 * be mapped and keeping the

					 * tail page _count zero at

					 * all times guarantees

					 * get_page_unless_zero() will

					 * never succeed on tail

					 * pages.

					 */

					atomic_t _mapcount;



					struct {

						unsigned inuse:16;

mm/huge_memory.c

+26 −11
Original line number Diff line number Diff line
@@ -990,7 +990,7 @@ struct page *follow_trans_huge_pmd(struct mm_struct *mm, 
	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;

	VM_BUG_ON(!PageCompound(page));

	if (flags & FOLL_GET)

		get_page(page);

		get_page_foll(page);



out:

	return page;
@@ -1202,6 +1202,7 @@ static void __split_huge_page_refcount(struct page *page) 
	unsigned long head_index = page->index;

	struct zone *zone = page_zone(page);

	int zonestat;

	int tail_count = 0;



	/* prevent PageLRU to go away from under us, and freeze lru stats */

	spin_lock_irq(&zone->lru_lock);
@@ -1210,11 +1211,27 @@ static void __split_huge_page_refcount(struct page *page) 
	for (i = 1; i < HPAGE_PMD_NR; i++) {

		struct page *page_tail = page + i;



		/* tail_page->_count cannot change */

		atomic_sub(atomic_read(&page_tail->_count), &page->_count);

		BUG_ON(page_count(page) <= 0);

		atomic_add(page_mapcount(page) + 1, &page_tail->_count);

		BUG_ON(atomic_read(&page_tail->_count) <= 0);

		/* tail_page->_mapcount cannot change */

		BUG_ON(page_mapcount(page_tail) < 0);

		tail_count += page_mapcount(page_tail);

		/* check for overflow */

		BUG_ON(tail_count < 0);

		BUG_ON(atomic_read(&page_tail->_count) != 0);

		/*

		 * tail_page->_count is zero and not changing from

		 * under us. But get_page_unless_zero() may be running

		 * from under us on the tail_page. If we used

		 * atomic_set() below instead of atomic_add(), we

		 * would then run atomic_set() concurrently with

		 * get_page_unless_zero(), and atomic_set() is

		 * implemented in C not using locked ops. spin_unlock

		 * on x86 sometime uses locked ops because of PPro

		 * errata 66, 92, so unless somebody can guarantee

		 * atomic_set() here would be safe on all archs (and

		 * not only on x86), it's safer to use atomic_add().

		 */

		atomic_add(page_mapcount(page) + page_mapcount(page_tail) + 1,

			   &page_tail->_count);



		/* after clearing PageTail the gup refcount can be released */

		smp_mb();
@@ -1232,10 +1249,7 @@ static void __split_huge_page_refcount(struct page *page) 
				      (1L << PG_uptodate)));

		page_tail->flags |= (1L << PG_dirty);



		/*

		 * 1) clear PageTail before overwriting first_page

		 * 2) clear PageTail before clearing PageHead for VM_BUG_ON

		 */

		/* clear PageTail before overwriting first_page */

		smp_wmb();



		/*
@@ -1252,7 +1266,6 @@ static void __split_huge_page_refcount(struct page *page) 
		 * status is achieved setting a reserved bit in the

		 * pmd, not by clearing the present bit.

		*/

		BUG_ON(page_mapcount(page_tail));

		page_tail->_mapcount = page->_mapcount;



		BUG_ON(page_tail->mapping);
@@ -1269,6 +1282,8 @@ static void __split_huge_page_refcount(struct page *page) 


		lru_add_page_tail(zone, page, page_tail);

	}

	atomic_sub(tail_count, &page->_count);

	BUG_ON(atomic_read(&page->_count) <= 0);



	__dec_zone_page_state(page, NR_ANON_TRANSPARENT_HUGEPAGES);

	__mod_zone_page_state(zone, NR_ANON_PAGES, HPAGE_PMD_NR);

CRA Git | Maintained and supported by SUSTech CRA and CCSE