Commit 282722b0 authored by Vlastimil Babka's avatar Vlastimil Babka Committed by Linus Torvalds
Browse files

mm, compaction: restrict async compaction to pageblocks of same migratetype 

The migrate scanner in async compaction is currently limited to
MIGRATE_MOVABLE pageblocks.  This is a heuristic intended to reduce
latency, based on the assumption that non-MOVABLE pageblocks are
unlikely to contain movable pages.

However, with the exception of THP's, most high-order allocations are
not movable.  Should the async compaction succeed, this increases the
chance that the non-MOVABLE allocations will fallback to a MOVABLE
pageblock, making the long-term fragmentation worse.

This patch attempts to help the situation by changing async direct
compaction so that the migrate scanner only scans the pageblocks of the
requested migratetype.  If it's a non-MOVABLE type and there are such
pageblocks that do contain movable pages, chances are that the
allocation can succeed within one of such pageblocks, removing the need
for a fallback.  If that fails, the subsequent sync attempt will ignore
this restriction.

In testing based on 4.9 kernel with stress-highalloc from mmtests
configured for order-4 GFP_KERNEL allocations, this patch has reduced
the number of unmovable allocations falling back to movable pageblocks
by 30%.  The number of movable allocations falling back is reduced by
12%.

Link: http://lkml.kernel.org/r/20170307131545.28577-8-vbabka@suse.cz


Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent d39773a0

mm/compaction.c

+9 −2
Original line number Diff line number Diff line
@@ -986,10 +986,17 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn, 
static bool suitable_migration_source(struct compact_control *cc,

							struct page *page)

{

	if (cc->mode != MIGRATE_ASYNC)

	int block_mt;



	if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction)

		return true;



	return is_migrate_movable(get_pageblock_migratetype(page));

	block_mt = get_pageblock_migratetype(page);



	if (cc->migratetype == MIGRATE_MOVABLE)

		return is_migrate_movable(block_mt);

	else

		return block_mt == cc->migratetype;

}



/* Returns true if the page is within a block suitable for migration to */

mm/page_alloc.c

+13 −7
Original line number Diff line number Diff line
@@ -3665,6 +3665,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, 
						struct alloc_context *ac)

{

	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;

	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;

	struct page *page = NULL;

	unsigned int alloc_flags;

	unsigned long did_some_progress;
@@ -3732,12 +3733,17 @@ retry_cpuset: 


	/*

	 * For costly allocations, try direct compaction first, as it's likely

	 * that we have enough base pages and don't need to reclaim. Don't try

	 * that for allocations that are allowed to ignore watermarks, as the

	 * ALLOC_NO_WATERMARKS attempt didn't yet happen.

	 */

	if (can_direct_reclaim && order > PAGE_ALLOC_COSTLY_ORDER &&

		!gfp_pfmemalloc_allowed(gfp_mask)) {

	 * that we have enough base pages and don't need to reclaim. For non-

	 * movable high-order allocations, do that as well, as compaction will

	 * try prevent permanent fragmentation by migrating from blocks of the

	 * same migratetype.

	 * Don't try this for allocations that are allowed to ignore

	 * watermarks, as the ALLOC_NO_WATERMARKS attempt didn't yet happen.

	 */

	if (can_direct_reclaim &&

			(costly_order ||

			   (order > 0 && ac->migratetype != MIGRATE_MOVABLE))

			&& !gfp_pfmemalloc_allowed(gfp_mask)) {

		page = __alloc_pages_direct_compact(gfp_mask, order,

						alloc_flags, ac,

						INIT_COMPACT_PRIORITY,
@@ -3749,7 +3755,7 @@ retry_cpuset: 
		 * Checks for costly allocations with __GFP_NORETRY, which

		 * includes THP page fault allocations

		 */

		if (gfp_mask & __GFP_NORETRY) {

		if (costly_order && (gfp_mask & __GFP_NORETRY)) {

			/*

			 * If compaction is deferred for high-order allocations,

			 * it is because sync compaction recently failed. If
@@ -3830,7 +3836,7 @@ retry: 
	 * Do not retry costly high order allocations unless they are

	 * __GFP_REPEAT

	 */

	if (order > PAGE_ALLOC_COSTLY_ORDER && !(gfp_mask & __GFP_REPEAT))

	if (costly_order && !(gfp_mask & __GFP_REPEAT))

		goto nopage;



	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,

CRA Git | Maintained and supported by SUSTech CRA and CCSE