Commit 41858966 authored by Mel Gorman's avatar Mel Gorman Committed by Linus Torvalds
Browse files

page allocator: use allocation flags as an index to the zone watermark 



ALLOC_WMARK_MIN, ALLOC_WMARK_LOW and ALLOC_WMARK_HIGH determin whether
pages_min, pages_low or pages_high is used as the zone watermark when
allocating the pages.  Two branches in the allocator hotpath determine
which watermark to use.

This patch uses the flags as an array index into a watermark array that is
indexed with WMARK_* defines accessed via helpers.  All call sites that
use zone->pages_* are updated to use the helpers for accessing the values
and the array offsets for setting.

Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
Reviewed-by: default avatarChristoph Lameter <cl@linux-foundation.org>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Cc: Dave Hansen <dave@linux.vnet.ibm.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent a3af9c38

Documentation/sysctl/vm.txt

+6 −5
Original line number Diff line number Diff line
@@ -233,8 +233,8 @@ These protections are added to score to judge whether this zone should be used 
for page allocation or should be reclaimed.



In this example, if normal pages (index=2) are required to this DMA zone and

pages_high is used for watermark, the kernel judges this zone should not be

used because pages_free(1355) is smaller than watermark + protection[2]

watermark[WMARK_HIGH] is used for watermark, the kernel judges this zone should

not be used because pages_free(1355) is smaller than watermark + protection[2]

(4 + 2004 = 2008). If this protection value is 0, this zone would be used for

normal page requirement. If requirement is DMA zone(index=0), protection[0]

(=0) is used.
@@ -280,9 +280,10 @@ The default value is 65536. 
min_free_kbytes:



This is used to force the Linux VM to keep a minimum number

of kilobytes free.  The VM uses this number to compute a pages_min

value for each lowmem zone in the system.  Each lowmem zone gets

a number of reserved free pages based proportionally on its size.

of kilobytes free.  The VM uses this number to compute a

watermark[WMARK_MIN] value for each lowmem zone in the system.

Each lowmem zone gets a number of reserved free pages based

proportionally on its size.



Some minimal amount of memory is needed to satisfy PF_MEMALLOC

allocations; if you set this to lower than 1024KB, your system will

Documentation/vm/balance

+9 −9
Original line number Diff line number Diff line
@@ -75,15 +75,15 @@ Page stealing from process memory and shm is done if stealing the page would 
alleviate memory pressure on any zone in the page's node that has fallen below

its watermark.



pages_min/pages_low/pages_high/low_on_memory/zone_wake_kswapd: These are 

per-zone fields, used to determine when a zone needs to be balanced. When

the number of pages falls below pages_min, the hysteric field low_on_memory

gets set. This stays set till the number of free pages becomes pages_high.

When low_on_memory is set, page allocation requests will try to free some

pages in the zone (providing GFP_WAIT is set in the request). Orthogonal

to this, is the decision to poke kswapd to free some zone pages. That

decision is not hysteresis based, and is done when the number of free

pages is below pages_low; in which case zone_wake_kswapd is also set.

watemark[WMARK_MIN/WMARK_LOW/WMARK_HIGH]/low_on_memory/zone_wake_kswapd: These

are per-zone fields, used to determine when a zone needs to be balanced. When

the number of pages falls below watermark[WMARK_MIN], the hysteric field

low_on_memory gets set. This stays set till the number of free pages becomes

watermark[WMARK_HIGH]. When low_on_memory is set, page allocation requests will

try to free some pages in the zone (providing GFP_WAIT is set in the request).

Orthogonal to this, is the decision to poke kswapd to free some zone pages.

That decision is not hysteresis based, and is done when the number of free

pages is below watermark[WMARK_LOW]; in which case zone_wake_kswapd is also set.





(Good) Ideas that I have heard:

arch/m32r/mm/discontig.c

+3 −3
Original line number Diff line number Diff line
@@ -154,9 +154,9 @@ unsigned long __init zone_sizes_init(void) 
	 *  Use all area of internal RAM.

	 *  see __alloc_pages()

	 */

	NODE_DATA(1)->node_zones->pages_min = 0;

	NODE_DATA(1)->node_zones->pages_low = 0;

	NODE_DATA(1)->node_zones->pages_high = 0;

	NODE_DATA(1)->node_zones->watermark[WMARK_MIN] = 0;

	NODE_DATA(1)->node_zones->watermark[WMARK_LOW] = 0;

	NODE_DATA(1)->node_zones->watermark[WMARK_HIGH] = 0;



	return holes;

}

include/linux/mmzone.h

+15 −1
Original line number Diff line number Diff line
@@ -163,6 +163,17 @@ static inline int is_unevictable_lru(enum lru_list l) 
#endif

}



enum zone_watermarks {

	WMARK_MIN,

	WMARK_LOW,

	WMARK_HIGH,

	NR_WMARK

};



#define min_wmark_pages(z) (z->watermark[WMARK_MIN])

#define low_wmark_pages(z) (z->watermark[WMARK_LOW])

#define high_wmark_pages(z) (z->watermark[WMARK_HIGH])



struct per_cpu_pages {

	int count;		/* number of pages in the list */

	int high;		/* high watermark, emptying needed */
@@ -275,7 +286,10 @@ struct zone_reclaim_stat { 


struct zone {

	/* Fields commonly accessed by the page allocator */

	unsigned long		pages_min, pages_low, pages_high;



	/* zone watermarks, access with *_wmark_pages(zone) macros */

	unsigned long watermark[NR_WMARK];



	/*

	 * We don't know if the memory that we're going to allocate will be freeable

	 * or/and it will be released eventually, so to avoid totally wasting several

mm/page_alloc.c

+26 −25
Original line number Diff line number Diff line
@@ -1150,10 +1150,15 @@ failed: 
	return NULL;

}



#define ALLOC_NO_WATERMARKS	0x01 /* don't check watermarks at all */

#define ALLOC_WMARK_MIN		0x02 /* use pages_min watermark */

#define ALLOC_WMARK_LOW		0x04 /* use pages_low watermark */

#define ALLOC_WMARK_HIGH	0x08 /* use pages_high watermark */

/* The ALLOC_WMARK bits are used as an index to zone->watermark */

#define ALLOC_WMARK_MIN		WMARK_MIN

#define ALLOC_WMARK_LOW		WMARK_LOW

#define ALLOC_WMARK_HIGH	WMARK_HIGH

#define ALLOC_NO_WATERMARKS	0x04 /* don't check watermarks at all */



/* Mask to get the watermark bits */

#define ALLOC_WMARK_MASK	(ALLOC_NO_WATERMARKS-1)



#define ALLOC_HARDER		0x10 /* try to alloc harder */

#define ALLOC_HIGH		0x20 /* __GFP_HIGH set */

#define ALLOC_CPUSET		0x40 /* check for correct cpuset */
@@ -1440,14 +1445,10 @@ zonelist_scan: 
			!cpuset_zone_allowed_softwall(zone, gfp_mask))

				goto try_next_zone;



		BUILD_BUG_ON(ALLOC_NO_WATERMARKS < NR_WMARK);

		if (!(alloc_flags & ALLOC_NO_WATERMARKS)) {

			unsigned long mark;

			if (alloc_flags & ALLOC_WMARK_MIN)

				mark = zone->pages_min;

			else if (alloc_flags & ALLOC_WMARK_LOW)

				mark = zone->pages_low;

			else

				mark = zone->pages_high;

			mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];

			if (!zone_watermark_ok(zone, order, mark,

				    classzone_idx, alloc_flags)) {

				if (!zone_reclaim_mode ||
@@ -1959,7 +1960,7 @@ static unsigned int nr_free_zone_pages(int offset) 


	for_each_zone_zonelist(zone, z, zonelist, offset) {

		unsigned long size = zone->present_pages;

		unsigned long high = zone->pages_high;

		unsigned long high = high_wmark_pages(zone);

		if (size > high)

			sum += size - high;

	}
@@ -2096,9 +2097,9 @@ void show_free_areas(void) 
			"\n",

			zone->name,

			K(zone_page_state(zone, NR_FREE_PAGES)),

			K(zone->pages_min),

			K(zone->pages_low),

			K(zone->pages_high),

			K(min_wmark_pages(zone)),

			K(low_wmark_pages(zone)),

			K(high_wmark_pages(zone)),

			K(zone_page_state(zone, NR_ACTIVE_ANON)),

			K(zone_page_state(zone, NR_INACTIVE_ANON)),

			K(zone_page_state(zone, NR_ACTIVE_FILE)),
@@ -2702,8 +2703,8 @@ static inline unsigned long wait_table_bits(unsigned long size) 


/*

 * Mark a number of pageblocks as MIGRATE_RESERVE. The number

 * of blocks reserved is based on zone->pages_min. The memory within the

 * reserve will tend to store contiguous free pages. Setting min_free_kbytes

 * of blocks reserved is based on min_wmark_pages(zone). The memory within

 * the reserve will tend to store contiguous free pages. Setting min_free_kbytes

 * higher will lead to a bigger reserve which will get freed as contiguous

 * blocks as reclaim kicks in

 */
@@ -2716,7 +2717,7 @@ static void setup_zone_migrate_reserve(struct zone *zone) 
	/* Get the start pfn, end pfn and the number of blocks to reserve */

	start_pfn = zone->zone_start_pfn;

	end_pfn = start_pfn + zone->spanned_pages;

	reserve = roundup(zone->pages_min, pageblock_nr_pages) >>

	reserve = roundup(min_wmark_pages(zone), pageblock_nr_pages) >>

							pageblock_order;



	for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
@@ -4319,8 +4320,8 @@ static void calculate_totalreserve_pages(void) 
					max = zone->lowmem_reserve[j];

			}



			/* we treat pages_high as reserved pages. */

			max += zone->pages_high;

			/* we treat the high watermark as reserved pages. */

			max += high_wmark_pages(zone);



			if (max > zone->present_pages)

				max = zone->present_pages;
@@ -4400,7 +4401,7 @@ void setup_per_zone_pages_min(void) 
			 * need highmem pages, so cap pages_min to a small

			 * value here.

			 *

			 * The (pages_high-pages_low) and (pages_low-pages_min)

			 * The WMARK_HIGH-WMARK_LOW and (WMARK_LOW-WMARK_MIN)

			 * deltas controls asynch page reclaim, and so should

			 * not be capped for highmem.

			 */
@@ -4411,17 +4412,17 @@ void setup_per_zone_pages_min(void) 
				min_pages = SWAP_CLUSTER_MAX;

			if (min_pages > 128)

				min_pages = 128;

			zone->pages_min = min_pages;

			zone->watermark[WMARK_MIN] = min_pages;

		} else {

			/*

			 * If it's a lowmem zone, reserve a number of pages

			 * proportionate to the zone's size.

			 */

			zone->pages_min = tmp;

			zone->watermark[WMARK_MIN] = tmp;

		}



		zone->pages_low   = zone->pages_min + (tmp >> 2);

		zone->pages_high  = zone->pages_min + (tmp >> 1);

		zone->watermark[WMARK_LOW]  = min_wmark_pages(zone) + (tmp >> 2);

		zone->watermark[WMARK_HIGH] = min_wmark_pages(zone) + (tmp >> 1);

		setup_zone_migrate_reserve(zone);

		spin_unlock_irqrestore(&zone->lock, flags);

	}
@@ -4566,7 +4567,7 @@ int sysctl_min_slab_ratio_sysctl_handler(ctl_table *table, int write, 
 *	whenever sysctl_lowmem_reserve_ratio changes.

 *

 * The reserve ratio obviously has absolutely no relation with the

 * pages_min watermarks. The lowmem reserve ratio can only make sense

 * minimum watermarks. The lowmem reserve ratio can only make sense

 * if in function of the boot time zone sizes.

 */

int lowmem_reserve_ratio_sysctl_handler(ctl_table *table, int write,

CRA Git | Maintained and supported by SUSTech CRA and CCSE