Merge branch 'akpm' (patches from Andrew)

8. LRU

======

        Each memcg has its own private LRU. Now, its handling is under global

	VM's control (means that it's handled under global pgdat->lru_lock).

	Almost all routines around memcg's LRU is called by global LRU's

	list management functions under pgdat->lru_lock.

	A special function is mem_cgroup_isolate_pages(). This scans

	memcg's private LRU and call __isolate_lru_page() to extract a page

	from LRU.

	(By __isolate_lru_page(), the page is removed from both of global and

	private LRU.)

	Each memcg has its own vector of LRUs (inactive anon, active anon,

	inactive file, active file, unevictable) of pages from each node,

	each LRU handled under a single lru_lock for that memcg and node.

9. Typical Tests.

=================

2.6 Locking

-----------

   lock_page_cgroup()/unlock_page_cgroup() should not be called under

   the i_pages lock.

Lock order is as follows:

   Other lock order is following:

  Page lock (PG_locked bit of page->flags)

    mm->page_table_lock or split pte_lock

      lock_page_memcg (memcg->move_lock)

        mapping->i_pages lock

          lruvec->lru_lock.

   PG_locked.

     mm->page_table_lock

         pgdat->lru_lock

	   lock_page_cgroup.

  In many cases, just lock_page_cgroup() is called.

  per-zone-per-cgroup LRU (cgroup's private LRU) is just guarded by

  pgdat->lru_lock, it has no lock of its own.

Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by

lruvec->lru_lock; PG_lru bit of page->flags is cleared before

isolating a page from its LRU under lruvec->lru_lock.

2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM)

-----------------------------------------------

Broadly speaking, pages are taken off the LRU lock in bulk and

freed in batch with a page list. Significant amounts of activity here could

indicate that the system is under memory pressure and can also indicate

contention on the zone->lru_lock.

contention on the lruvec->lru_lock.

4. Per-CPU Allocator Activity

=============================

memory x86_64 systems.

To illustrate this with an example, a non-NUMA x86_64 platform with 128GB of

main memory will have over 32 million 4k pages in a single zone.  When a large

main memory will have over 32 million 4k pages in a single node.  When a large

fraction of these pages are not evictable for any reason [see below], vmscan

will spend a lot of time scanning the LRU lists looking for the small fraction

of pages that are evictable.  This can result in a situation where all CPUs are

The Unevictable Page List

-------------------------

The Unevictable LRU infrastructure consists of an additional, per-zone, LRU list

The Unevictable LRU infrastructure consists of an additional, per-node, LRU list

called the "unevictable" list and an associated page flag, PG_unevictable, to

indicate that the page is being managed on the unevictable list.

swap-backed pages.  This differentiation is only important while the pages are,

in fact, evictable.

The unevictable list benefits from the "arrayification" of the per-zone LRU

The unevictable list benefits from the "arrayification" of the per-node LRU

lists and statistics originally proposed and posted by Christoph Lameter.

The unevictable list does not use the LRU pagevec mechanism. Rather,

unevictable pages are placed directly on the page's zone's unevictable list

under the zone lru_lock.  This allows us to prevent the stranding of pages on

the unevictable list when one task has the page isolated from the LRU and other

tasks are changing the "evictability" state of the page.

Memory Control Group Interaction

--------------------------------

memory controller; see Documentation/admin-guide/cgroup-v1/memory.rst] by extending the

lru_list enum.

The memory controller data structure automatically gets a per-zone unevictable

list as a result of the "arrayification" of the per-zone LRU lists (one per

The memory controller data structure automatically gets a per-node unevictable

list as a result of the "arrayification" of the per-node LRU lists (one per

lru_list enum element).  The memory controller tracks the movement of pages to

and from the unevictable list.

active/inactive LRU lists for vmscan to deal with.  vmscan checks for such

pages in all of the shrink_{active|inactive|page}_list() functions and will

"cull" such pages that it encounters: that is, it diverts those pages to the

unevictable list for the zone being scanned.

unevictable list for the node being scanned.

There may be situations where a page is mapped into a VM_LOCKED VMA, but the

page is not marked as PG_mlocked.  Such pages will make it all the way to

page from the LRU, as it is likely on the appropriate active or inactive list

at that time.  If the isolate_lru_page() succeeds, mlock_vma_page() will put

back the page - by calling putback_lru_page() - which will notice that the page

is now mlocked and divert the page to the zone's unevictable list.  If

is now mlocked and divert the page to the node's unevictable list.  If

mlock_vma_page() is unable to isolate the page from the LRU, vmscan will handle

it later if and when it attempts to reclaim the page.

     unevictable list in mlock_vma_page().

shrink_inactive_list() also diverts any unevictable pages that it finds on the

inactive lists to the appropriate zone's unevictable list.

inactive lists to the appropriate node's unevictable list.

shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd

after shrink_active_list() had moved them to the inactive list, or pages mapped

struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *);

static inline bool lruvec_holds_page_lru_lock(struct page *page,

					      struct lruvec *lruvec)

{

	pg_data_t *pgdat = page_pgdat(page);

	const struct mem_cgroup *memcg;

	struct mem_cgroup_per_node *mz;

	if (mem_cgroup_disabled())

		return lruvec == &pgdat->__lruvec;

	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);

	memcg = page_memcg(page) ? : root_mem_cgroup;

	return lruvec->pgdat == pgdat && mz->memcg == memcg;

}

struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p);

struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm);

struct mem_cgroup *get_mem_cgroup_from_page(struct page *page);

struct lruvec *lock_page_lruvec(struct page *page);

struct lruvec *lock_page_lruvec_irq(struct page *page);

struct lruvec *lock_page_lruvec_irqsave(struct page *page,

						unsigned long *flags);

#ifdef CONFIG_DEBUG_VM

void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page);

#else

static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)

{

}

#endif

static inline

struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){

	return css ? container_of(css, struct mem_cgroup, css) : NULL;

	return &pgdat->__lruvec;

}

static inline bool lruvec_holds_page_lru_lock(struct page *page,

					      struct lruvec *lruvec)

{

	pg_data_t *pgdat = page_pgdat(page);

	return lruvec == &pgdat->__lruvec;

}

static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memcg)

{

	return NULL;

{

}

static inline struct lruvec *lock_page_lruvec(struct page *page)

{

	struct pglist_data *pgdat = page_pgdat(page);

	spin_lock(&pgdat->__lruvec.lru_lock);

	return &pgdat->__lruvec;

}

static inline struct lruvec *lock_page_lruvec_irq(struct page *page)

{

	struct pglist_data *pgdat = page_pgdat(page);

	spin_lock_irq(&pgdat->__lruvec.lru_lock);

	return &pgdat->__lruvec;

}

static inline struct lruvec *lock_page_lruvec_irqsave(struct page *page,

		unsigned long *flagsp)

{

	struct pglist_data *pgdat = page_pgdat(page);

	spin_lock_irqsave(&pgdat->__lruvec.lru_lock, *flagsp);

	return &pgdat->__lruvec;

}

static inline struct mem_cgroup *

mem_cgroup_iter(struct mem_cgroup *root,

		struct mem_cgroup *prev,

void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx)

{

}

static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page)

{

}

#endif /* CONFIG_MEMCG */

/* idx can be of type enum memcg_stat_item or node_stat_item */

	return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec));

}

static inline void unlock_page_lruvec(struct lruvec *lruvec)

{

	spin_unlock(&lruvec->lru_lock);

}

static inline void unlock_page_lruvec_irq(struct lruvec *lruvec)

{

	spin_unlock_irq(&lruvec->lru_lock);

}

static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec,

		unsigned long flags)

{

	spin_unlock_irqrestore(&lruvec->lru_lock, flags);

}

/* Don't lock again iff page's lruvec locked */

static inline struct lruvec *relock_page_lruvec_irq(struct page *page,

		struct lruvec *locked_lruvec)

{

	if (locked_lruvec) {

		if (lruvec_holds_page_lru_lock(page, locked_lruvec))

			return locked_lruvec;

		unlock_page_lruvec_irq(locked_lruvec);

	}

	return lock_page_lruvec_irq(page);

}

/* Don't lock again iff page's lruvec locked */

static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page,

		struct lruvec *locked_lruvec, unsigned long *flags)

{

	if (locked_lruvec) {

		if (lruvec_holds_page_lru_lock(page, locked_lruvec))

			return locked_lruvec;

		unlock_page_lruvec_irqrestore(locked_lruvec, *flags);

	}

	return lock_page_lruvec_irqsave(page, flags);

}

#ifdef CONFIG_CGROUP_WRITEBACK

struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb);