Merge branch 'for-3.7' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup

  3.1 Overview

  3.2 Synchronization

  3.3 Subsystem API

4. Questions

4. Extended attributes usage

5. Questions

1. Control Groups

=================

At any one time there may be multiple active hierarchies of task

cgroups. Each hierarchy is a partition of all tasks in the system.

User level code may create and destroy cgroups by name in an

User-level code may create and destroy cgroups by name in an

instance of the cgroup virtual file system, specify and query to

which cgroup a task is assigned, and list the task pids assigned to

which cgroup a task is assigned, and list the task PIDs assigned to

a cgroup. Those creations and assignments only affect the hierarchy

associated with that instance of the cgroup file system.

tracking. The intention is that other subsystems hook into the generic

cgroup support to provide new attributes for cgroups, such as

accounting/limiting the resources which processes in a cgroup can

access. For example, cpusets (see Documentation/cgroups/cpusets.txt) allows

access. For example, cpusets (see Documentation/cgroups/cpusets.txt) allow

you to associate a set of CPUs and a set of memory nodes with the

tasks in each cgroup.

----------------------------

There are multiple efforts to provide process aggregations in the

Linux kernel, mainly for resource tracking purposes. Such efforts

Linux kernel, mainly for resource-tracking purposes. Such efforts

include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server

namespaces. These all require the basic notion of a

grouping/partitioning of processes, with newly forked processes ending

in the same group (cgroup) as their parent process.

up in the same group (cgroup) as their parent process.

The kernel cgroup patch provides the minimum essential kernel

mechanisms required to efficiently implement such groups. It has

                               / \

               Professors (15%)  students (5%)

Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd go

into NFS network class.

Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes

into the NFS network class.

At the same time Firefox/Lynx will share an appropriate CPU/Memory class

depending on who launched it (prof/student).

With the ability to classify tasks differently for different resources

(by putting those resource subsystems in different hierarchies) then

(by putting those resource subsystems in different hierarchies),

the admin can easily set up a script which receives exec notifications

and depending on who is launching the browser he can

appropriate network and other resource class.  This may lead to

proliferation of such cgroups.

Also lets say that the administrator would like to give enhanced network

Also let's say that the administrator would like to give enhanced network

access temporarily to a student's browser (since it is night and the user

wants to do online gaming :))  OR give one of the students simulation

apps enhanced CPU power,

wants to do online gaming :))  OR give one of the student's simulation

apps enhanced CPU power.

With ability to write pids directly to resource classes, it's just a

With ability to write PIDs directly to resource classes, it's just a

matter of:

       # echo pid > /sys/fs/cgroup/network/<new_class>/tasks

       (after some time)

       # echo pid > /sys/fs/cgroup/network/<orig_class>/tasks

Without this ability, he would have to split the cgroup into

Without this ability, the administrator would have to split the cgroup into

multiple separate ones and then associate the new cgroups with the

new resource classes.

 - A cgroup hierarchy filesystem can be mounted for browsing and

   manipulation from user space.

 - You can list all the tasks (by pid) attached to any cgroup.

 - You can list all the tasks (by PID) attached to any cgroup.

The implementation of cgroups requires a few, simple hooks

into the rest of the kernel, none in performance critical paths:

into the rest of the kernel, none in performance-critical paths:

 - in init/main.c, to initialize the root cgroups and initial

   css_set at system boot.

 - in fork and exit, to attach and detach a task from its css_set.

In addition a new file system, of type "cgroup" may be mounted, to

In addition, a new file system of type "cgroup" may be mounted, to

enable browsing and modifying the cgroups presently known to the

kernel.  When mounting a cgroup hierarchy, you may specify a

comma-separated list of subsystems to mount as the filesystem mount

Each cgroup is represented by a directory in the cgroup file system

containing the following files describing that cgroup:

 - tasks: list of tasks (by pid) attached to that cgroup.  This list

   is not guaranteed to be sorted.  Writing a thread id into this file

 - tasks: list of tasks (by PID) attached to that cgroup.  This list

   is not guaranteed to be sorted.  Writing a thread ID into this file

   moves the thread into this cgroup.

 - cgroup.procs: list of tgids in the cgroup.  This list is not

   guaranteed to be sorted or free of duplicate tgids, and userspace

 - cgroup.procs: list of thread group IDs in the cgroup.  This list is

   not guaranteed to be sorted or free of duplicate TGIDs, and userspace

   should sort/uniquify the list if this property is required.

   Writing a thread group id into this file moves all threads in that

   Writing a thread group ID into this file moves all threads in that

   group into this cgroup.

 - notify_on_release flag: run the release agent on exit?

 - release_agent: the path to use for release notifications (this file

When a task is moved from one cgroup to another, it gets a new

css_set pointer - if there's an already existing css_set with the

desired collection of cgroups then that group is reused, else a new

desired collection of cgroups then that group is reused, otherwise a new

css_set is allocated. The appropriate existing css_set is located by

looking into a hash table.

removal of abandoned cgroups.  The default value of

notify_on_release in the root cgroup at system boot is disabled

(0).  The default value of other cgroups at creation is the current

value of their parents notify_on_release setting. The default value of

value of their parents' notify_on_release settings. The default value of

a cgroup hierarchy's release_agent path is empty.

1.5 What does clone_children do ?

 4) Create the new cgroup by doing mkdir's and write's (or echo's) in

    the /sys/fs/cgroup virtual file system.

 5) Start a task that will be the "founding father" of the new job.

 6) Attach that task to the new cgroup by writing its pid to the

 6) Attach that task to the new cgroup by writing its PID to the

    /sys/fs/cgroup/cpuset/tasks file for that cgroup.

 7) fork, exec or clone the job tasks from this founding father task.

2.1 Basic Usage

---------------

Creating, modifying, using the cgroups can be done through the cgroup

Creating, modifying, using cgroups can be done through the cgroup

virtual filesystem.

To mount a cgroup hierarchy with all available subsystems, type:

# echo 0 > tasks

You can use the cgroup.procs file instead of the tasks file to move all

threads in a threadgroup at once. Echoing the pid of any task in a

threads in a threadgroup at once. Echoing the PID of any task in a

threadgroup to cgroup.procs causes all tasks in that threadgroup to be

be attached to the cgroup. Writing 0 to cgroup.procs moves all tasks

in the writing task's threadgroup.

There is mechanism which allows to get notifications about changing

status of a cgroup.

To register new notification handler you need:

To register a new notification handler you need to:

 - create a file descriptor for event notification using eventfd(2);

 - open a control file to be monitored (e.g. memory.usage_in_bytes);

 - write "<event_fd> <control_fd> <args>" to cgroup.event_control.

eventfd will be woken up by control file implementation or when the

cgroup is removed.

To unregister notification handler just close eventfd.

To unregister a notification handler just close eventfd.

NOTE: Support of notifications should be implemented for the control

file. See documentation for the subsystem.

Each kernel subsystem that wants to hook into the generic cgroup

system needs to create a cgroup_subsys object. This contains

various methods, which are callbacks from the cgroup system, along

with a subsystem id which will be assigned by the cgroup system.

with a subsystem ID which will be assigned by the cgroup system.

Other fields in the cgroup_subsys object include:

  at system boot.

Each cgroup object created by the system has an array of pointers,

indexed by subsystem id; this pointer is entirely managed by the

indexed by subsystem ID; this pointer is entirely managed by the

subsystem; the generic cgroup code will never touch this pointer.

3.2 Synchronization

Called during cgroup_create() to do any parameter

initialization which might be required before a task could attach.  For

example in cpusets, no task may attach before 'cpus' and 'mems' are set

example, in cpusets, no task may attach before 'cpus' and 'mems' are set

up.

void bind(struct cgroup *root)

the default hierarchy (which never has sub-cgroups) and a hierarchy

that is being created/destroyed (and hence has no sub-cgroups).

4. Questions

4. Extended attribute usage

===========================

cgroup filesystem supports certain types of extended attributes in its

directories and files.  The current supported types are:

	- Trusted (XATTR_TRUSTED)

	- Security (XATTR_SECURITY)

Both require CAP_SYS_ADMIN capability to set.

Like in tmpfs, the extended attributes in cgroup filesystem are stored

using kernel memory and it's advised to keep the usage at minimum.  This

is the reason why user defined extended attributes are not supported, since

any user can do it and there's no limit in the value size.

The current known users for this feature are SELinux to limit cgroup usage

in containers and systemd for assorted meta data like main PID in a cgroup

(systemd creates a cgroup per service).

5. Questions

============

Q: what's up with this '/bin/echo' ?

Q: When I attach processes, only the first of the line gets really attached !

A: We can only return one error code per call to write(). So you should also

   put only ONE pid.

   put only ONE PID.

#include <net/netns/generic.h>

#include <net/rtnetlink.h>

#include <net/sock.h>

#include <net/cls_cgroup.h>

#include <asm/uaccess.h>

EXPORT_SYMBOL(generic_listxattr);

EXPORT_SYMBOL(generic_setxattr);

EXPORT_SYMBOL(generic_removexattr);

/*

 * Allocate new xattr and copy in the value; but leave the name to callers.

 */

struct simple_xattr *simple_xattr_alloc(const void *value, size_t size)

{

	struct simple_xattr *new_xattr;

	size_t len;

	/* wrap around? */

	len = sizeof(*new_xattr) + size;

	if (len <= sizeof(*new_xattr))

		return NULL;

	new_xattr = kmalloc(len, GFP_KERNEL);

	if (!new_xattr)

		return NULL;

	new_xattr->size = size;

	memcpy(new_xattr->value, value, size);

	return new_xattr;

}

/*

 * xattr GET operation for in-memory/pseudo filesystems

 */

int simple_xattr_get(struct simple_xattrs *xattrs, const char *name,

		     void *buffer, size_t size)

{

	struct simple_xattr *xattr;

	int ret = -ENODATA;

	spin_lock(&xattrs->lock);

	list_for_each_entry(xattr, &xattrs->head, list) {

		if (strcmp(name, xattr->name))

			continue;

		ret = xattr->size;

		if (buffer) {

			if (size < xattr->size)

				ret = -ERANGE;

			else

				memcpy(buffer, xattr->value, xattr->size);

		}

		break;

	}

	spin_unlock(&xattrs->lock);

	return ret;

}

static int __simple_xattr_set(struct simple_xattrs *xattrs, const char *name,

			      const void *value, size_t size, int flags)

{

	struct simple_xattr *xattr;

	struct simple_xattr *uninitialized_var(new_xattr);

	int err = 0;

	/* value == NULL means remove */

	if (value) {

		new_xattr = simple_xattr_alloc(value, size);

		if (!new_xattr)

			return -ENOMEM;

		new_xattr->name = kstrdup(name, GFP_KERNEL);

		if (!new_xattr->name) {

			kfree(new_xattr);

			return -ENOMEM;

		}

	}

	spin_lock(&xattrs->lock);

	list_for_each_entry(xattr, &xattrs->head, list) {

		if (!strcmp(name, xattr->name)) {

			if (flags & XATTR_CREATE) {

				xattr = new_xattr;

				err = -EEXIST;

			} else if (new_xattr) {

				list_replace(&xattr->list, &new_xattr->list);

			} else {

				list_del(&xattr->list);

			}

			goto out;

		}

	}

	if (flags & XATTR_REPLACE) {

		xattr = new_xattr;

		err = -ENODATA;

	} else {

		list_add(&new_xattr->list, &xattrs->head);

		xattr = NULL;

	}

out:

	spin_unlock(&xattrs->lock);

	if (xattr) {

		kfree(xattr->name);

		kfree(xattr);

	}

	return err;

}

/**

 * simple_xattr_set - xattr SET operation for in-memory/pseudo filesystems

 * @xattrs: target simple_xattr list

 * @name: name of the new extended attribute

 * @value: value of the new xattr. If %NULL, will remove the attribute

 * @size: size of the new xattr

 * @flags: %XATTR_{CREATE|REPLACE}

 *

 * %XATTR_CREATE is set, the xattr shouldn't exist already; otherwise fails

 * with -EEXIST.  If %XATTR_REPLACE is set, the xattr should exist;

 * otherwise, fails with -ENODATA.

 *

 * Returns 0 on success, -errno on failure.

 */

int simple_xattr_set(struct simple_xattrs *xattrs, const char *name,

		     const void *value, size_t size, int flags)

{

	if (size == 0)

		value = ""; /* empty EA, do not remove */

	return __simple_xattr_set(xattrs, name, value, size, flags);

}

/*

 * xattr REMOVE operation for in-memory/pseudo filesystems

 */

int simple_xattr_remove(struct simple_xattrs *xattrs, const char *name)

{

	return __simple_xattr_set(xattrs, name, NULL, 0, XATTR_REPLACE);

}

static bool xattr_is_trusted(const char *name)

{

	return !strncmp(name, XATTR_TRUSTED_PREFIX, XATTR_TRUSTED_PREFIX_LEN);

}

/*

 * xattr LIST operation for in-memory/pseudo filesystems

 */

ssize_t simple_xattr_list(struct simple_xattrs *xattrs, char *buffer,

			  size_t size)

{

	bool trusted = capable(CAP_SYS_ADMIN);

	struct simple_xattr *xattr;

	size_t used = 0;

	spin_lock(&xattrs->lock);

	list_for_each_entry(xattr, &xattrs->head, list) {

		size_t len;

		/* skip "trusted." attributes for unprivileged callers */

		if (!trusted && xattr_is_trusted(xattr->name))

			continue;

		len = strlen(xattr->name) + 1;

		used += len;

		if (buffer) {

			if (size < used) {

				used = -ERANGE;

				break;

			}

			memcpy(buffer, xattr->name, len);

			buffer += len;

		}

	}

	spin_unlock(&xattrs->lock);

	return used;

}

/*

 * Adds an extended attribute to the list

 */

void simple_xattr_list_add(struct simple_xattrs *xattrs,

			   struct simple_xattr *new_xattr)

{

	spin_lock(&xattrs->lock);

	list_add(&new_xattr->list, &xattrs->head);

	spin_unlock(&xattrs->lock);

}

#include <linux/rwsem.h>

#include <linux/idr.h>

#include <linux/workqueue.h>

#include <linux/xattr.h>

#ifdef CONFIG_CGROUPS

/* Define the enumeration of all builtin cgroup subsystems */

#define SUBSYS(_x) _x ## _subsys_id,

#define IS_SUBSYS_ENABLED(option) IS_ENABLED(option)

enum cgroup_subsys_id {

#include <linux/cgroup_subsys.h>

	CGROUP_BUILTIN_SUBSYS_COUNT

	CGROUP_SUBSYS_COUNT,

};

#undef IS_SUBSYS_ENABLED

#undef SUBSYS

/*

 * This define indicates the maximum number of subsystems that can be loaded

 * at once. We limit to this many since cgroupfs_root has subsys_bits to keep

 * track of all of them.

 */

#define CGROUP_SUBSYS_COUNT (BITS_PER_BYTE*sizeof(unsigned long))

/* Per-subsystem/per-cgroup state maintained by the system. */

struct cgroup_subsys_state {

	/* List of events which userspace want to receive */

	struct list_head event_list;

	spinlock_t event_list_lock;

	/* directory xattrs */

	struct simple_xattrs xattrs;

};

/*

	/* CFTYPE_* flags */

	unsigned int flags;

	/* file xattrs */

	struct simple_xattrs xattrs;

	int (*open)(struct inode *inode, struct file *file);

	ssize_t (*read)(struct cgroup *cgrp, struct cftype *cft,

			struct file *file,

 */

struct cftype_set {

	struct list_head		node;	/* chained at subsys->cftsets */

	const struct cftype		*cfts;

	struct cftype			*cfts;

};

struct cgroup_scanner {

	void *data;

};

int cgroup_add_cftypes(struct cgroup_subsys *ss, const struct cftype *cfts);

int cgroup_rm_cftypes(struct cgroup_subsys *ss, const struct cftype *cfts);

int cgroup_add_cftypes(struct cgroup_subsys *ss, struct cftype *cfts);

int cgroup_rm_cftypes(struct cgroup_subsys *ss, struct cftype *cfts);

int cgroup_is_removed(const struct cgroup *cgrp);

};

#define SUBSYS(_x) extern struct cgroup_subsys _x ## _subsys;

#define IS_SUBSYS_ENABLED(option) IS_BUILTIN(option)

#include <linux/cgroup_subsys.h>

#undef IS_SUBSYS_ENABLED

#undef SUBSYS

static inline struct cgroup_subsys_state *cgroup_subsys_state(

/* */

#ifdef CONFIG_CPUSETS

#if IS_SUBSYS_ENABLED(CONFIG_CPUSETS)

SUBSYS(cpuset)

#endif

/* */

#ifdef CONFIG_CGROUP_DEBUG

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_DEBUG)

SUBSYS(debug)

#endif

/* */

#ifdef CONFIG_CGROUP_SCHED

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_SCHED)

SUBSYS(cpu_cgroup)

#endif

/* */

#ifdef CONFIG_CGROUP_CPUACCT

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_CPUACCT)

SUBSYS(cpuacct)

#endif

/* */

#ifdef CONFIG_MEMCG

#if IS_SUBSYS_ENABLED(CONFIG_MEMCG)

SUBSYS(mem_cgroup)

#endif

/* */

#ifdef CONFIG_CGROUP_DEVICE

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_DEVICE)

SUBSYS(devices)

#endif

/* */

#ifdef CONFIG_CGROUP_FREEZER

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_FREEZER)

SUBSYS(freezer)

#endif

/* */

#ifdef CONFIG_NET_CLS_CGROUP

#if IS_SUBSYS_ENABLED(CONFIG_NET_CLS_CGROUP)

SUBSYS(net_cls)

#endif

/* */

#ifdef CONFIG_BLK_CGROUP

#if IS_SUBSYS_ENABLED(CONFIG_BLK_CGROUP)

SUBSYS(blkio)

#endif

/* */

#ifdef CONFIG_CGROUP_PERF

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_PERF)

SUBSYS(perf)

#endif

/* */

#ifdef CONFIG_NETPRIO_CGROUP

#if IS_SUBSYS_ENABLED(CONFIG_NETPRIO_CGROUP)

SUBSYS(net_prio)

#endif

/* */

#ifdef CONFIG_CGROUP_HUGETLB

#if IS_SUBSYS_ENABLED(CONFIG_CGROUP_HUGETLB)

SUBSYS(hugetlb)

#endif