Merge branch 'rcu/next' of...

(In contrast, implementation of RCU is permitted only in software licensed

under either GPL or LGPL.  Sorry!!!)

In 1987, Rashid et al. described lazy TLB-flush [RichardRashid87a].

At first glance, this has nothing to do with RCU, but nevertheless

this paper helped inspire the update-side batching used in the later

RCU implementation in DYNIX/ptx.  In 1988, Barbara Liskov published

a description of Argus that noted that use of out-of-date values can

be tolerated in some situations.  Thus, this paper provides some early

theoretical justification for use of stale data.

In 1990, Pugh [Pugh90] noted that explicitly tracking which threads

were reading a given data structure permitted deferred free to operate

in the presence of non-terminating threads.  However, this explicit

to see how much of the performance advantage reported in 1990 remains

today.

At about this same time, Adams [Adams91] described ``chaotic relaxation'',

where the normal barriers between successive iterations of convergent

numerical algorithms are relaxed, so that iteration $n$ might use

data from iteration $n-1$ or even $n-2$.  This introduces error,

which typically slows convergence and thus increases the number of

At about this same time, Andrews [Andrews91textbook] described ``chaotic

relaxation'', where the normal barriers between successive iterations

of convergent numerical algorithms are relaxed, so that iteration $n$

might use data from iteration $n-1$ or even $n-2$.  This introduces

error, which typically slows convergence and thus increases the number of

iterations required.  However, this increase is sometimes more than made

up for by a reduction in the number of expensive barrier operations,

which are otherwise required to synchronize the threads at the end

In 1992, Henry (now Alexia) Massalin completed a dissertation advising

parallel programmers to defer processing when feasible to simplify

synchronization.  RCU makes extremely heavy use of this advice.

synchronization [HMassalinPhD].  RCU makes extremely heavy use of

this advice.

In 1993, Jacobson [Jacobson93] verbally described what is perhaps the

simplest deferred-free technique: simply waiting a fixed amount of time

systems made pervasive use of RCU in place of "existence locks", which

greatly simplifies locking hierarchies and helps avoid deadlocks.

2001 saw the first RCU presentation involving Linux [McKenney01a]

at OLS.  The resulting abundance of RCU patches was presented the

following year [McKenney02a], and use of RCU in dcache was first

described that same year [Linder02a].

The year 2000 saw an email exchange that would likely have

led to yet another independent invention of something like RCU

[RustyRussell2000a,RustyRussell2000b].  Instead, 2001 saw the first

RCU presentation involving Linux [McKenney01a] at OLS.  The resulting

abundance of RCU patches was presented the following year [McKenney02a],

and use of RCU in dcache was first described that same year [Linder02a].

Also in 2002, Michael [Michael02b,Michael02a] presented "hazard-pointer"

techniques that defer the destruction of data structures to simplify

non-blocking synchronization (wait-free synchronization, lock-free

synchronization, and obstruction-free synchronization are all examples of

non-blocking synchronization).  In particular, this technique eliminates

locking, reduces contention, reduces memory latency for readers, and

parallelizes pipeline stalls and memory latency for writers.  However,

these techniques still impose significant read-side overhead in the

form of memory barriers.  Researchers at Sun worked along similar lines

in the same timeframe [HerlihyLM02].  These techniques can be thought

of as inside-out reference counts, where the count is represented by the

number of hazard pointers referencing a given data structure rather than

the more conventional counter field within the data structure itself.

The key advantage of inside-out reference counts is that they can be

stored in immortal variables, thus allowing races between access and

deletion to be avoided.

non-blocking synchronization).  The corresponding journal article appeared

in 2004 [MagedMichael04a].  This technique eliminates locking, reduces

contention, reduces memory latency for readers, and parallelizes pipeline

stalls and memory latency for writers.  However, these techniques still

impose significant read-side overhead in the form of memory barriers.

Researchers at Sun worked along similar lines in the same timeframe

[HerlihyLM02].  These techniques can be thought of as inside-out reference

counts, where the count is represented by the number of hazard pointers

referencing a given data structure rather than the more conventional

counter field within the data structure itself.  The key advantage

of inside-out reference counts is that they can be stored in immortal

variables, thus allowing races between access and deletion to be avoided.

By the same token, RCU can be thought of as a "bulk reference count",

where some form of reference counter covers all reference by a given CPU

In 2003, the K42 group described how RCU could be used to create

hot-pluggable implementations of operating-system functions [Appavoo03a].

Later that year saw a paper describing an RCU implementation of System

V IPC [Arcangeli03], and an introduction to RCU in Linux Journal

Later that year saw a paper describing an RCU implementation

of System V IPC [Arcangeli03] (following up on a suggestion by

Hugh Dickins [Dickins02a] and an implementation by Mingming Cao

[MingmingCao2002IPCRCU]), and an introduction to RCU in Linux Journal

[McKenney03a].

2004 has seen a Linux-Journal article on use of RCU in dcache

}

}

@phdthesis{HMassalinPhD

,author="H. Massalin"

,title="Synthesis: An Efficient Implementation of Fundamental Operating

System Services"

,school="Columbia University"

,address="New York, NY"

,year="1992"

,annotation={

	Mondo optimizing compiler.

	Wait-free stuff.

	Good advice: defer work to avoid synchronization.  See page 90

		(PDF page 106), Section 5.4, fourth bullet point.

}

}

@unpublished{Jacobson93

,author="Van Jacobson"

,title="Avoid Read-Side Locking Via Delayed Free"

[Viewed October 18, 2004]"

}

@conference{Michael02b

,author="Maged M. Michael"

,title="High Performance Dynamic Lock-Free Hash Tables and List-Based Sets"

,Year="2002"

,Month="August"

,booktitle="{Proceedings of the 14\textsuperscript{th} Annual ACM

Symposium on Parallel

Algorithms and Architecture}"

,pages="73-82"

,annotation={

Like the title says...

}

}

@Conference{Linder02a

,Author="Hanna Linder and Dipankar Sarma and Maneesh Soni"

,Title="Scalability of the Directory Entry Cache"

}

}

@conference{Michael02a

,author="Maged M. Michael"

,title="Safe Memory Reclamation for Dynamic Lock-Free Objects Using Atomic

Reads and Writes"

,Year="2002"

,Month="August"

,booktitle="{Proceedings of the 21\textsuperscript{st} Annual ACM

Symposium on Principles of Distributed Computing}"

,pages="21-30"

,annotation={

	Each thread keeps an array of pointers to items that it is

	currently referencing.	Sort of an inside-out garbage collection

	mechanism, but one that requires the accessing code to explicitly

	state its needs.  Also requires read-side memory barriers on

	most architectures.

}

}

@unpublished{Dickins02a

,author="Hugh Dickins"

,title="Use RCU for System-V IPC"

,note="private communication"

}

@InProceedings{HerlihyLM02

,author={Maurice Herlihy and Victor Luchangco and Mark Moir}

,title="The Repeat Offender Problem: A Mechanism for Supporting Dynamic-Sized,

Lock-Free Data Structures"

,booktitle={Proceedings of 16\textsuperscript{th} International

Symposium on Distributed Computing}

,year=2002

,month="October"

,pages="339-353"

}

@unpublished{Sarma02b

,Author="Dipankar Sarma"

,Title="Some dcache\_rcu benchmark numbers"

}

}

@unpublished{MingmingCao2002IPCRCU

,Author="Mingming Cao"

,Title="[PATCH]updated ipc lock patch"

,month="October"

,year="2002"

,note="Available:

\url{https://lkml.org/lkml/2002/10/24/262}

[Viewed February 15, 2014]"

,annotation={

	Mingming Cao's patch to introduce RCU to SysV IPC.

}

}

@unpublished{LinusTorvalds2003a

,Author="Linus Torvalds"

,Title="Re: {[PATCH]} small fixes in brlock.h"

}

}

@article{MagedMichael04a

,author="Maged M. Michael"

,title="Hazard Pointers: Safe Memory Reclamation for Lock-Free Objects"

,Year="2004"

,Month="June"

,journal="IEEE Transactions on Parallel and Distributed Systems"

,volume="15"

,number="6"

,pages="491-504"

,url="Available:

\url{http://www.research.ibm.com/people/m/michael/ieeetpds-2004.pdf}

[Viewed March 1, 2005]"

,annotation={

	New canonical hazard-pointer citation.

}

}

@phdthesis{PaulEdwardMcKenneyPhD

,author="Paul E. McKenney"

,title="Exploiting Deferred Destruction:

		variations on this theme.

	b.	Limiting update rate.  For example, if updates occur only

		once per hour, then no explicit rate limiting is required,

		unless your system is already badly broken.  The dcache

		subsystem takes this approach -- updates are guarded

		by a global lock, limiting their rate.

		once per hour, then no explicit rate limiting is

		required, unless your system is already badly broken.

		Older versions of the dcache subsystem take this approach,

		guarding updates with a global lock, limiting their rate.

	c.	Trusted update -- if updates can only be done manually by

		superuser or some other trusted user, then it might not

		the machine.

	d.	Use call_rcu_bh() rather than call_rcu(), in order to take

		advantage of call_rcu_bh()'s faster grace periods.

		advantage of call_rcu_bh()'s faster grace periods.  (This

		is only a partial solution, though.)

	e.	Periodically invoke synchronize_rcu(), permitting a limited

		number of updates per grace period.

	The same cautions apply to call_rcu_bh(), call_rcu_sched(),

	call_srcu(), and kfree_rcu().

	Note that although these primitives do take action to avoid memory

	exhaustion when any given CPU has too many callbacks, a determined

	user could still exhaust memory.  This is especially the case

	if a system with a large number of CPUs has been configured to

	offload all of its RCU callbacks onto a single CPU, or if the

	system has relatively little free memory.

9.	All RCU list-traversal primitives, which include

	rcu_dereference(), list_for_each_entry_rcu(), and

	list_for_each_safe_rcu(), must be either within an RCU read-side

To reduce its OS jitter, do any of the following:

1.	Run your workload at a real-time priority, which will allow

	preempting the kworker daemons.

2.	Do any of the following needed to avoid jitter that your

2.	A given workqueue can be made visible in the sysfs filesystem

	by passing the WQ_SYSFS to that workqueue's alloc_workqueue().

	Such a workqueue can be confined to a given subset of the

	CPUs using the /sys/devices/virtual/workqueue/*/cpumask sysfs

	files.	The set of WQ_SYSFS workqueues can be displayed using

	"ls sys/devices/virtual/workqueue".  That said, the workqueues

	maintainer would like to caution people against indiscriminately

	sprinkling WQ_SYSFS across all the workqueues.	The reason for

	caution is that it is easy to add WQ_SYSFS, but because sysfs is

	part of the formal user/kernel API, it can be nearly impossible

	to remove it, even if its addition was a mistake.

3.	Do any of the following needed to avoid jitter that your

	application cannot tolerate:

	a.	Build your kernel with CONFIG_SLUB=y rather than

		CONFIG_SLAB=y, thus avoiding the slab allocator's periodic

	b = p;  /* BUG: Compiler can reorder!!! */

	do_something();

The solution is again ACCESS_ONCE(), which preserves the ordering between

the load from variable 'a' and the store to variable 'b':

The solution is again ACCESS_ONCE() and barrier(), which preserves the

ordering between the load from variable 'a' and the store to variable 'b':

	q = ACCESS_ONCE(a);

	if (q) {

		barrier();

		ACCESS_ONCE(b) = p;

		do_something();

	} else {

		barrier();

		ACCESS_ONCE(b) = p;

		do_something_else();

	}

You could also use barrier() to prevent the compiler from moving

the stores to variable 'b', but barrier() would not prevent the

compiler from proving to itself that a==1 always, so ACCESS_ONCE()

is also needed.

The initial ACCESS_ONCE() is required to prevent the compiler from

proving the value of 'a', and the pair of barrier() invocations are

required to prevent the compiler from pulling the two identical stores

to 'b' out from the legs of the "if" statement.

It is important to note that control dependencies absolutely require a

a conditional.  For example, the following "optimized" version of

the above example breaks ordering:

the above example breaks ordering, which is why the barrier() invocations

are absolutely required if you have identical stores in both legs of

the "if" statement:

	q = ACCESS_ONCE(a);

	ACCESS_ONCE(b) = p;  /* BUG: No ordering vs. load from a!!! */

for example, as follows:

	if (ACCESS_ONCE(a) > 0) {

		barrier();

		ACCESS_ONCE(b) = q / 2;

		do_something();

	} else {

		barrier();

		ACCESS_ONCE(b) = q / 3;

		do_something_else();

	}

	q = ACCESS_ONCE(a);

	if (q % MAX) {

		barrier();

		ACCESS_ONCE(b) = p;

		do_something();

	} else {

		barrier();

		ACCESS_ONCE(b) = p;

		do_something_else();

	}

      use smb_rmb(), smp_wmb(), or, in the case of prior stores and

      later loads, smp_mb().

  (*) If both legs of the "if" statement begin with identical stores

      to the same variable, a barrier() statement is required at the

      beginning of each leg of the "if" statement.

  (*) Control dependencies require at least one run-time conditional

      between the prior load and the subsequent store.  If the compiler

      between the prior load and the subsequent store, and this

      conditional must involve the prior load.  If the compiler

      is able to optimize the conditional away, it will have also

      optimized away the ordering.  Careful use of ACCESS_ONCE() can

      help to preserve the needed conditional.

while perfectly safe in single-threaded code, can be fatal in concurrent

code.  Here are some examples of these sorts of optimizations:

 (*) The compiler is within its rights to reorder loads and stores

     to the same variable, and in some cases, the CPU is within its

     rights to reorder loads to the same variable.  This means that

     the following code:

	a[0] = x;

	a[1] = x;

     Might result in an older value of x stored in a[1] than in a[0].

     Prevent both the compiler and the CPU from doing this as follows:

	a[0] = ACCESS_ONCE(x);

	a[1] = ACCESS_ONCE(x);

     In short, ACCESS_ONCE() provides cache coherence for accesses from

     multiple CPUs to a single variable.

 (*) The compiler is within its rights to merge successive loads from

     the same variable.  Such merging can cause the compiler to "optimize"

     the following code:

     Memory operations issued after the ACQUIRE will be completed after the

     ACQUIRE operation has completed.

     Memory operations issued before the ACQUIRE may be completed after the

     ACQUIRE operation has completed.  An smp_mb__before_spinlock(), combined

     with a following ACQUIRE, orders prior loads against subsequent stores and

     stores and prior stores against subsequent stores.  Note that this is

     weaker than smp_mb()!  The smp_mb__before_spinlock() primitive is free on

     many architectures.

     Memory operations issued before the ACQUIRE may be completed after

     the ACQUIRE operation has completed.  An smp_mb__before_spinlock(),

     combined with a following ACQUIRE, orders prior loads against

     subsequent loads and stores and also orders prior stores against

     subsequent stores.  Note that this is weaker than smp_mb()!  The

     smp_mb__before_spinlock() primitive is free on many architectures.

 (2) RELEASE operation implication:

	ACQUIRE M, STORE *B, STORE *A, RELEASE M

This same reordering can of course occur if the lock's ACQUIRE and RELEASE are

to the same lock variable, but only from the perspective of another CPU not

holding that lock.

In short, a RELEASE followed by an ACQUIRE may -not- be assumed to be a full

memory barrier because it is possible for a preceding RELEASE to pass a

later ACQUIRE from the viewpoint of the CPU, but not from the viewpoint

of the compiler.  Note that deadlocks cannot be introduced by this

interchange because if such a deadlock threatened, the RELEASE would

simply complete.

If it is necessary for a RELEASE-ACQUIRE pair to produce a full barrier, the

ACQUIRE can be followed by an smp_mb__after_unlock_lock() invocation.  This

will produce a full barrier if either (a) the RELEASE and the ACQUIRE are

executed by the same CPU or task, or (b) the RELEASE and ACQUIRE act on the

same variable.  The smp_mb__after_unlock_lock() primitive is free on many

architectures.  Without smp_mb__after_unlock_lock(), the critical sections

corresponding to the RELEASE and the ACQUIRE can cross:

When the ACQUIRE and RELEASE are a lock acquisition and release,

respectively, this same reordering can occur if the lock's ACQUIRE and

RELEASE are to the same lock variable, but only from the perspective of

another CPU not holding that lock.  In short, a ACQUIRE followed by an

RELEASE may -not- be assumed to be a full memory barrier.

Similarly, the reverse case of a RELEASE followed by an ACQUIRE does not

imply a full memory barrier.  If it is necessary for a RELEASE-ACQUIRE

pair to produce a full barrier, the ACQUIRE can be followed by an

smp_mb__after_unlock_lock() invocation.  This will produce a full barrier

if either (a) the RELEASE and the ACQUIRE are executed by the same

CPU or task, or (b) the RELEASE and ACQUIRE act on the same variable.

The smp_mb__after_unlock_lock() primitive is free on many architectures.

Without smp_mb__after_unlock_lock(), the CPU's execution of the critical

sections corresponding to the RELEASE and the ACQUIRE can cross, so that:

	*A = a;

	RELEASE M

	ACQUIRE N, STORE *B, STORE *A, RELEASE M

With smp_mb__after_unlock_lock(), they cannot, so that:

It might appear that this reordering could introduce a deadlock.

However, this cannot happen because if such a deadlock threatened,

the RELEASE would simply complete, thereby avoiding the deadlock.

	Why does this work?

	One key point is that we are only talking about the CPU doing

	the reordering, not the compiler.  If the compiler (or, for

	that matter, the developer) switched the operations, deadlock

	-could- occur.

	But suppose the CPU reordered the operations.  In this case,

	the unlock precedes the lock in the assembly code.  The CPU

	simply elected to try executing the later lock operation first.

	If there is a deadlock, this lock operation will simply spin (or

	try to sleep, but more on that later).	The CPU will eventually

	execute the unlock operation (which preceded the lock operation

	in the assembly code), which will unravel the potential deadlock,

	allowing the lock operation to succeed.

	But what if the lock is a sleeplock?  In that case, the code will

	try to enter the scheduler, where it will eventually encounter

	a memory barrier, which will force the earlier unlock operation

	to complete, again unraveling the deadlock.  There might be

	a sleep-unlock race, but the locking primitive needs to resolve

	such races properly in any case.

With smp_mb__after_unlock_lock(), the two critical sections cannot overlap.

For example, with the following code, the store to *A will always be

seen by other CPUs before the store to *B:

	*A = a;

	RELEASE M

	smp_mb__after_unlock_lock();

	*B = b;

will always occur as either of the following:

The operations will always occur in one of the following orders:

	STORE *A, RELEASE, ACQUIRE, STORE *B

	STORE *A, ACQUIRE, RELEASE, STORE *B

	STORE *A, RELEASE, ACQUIRE, smp_mb__after_unlock_lock(), STORE *B

	STORE *A, ACQUIRE, RELEASE, smp_mb__after_unlock_lock(), STORE *B

	ACQUIRE, STORE *A, RELEASE, smp_mb__after_unlock_lock(), STORE *B

If the RELEASE and ACQUIRE were instead both operating on the same lock

variable, only the first of these two alternatives can occur.

variable, only the first of these alternatives can occur.  In addition,

the more strongly ordered systems may rule out some of the above orders.

But in any case, as noted earlier, the smp_mb__after_unlock_lock()

ensures that the store to *A will always be seen as happening before

the store to *B.

Locks and semaphores may not provide any guarantee of ordering on UP compiled

systems, and so cannot be counted on in such a situation to actually achieve

combination of elements combined or discarded, provided the program's view of

the world remains consistent.  Note that ACCESS_ONCE() is -not- optional

in the above example, as there are architectures where a given CPU might

interchange successive loads to the same location.  On such architectures,

reorder successive loads to the same location.  On such architectures,

ACCESS_ONCE() does whatever is necessary to prevent this, for example, on

Itanium the volatile casts used by ACCESS_ONCE() cause GCC to emit the

special ld.acq and st.rel instructions that prevent such reordering.

	error = fd;

#if 1

	/* Sanity check */

	if (rcu_dereference_raw(fdt->fd[fd]) != NULL) {

	if (rcu_access_pointer(fdt->fd[fd]) != NULL) {

		printk(KERN_WARNING "alloc_fd: slot %d not NULL!\n", fd);

		rcu_assign_pointer(fdt->fd[fd], NULL);

	}