Merge branch 'core/rcu-for-linus' of...

not to return until all ongoing NMI handlers exit.  It is therefore safe

to free up the handler's data as soon as synchronize_sched() returns.

Important note: for this to work, the architecture in question must

invoke irq_enter() and irq_exit() on NMI entry and exit, respectively.

Answer to Quick Quiz

structured data, such as the matrices used in scientific programs, and

is thus inapplicable to most data structures in operating-system kernels.

In 1992, Henry (now Alexia) Massalin completed a dissertation advising

parallel programmers to defer processing when feasible to simplify

synchronization.  RCU makes extremely heavy use of this advice.

In 1993, Jacobson [Jacobson93] verbally described what is perhaps the

simplest deferred-free technique: simply waiting a fixed amount of time

before freeing blocks awaiting deferred free.  Jacobson did not describe

Robert Olsson described an RCU-protected trie-hash combination

[RobertOlsson2006a].

2007 saw the journal version of the award-winning RCU paper from 2006

[ThomasEHart2007a], as well as a paper demonstrating use of Promela

and Spin to mechanically verify an optimization to Oleg Nesterov's

QRCU [PaulEMcKenney2007QRCUspin], a design document describing

preemptible RCU [PaulEMcKenney2007PreemptibleRCU], and the three-part

LWN "What is RCU?" series [PaulEMcKenney2007WhatIsRCUFundamentally,

PaulEMcKenney2008WhatIsRCUUsage, and PaulEMcKenney2008WhatIsRCUAPI].

Bibtex Entries

,Year="1991"

}

@phdthesis{HMassalinPhD

,author="H. Massalin"

,title="Synthesis: An Efficient Implementation of Fundamental Operating

System Services"

,school="Columbia University"

,address="New York, NY"

,year="1992"

,annotation="

	Mondo optimizing compiler.

	Wait-free stuff.

	Good advice: defer work to avoid synchronization.

"

}

@unpublished{Jacobson93

,author="Van Jacobson"

,title="Avoid Read-Side Locking Via Delayed Free"

"

}

@unpublished{PaulEMcKenney2007PreemptibleRCU

,Author="Paul E. McKenney"

,Title="The design of preemptible read-copy-update"

,month="October"

,day="8"

,year="2007"

,note="Available:

\url{http://lwn.net/Articles/253651/}

[Viewed October 25, 2007]"

,annotation="

	LWN article describing the design of preemptible RCU.

"

}

########################################################################

#

#	"What is RCU?" LWN series.

#

@unpublished{PaulEMcKenney2007WhatIsRCUFundamentally

,Author="Paul E. McKenney and Jonathan Walpole"

,Title="What is {RCU}, Fundamentally?"

,month="December"

,day="17"

,year="2007"

,note="Available:

\url{http://lwn.net/Articles/262464/}

[Viewed December 27, 2007]"

,annotation="

	Lays out the three basic components of RCU: (1) publish-subscribe,

	(2) wait for pre-existing readers to complete, and (2) maintain

	multiple versions.

"

}

@unpublished{PaulEMcKenney2008WhatIsRCUUsage

,Author="Paul E. McKenney"

,Title="What is {RCU}? Part 2: Usage"

,month="January"

,day="4"

,year="2008"

,note="Available:

\url{http://lwn.net/Articles/263130/}

[Viewed January 4, 2008]"

,annotation="

	Lays out six uses of RCU:

	1. RCU is a Reader-Writer Lock Replacement

	2. RCU is a Restricted Reference-Counting Mechanism

	3. RCU is a Bulk Reference-Counting Mechanism

	4. RCU is a Poor Man's Garbage Collector

	5. RCU is a Way of Providing Existence Guarantees

	6. RCU is a Way of Waiting for Things to Finish 

"

}

@unpublished{PaulEMcKenney2008WhatIsRCUAPI

,Author="Paul E. McKenney"

,Title="{RCU} part 3: the {RCU} {API}"

,month="January"

,day="17"

,year="2008"

,note="Available:

\url{http://lwn.net/Articles/264090/}

[Viewed January 10, 2008]"

,annotation="

	Gives an overview of the Linux-kernel RCU API and a brief annotated RCU

	bibliography.

"

}

@article{DinakarGuniguntala2008IBMSysJ

,author="D. Guniguntala and P. E. McKenney and J. Triplett and J. Walpole"

,title="The read-copy-update mechanism for supporting real-time applications on shared-memory multiprocessor systems with {Linux}"

,Year="2008"

,Month="April"

,journal="IBM Systems Journal"

,volume="47"

,number="2"

,pages="@@-@@"

,annotation="

	RCU, realtime RCU, sleepable RCU, performance.

"

}

	detailed performance measurements show that RCU is nonetheless

	the right tool for the job.

	The other exception would be where performance is not an issue,

	and RCU provides a simpler implementation.  An example of this

	situation is the dynamic NMI code in the Linux 2.6 kernel,

	at least on architectures where NMIs are rare.

	Another exception is where performance is not an issue, and RCU

	provides a simpler implementation.  An example of this situation

	is the dynamic NMI code in the Linux 2.6 kernel, at least on

	architectures where NMIs are rare.

	Yet another exception is where the low real-time latency of RCU's

	read-side primitives is critically important.

1.	Does the update code have proper mutual exclusion?

2.	Do the RCU read-side critical sections make proper use of

	rcu_read_lock() and friends?  These primitives are needed

	to suppress preemption (or bottom halves, in the case of

	rcu_read_lock_bh()) in the read-side critical sections,

	and are also an excellent aid to readability.

	to prevent grace periods from ending prematurely, which

	could result in data being unceremoniously freed out from

	under your read-side code, which can greatly increase the

	actuarial risk of your kernel.

	As a rough rule of thumb, any dereference of an RCU-protected

	pointer must be covered by rcu_read_lock() or rcu_read_lock_bh()

	be running while updates are in progress.  There are a number

	of ways to handle this concurrency, depending on the situation:

	a.	Make updates appear atomic to readers.  For example,

	a.	Use the RCU variants of the list and hlist update

		primitives to add, remove, and replace elements on an

		RCU-protected list.  Alternatively, use the RCU-protected

		trees that have been added to the Linux kernel.

		This is almost always the best approach.

	b.	Proceed as in (a) above, but also maintain per-element

		locks (that are acquired by both readers and writers)

		that guard per-element state.  Of course, fields that

		the readers refrain from accessing can be guarded by the

		update-side lock.

		This works quite well, also.

	c.	Make updates appear atomic to readers.  For example,

		pointer updates to properly aligned fields will appear

		atomic, as will individual atomic primitives.  Operations

		performed under a lock and sequences of multiple atomic

		primitives will -not- appear to be atomic.

		This is almost always the best approach.

		This can work, but is starting to get a bit tricky.

	b.	Carefully order the updates and the reads so that

	d.	Carefully order the updates and the reads so that

		readers see valid data at all phases of the update.

		This is often more difficult than it sounds, especially

		given modern CPUs' tendency to reorder memory references.

		when publicizing a pointer to a structure that can

		be traversed by an RCU read-side critical section.

5.	If call_rcu(), or a related primitive such as call_rcu_bh(),

	is used, the callback function must be written to be called

	from softirq context.  In particular, it cannot block.

5.	If call_rcu(), or a related primitive such as call_rcu_bh() or

	call_rcu_sched(), is used, the callback function must be

	written to be called from softirq context.  In particular,

	it cannot block.

6.	Since synchronize_rcu() can block, it cannot be called from

	any sort of irq context.

	any sort of irq context.  Ditto for synchronize_sched() and

	synchronize_srcu().

7.	If the updater uses call_rcu(), then the corresponding readers

	must use rcu_read_lock() and rcu_read_unlock().  If the updater

	uses call_rcu_bh(), then the corresponding readers must use

	rcu_read_lock_bh() and rcu_read_unlock_bh().  Mixing things up

	will result in confusion and broken kernels.

	rcu_read_lock_bh() and rcu_read_unlock_bh().  If the updater

	uses call_rcu_sched(), then the corresponding readers must

	disable preemption.  Mixing things up will result in confusion

	and broken kernels.

	One exception to this rule: rcu_read_lock() and rcu_read_unlock()

	may be substituted for rcu_read_lock_bh() and rcu_read_unlock_bh()

	such cases is a must, of course!  And the jury is still out on

	whether the increased speed is worth it.

8.	Although synchronize_rcu() is a bit slower than is call_rcu(),

	it usually results in simpler code.  So, unless update

	performance is critically important or the updaters cannot block,

8.	Although synchronize_rcu() is slower than is call_rcu(), it

	usually results in simpler code.  So, unless update performance

	is critically important or the updaters cannot block,

	synchronize_rcu() should be used in preference to call_rcu().

	An especially important property of the synchronize_rcu()

		number of updates per grace period.

9.	All RCU list-traversal primitives, which include

	list_for_each_rcu(), list_for_each_entry_rcu(),

	rcu_dereference(), list_for_each_rcu(), list_for_each_entry_rcu(),

	list_for_each_continue_rcu(), and list_for_each_safe_rcu(),

	must be within an RCU read-side critical section.  RCU

	must be either within an RCU read-side critical section or

	must be protected by appropriate update-side locks.  RCU

	read-side critical sections are delimited by rcu_read_lock()

	and rcu_read_unlock(), or by similar primitives such as

	rcu_read_lock_bh() and rcu_read_unlock_bh().

	Use of the _rcu() list-traversal primitives outside of an

	RCU read-side critical section causes no harm other than

	a slight performance degradation on Alpha CPUs.  It can

	also be quite helpful in reducing code bloat when common

	code is shared between readers and updaters.

	The reason that it is permissible to use RCU list-traversal

	primitives when the update-side lock is held is that doing so

	can be quite helpful in reducing code bloat when common code is

	shared between readers and updaters.

10.	Conversely, if you are in an RCU read-side critical section,

	you -must- use the "_rcu()" variants of the list macros.

	Failing to do so will break Alpha and confuse people reading

	your code.

	and you don't hold the appropriate update-side lock, you -must-

	use the "_rcu()" variants of the list macros.  Failing to do so

	will break Alpha and confuse people reading your code.

11.	Note that synchronize_rcu() -only- guarantees to wait until

	all currently executing rcu_read_lock()-protected RCU read-side

	must use whatever locking or other synchronization is required

	to safely access and/or modify that data structure.

	RCU callbacks are -usually- executed on the same CPU that executed

	the corresponding call_rcu(), call_rcu_bh(), or call_rcu_sched(),

	but are by -no- means guaranteed to be.  For example, if a given

	CPU goes offline while having an RCU callback pending, then that

	RCU callback will execute on some surviving CPU.  (If this was

	not the case, a self-spawning RCU callback would prevent the

	victim CPU from ever going offline.)

14.	SRCU (srcu_read_lock(), srcu_read_unlock(), and synchronize_srcu())

	may only be invoked from process context.  Unlike other forms of

	RCU, it -is- permissible to block in an SRCU read-side critical

command (perhaps grepping for "torture").  The test is started

when the module is loaded, and stops when the module is unloaded.

However, actually setting this config option to "y" results in the system

running the test immediately upon boot, and ending only when the system

is taken down.  Normally, one will instead want to build the system

with CONFIG_RCU_TORTURE_TEST=m and to use modprobe and rmmod to control

the test, perhaps using a script similar to the one shown at the end of

this document.  Note that you will need CONFIG_MODULE_UNLOAD in order

to be able to end the test.

CONFIG_RCU_TORTURE_TEST_RUNNABLE

It is also possible to specify CONFIG_RCU_TORTURE_TEST=y, which will

result in the tests being loaded into the base kernel.  In this case,

the CONFIG_RCU_TORTURE_TEST_RUNNABLE config option is used to specify

whether the RCU torture tests are to be started immediately during

boot or whether the /proc/sys/kernel/rcutorture_runnable file is used

to enable them.  This /proc file can be used to repeatedly pause and

restart the tests, regardless of the initial state specified by the

CONFIG_RCU_TORTURE_TEST_RUNNABLE config option.

You will normally -not- want to start the RCU torture tests during boot

(and thus the default is CONFIG_RCU_TORTURE_TEST_RUNNABLE=n), but doing

this can sometimes be useful in finding boot-time bugs.

MODULE PARAMETERS

This module has the following parameters:

nreaders	This is the number of RCU reading threads supported.

		The default is twice the number of CPUs.  Why twice?

		To properly exercise RCU implementations with preemptible

		read-side critical sections.

irqreaders	Says to invoke RCU readers from irq level.  This is currently

		done via timers.  Defaults to "1" for variants of RCU that

		permit this.  (Or, more accurately, variants of RCU that do

		-not- permit this know to ignore this variable.)

nfakewriters	This is the number of RCU fake writer threads to run.  Fake

		writer threads repeatedly use the synchronous "wait for

		to trigger special cases caused by multiple writers, such as

		the synchronize_srcu() early return optimization.

nreaders	This is the number of RCU reading threads supported.

		The default is twice the number of CPUs.  Why twice?

		To properly exercise RCU implementations with preemptible

		read-side critical sections.

shuffle_interval

		The number of seconds to keep the test threads affinitied

		to a particular subset of the CPUs, defaults to 3 seconds.

		Used in conjunction with test_no_idle_hz.

stat_interval	The number of seconds between output of torture

		statistics (via printk()).  Regardless of the interval,

		statistics are printed when the module is unloaded.

		be printed -only- when the module is unloaded, and this

		is the default.

shuffle_interval

		The number of seconds to keep the test threads affinitied

		to a particular subset of the CPUs, defaults to 5 seconds.

		Used in conjunction with test_no_idle_hz.

stutter		The length of time to run the test before pausing for this

		same period of time.  Defaults to "stutter=5", so as

		to run and pause for (roughly) five-second intervals.

		Specifying "stutter=0" causes the test to run continuously

		without pausing, which is the old default behavior.

test_no_idle_hz	Whether or not to test the ability of RCU to operate in

		a kernel that disables the scheduling-clock interrupt to

Please note that the "What is RCU?" LWN series is an excellent place

to start learning about RCU:

1.	What is RCU, Fundamentally?  http://lwn.net/Articles/262464/

2.	What is RCU? Part 2: Usage   http://lwn.net/Articles/263130/

3.	RCU part 3: the RCU API      http://lwn.net/Articles/264090/

What is RCU?

RCU is a synchronization mechanism that was added to the Linux kernel

APIs, since there does not appear to be a way to categorize them

in docbook.  Here is the list, by category.

Markers for RCU read-side critical sections:

	rcu_read_lock

	rcu_read_unlock

	rcu_read_lock_bh

	rcu_read_unlock_bh

	srcu_read_lock

	srcu_read_unlock

RCU pointer/list traversal:

	rcu_dereference

	list_for_each_entry_rcu

	hlist_for_each_entry_rcu

	list_for_each_rcu		(to be deprecated in favor of

					 list_for_each_entry_rcu)

	list_for_each_entry_rcu

	list_for_each_continue_rcu	(to be deprecated in favor of new

					 list_for_each_entry_continue_rcu)

	hlist_for_each_entry_rcu

RCU pointer update:

RCU pointer/list update:

	rcu_assign_pointer

	list_add_rcu

	list_del_rcu

	list_replace_rcu

	hlist_del_rcu

	hlist_add_after_rcu

	hlist_add_before_rcu

	hlist_add_head_rcu

	hlist_replace_rcu

	list_splice_init_rcu()

RCU grace period:

RCU:	Critical sections	Grace period		Barrier

	synchronize_net

	synchronize_sched

	synchronize_rcu

	synchronize_srcu

	rcu_read_lock		synchronize_net		rcu_barrier

	rcu_read_unlock		synchronize_rcu

				call_rcu

	call_rcu_bh

bh:	Critical sections	Grace period		Barrier

	rcu_read_lock_bh	call_rcu_bh		rcu_barrier_bh

	rcu_read_unlock_bh

sched:	Critical sections	Grace period		Barrier

	[preempt_disable]	synchronize_sched	rcu_barrier_sched

	[and friends]		call_rcu_sched

SRCU:	Critical sections	Grace period		Barrier

	srcu_read_lock		synchronize_srcu	N/A

	srcu_read_unlock

See the comment headers in the source code (or the docbook generated

from them) for more information.