Merge branches 'doc.2018.08.30a', 'dynticks.2018.08.30b', 'srcu.2018.08.30b'...

<p>

RCU depends on the scheduler, and the scheduler uses RCU to

protect some of its data structures.

This means the scheduler is forbidden from acquiring

the runqueue locks and the priority-inheritance locks

in the middle of an outermost RCU read-side critical section unless either

(1) it releases them before exiting that same

RCU read-side critical section, or

(2) interrupts are disabled across

that entire RCU read-side critical section.

This same prohibition also applies (recursively!) to any lock that is acquired

while holding any lock to which this prohibition applies.

Adhering to this rule prevents preemptible RCU from invoking

<tt>rcu_read_unlock_special()</tt> while either runqueue or

priority-inheritance locks are held, thus avoiding deadlock.

<p>

Prior to v4.4, it was only necessary to disable preemption across

RCU read-side critical sections that acquired scheduler locks.

In v4.4, expedited grace periods started using IPIs, and these

IPIs could force a <tt>rcu_read_unlock()</tt> to take the slowpath.

Therefore, this expedited-grace-period change required disabling of

interrupts, not just preemption.

<p>

For RCU's part, the preemptible-RCU <tt>rcu_read_unlock()</tt>

implementation must be written carefully to avoid similar deadlocks.

The preemptible-RCU <tt>rcu_read_unlock()</tt>

implementation must therefore be written carefully to avoid deadlocks

involving the scheduler's runqueue and priority-inheritance locks.

In particular, <tt>rcu_read_unlock()</tt> must tolerate an

interrupt where the interrupt handler invokes both

<tt>rcu_read_lock()</tt> and <tt>rcu_read_unlock()</tt>.

interrupt handler's use of RCU.

<p>

This pair of mutual scheduler-RCU requirements came as a

This scheduler-RCU requirement came as a

<a href="https://lwn.net/Articles/453002/">complete surprise</a>.

<p>

<tt>CONFIG_NO_HZ_FULL=y</tt>

<a href="http://www.rdrop.com/users/paulmck/scalability/paper/BareMetal.2015.01.15b.pdf">did come as a surprise [PDF]</a>.

RCU has made good progress towards meeting this requirement, even

for context-switch-have <tt>CONFIG_NO_HZ_FULL=y</tt> workloads,

for context-switch-heavy <tt>CONFIG_NO_HZ_FULL=y</tt> workloads,

but there is room for further improvement.

<p>

In the past, it was forbidden to disable interrupts across an

<tt>rcu_read_unlock()</tt> unless that interrupt-disabled region

of code also included the matching <tt>rcu_read_lock()</tt>.

Violating this restriction could result in deadlocks involving the

scheduler's runqueue and priority-inheritance spinlocks.

This restriction was lifted when interrupt-disabled calls to

<tt>rcu_read_unlock()</tt> started deferring the reporting of

the resulting RCU-preempt quiescent state until the end of that

interrupts-disabled region.

This deferred reporting means that the scheduler's runqueue and

priority-inheritance locks cannot be held while reporting an RCU-preempt

quiescent state, which lifts the earlier restriction, at least from

a deadlock perspective.

Unfortunately, real-time systems using RCU priority boosting may

need this restriction to remain in effect because deferred

quiescent-state reporting also defers deboosting, which in turn

degrades real-time latencies.

<h3><a name="Tracing and RCU">Tracing and RCU</a></h3>

<p>

			Set required age in jiffies for a

			given grace period before RCU starts

			soliciting quiescent-state help from

			rcu_note_context_switch().

			rcu_note_context_switch().  If not specified, the

			kernel will calculate a value based on the most

			recent settings of rcutree.jiffies_till_first_fqs

			and rcutree.jiffies_till_next_fqs.

			This calculated value may be viewed in

			rcutree.jiffies_to_sched_qs.  Any attempt to

			set rcutree.jiffies_to_sched_qs will be

			cheerfully overwritten.

	rcutree.jiffies_till_first_fqs= [KNL]

			Set delay from grace-period initialization to

	rcupdate.rcu_self_test= [KNL]

			Run the RCU early boot self tests

	rcupdate.rcu_self_test_bh= [KNL]

			Run the RCU bh early boot self tests

	rcupdate.rcu_self_test_sched= [KNL]

			Run the RCU sched early boot self tests

	rdinit=		[KNL]

			Format: <full_path>

			Run specified binary instead of /init from the ramdisk,

 * @list:	the RCU-protected list to splice

 * @prev:	points to the last element of the existing list

 * @next:	points to the first element of the existing list

 * @sync:	function to sync: synchronize_rcu(), synchronize_sched(), ...

 * @sync:	synchronize_rcu, synchronize_rcu_expedited, ...

 *

 * The list pointed to by @prev and @next can be RCU-read traversed

 * concurrently with this function.

 *                        designed for stacks.

 * @list:	the RCU-protected list to splice

 * @head:	the place in the existing list to splice the first list into

 * @sync:	function to sync: synchronize_rcu(), synchronize_sched(), ...

 * @sync:	synchronize_rcu, synchronize_rcu_expedited, ...

 */

static inline void list_splice_init_rcu(struct list_head *list,

					struct list_head *head,

 *                             list, designed for queues.

 * @list:	the RCU-protected list to splice

 * @head:	the place in the existing list to splice the first list into

 * @sync:	function to sync: synchronize_rcu(), synchronize_sched(), ...

 * @sync:	synchronize_rcu, synchronize_rcu_expedited, ...

 */

static inline void list_splice_tail_init_rcu(struct list_head *list,

					     struct list_head *head,

 * @type:       the type of the struct this is embedded in.

 * @member:     the name of the list_head within the struct.

 *

 * This primitive may safely run concurrently with the _rcu list-mutation

 * primitives such as list_add_rcu(), but requires some implicit RCU

 * read-side guarding.  One example is running within a special

 * exception-time environment where preemption is disabled and where

 * lockdep cannot be invoked (in which case updaters must use RCU-sched,

 * as in synchronize_sched(), call_rcu_sched(), and friends).  Another

 * example is when items are added to the list, but never deleted.

 * This primitive may safely run concurrently with the _rcu

 * list-mutation primitives such as list_add_rcu(), but requires some

 * implicit RCU read-side guarding.  One example is running within a special

 * exception-time environment where preemption is disabled and where lockdep

 * cannot be invoked.  Another example is when items are added to the list,

 * but never deleted.

 */

#define list_entry_lockless(ptr, type, member) \

	container_of((typeof(ptr))READ_ONCE(ptr), type, member)

 * @head:	the head for your list.

 * @member:	the name of the list_struct within the struct.

 *

 * This primitive may safely run concurrently with the _rcu list-mutation

 * primitives such as list_add_rcu(), but requires some implicit RCU

 * read-side guarding.  One example is running within a special

 * exception-time environment where preemption is disabled and where

 * lockdep cannot be invoked (in which case updaters must use RCU-sched,

 * as in synchronize_sched(), call_rcu_sched(), and friends).  Another

 * example is when items are added to the list, but never deleted.

 * This primitive may safely run concurrently with the _rcu

 * list-mutation primitives such as list_add_rcu(), but requires some

 * implicit RCU read-side guarding.  One example is running within a special

 * exception-time environment where preemption is disabled and where lockdep

 * cannot be invoked.  Another example is when items are added to the list,

 * but never deleted.

 */

#define list_for_each_entry_lockless(pos, head, member) \

	for (pos = list_entry_lockless((head)->next, typeof(*pos), member); \

#define ulong2long(a)		(*(long *)(&(a)))

/* Exported common interfaces */

#ifdef CONFIG_PREEMPT_RCU

void call_rcu(struct rcu_head *head, rcu_callback_t func);

#else /* #ifdef CONFIG_PREEMPT_RCU */

#define	call_rcu	call_rcu_sched

#endif /* #else #ifdef CONFIG_PREEMPT_RCU */

void call_rcu_bh(struct rcu_head *head, rcu_callback_t func);

void call_rcu_sched(struct rcu_head *head, rcu_callback_t func);

void synchronize_sched(void);

void rcu_barrier_tasks(void);

void synchronize_rcu(void);

#ifdef CONFIG_PREEMPT_RCU

void __rcu_read_lock(void);

void __rcu_read_unlock(void);

void synchronize_rcu(void);

/*

 * Defined as a macro as it is a very low level header included from

		preempt_enable();

}

static inline void synchronize_rcu(void)

{

	synchronize_sched();

}

static inline int rcu_preempt_depth(void)

{

	return 0;

/* Internal to kernel */

void rcu_init(void);

extern int rcu_scheduler_active __read_mostly;

void rcu_sched_qs(void);

void rcu_bh_qs(void);

void rcu_check_callbacks(int user);

void rcu_report_dead(unsigned int cpu);

void rcutree_migrate_callbacks(int cpu);

 * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers

 * @a: Code that RCU needs to pay attention to.

 *

 * RCU, RCU-bh, and RCU-sched read-side critical sections are forbidden

 * in the inner idle loop, that is, between the rcu_idle_enter() and

 * the rcu_idle_exit() -- RCU will happily ignore any such read-side

 * critical sections.  However, things like powertop need tracepoints

 * in the inner idle loop.

 * RCU read-side critical sections are forbidden in the inner idle loop,

 * that is, between the rcu_idle_enter() and the rcu_idle_exit() -- RCU

 * will happily ignore any such read-side critical sections.  However,

 * things like powertop need tracepoints in the inner idle loop.

 *

 * This macro provides the way out:  RCU_NONIDLE(do_something_with_RCU())

 * will tell RCU that it needs to pay attention, invoke its argument

		if (READ_ONCE((t)->rcu_tasks_holdout)) \

			WRITE_ONCE((t)->rcu_tasks_holdout, false); \

	} while (0)

#define rcu_note_voluntary_context_switch(t) \

	do { \

		rcu_all_qs(); \

		rcu_tasks_qs(t); \

	} while (0)

#define rcu_note_voluntary_context_switch(t) rcu_tasks_qs(t)

void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);

void synchronize_rcu_tasks(void);

void exit_tasks_rcu_start(void);

void exit_tasks_rcu_finish(void);

#else /* #ifdef CONFIG_TASKS_RCU */

#define rcu_tasks_qs(t)	do { } while (0)

#define rcu_note_voluntary_context_switch(t)		rcu_all_qs()

#define call_rcu_tasks call_rcu_sched

#define synchronize_rcu_tasks synchronize_sched

#define rcu_note_voluntary_context_switch(t) do { } while (0)

#define call_rcu_tasks call_rcu

#define synchronize_rcu_tasks synchronize_rcu

static inline void exit_tasks_rcu_start(void) { }

static inline void exit_tasks_rcu_finish(void) { }

#endif /* #else #ifdef CONFIG_TASKS_RCU */

 * Helper functions for rcu_dereference_check(), rcu_dereference_protected()

 * and rcu_assign_pointer().  Some of these could be folded into their

 * callers, but they are left separate in order to ease introduction of

 * multiple flavors of pointers to match the multiple flavors of RCU

 * (e.g., __rcu_bh, * __rcu_sched, and __srcu), should this make sense in

 * the future.

 * multiple pointers markings to match different RCU implementations

 * (e.g., __srcu), should this make sense in the future.

 */

#ifdef __CHECKER__

/**

 * rcu_read_lock_bh() - mark the beginning of an RCU-bh critical section

 *

 * This is equivalent of rcu_read_lock(), but to be used when updates

 * are being done using call_rcu_bh() or synchronize_rcu_bh(). Since

 * both call_rcu_bh() and synchronize_rcu_bh() consider completion of a

 * softirq handler to be a quiescent state, a process in RCU read-side

 * critical section must be protected by disabling softirqs. Read-side

 * critical sections in interrupt context can use just rcu_read_lock(),

 * though this should at least be commented to avoid confusing people

 * reading the code.

 * This is equivalent of rcu_read_lock(), but also disables softirqs.

 * Note that anything else that disables softirqs can also serve as

 * an RCU read-side critical section.

 *

 * Note that rcu_read_lock_bh() and the matching rcu_read_unlock_bh()

 * must occur in the same context, for example, it is illegal to invoke

/**

 * rcu_read_lock_sched() - mark the beginning of a RCU-sched critical section

 *

 * This is equivalent of rcu_read_lock(), but to be used when updates

 * are being done using call_rcu_sched() or synchronize_rcu_sched().

 * Read-side critical sections can also be introduced by anything that

 * disables preemption, including local_irq_disable() and friends.

 * This is equivalent of rcu_read_lock(), but disables preemption.

 * Read-side critical sections can also be introduced by anything else

 * that disables preemption, including local_irq_disable() and friends.

 *

 * Note that rcu_read_lock_sched() and the matching rcu_read_unlock_sched()

 * must occur in the same context, for example, it is illegal to invoke

#endif /* #else #ifdef CONFIG_ARCH_WEAK_RELEASE_ACQUIRE */

/* Has the specified rcu_head structure been handed to call_rcu()? */

/*

 * rcu_head_init - Initialize rcu_head for rcu_head_after_call_rcu()

 * @rhp: The rcu_head structure to initialize.

 *

 * If you intend to invoke rcu_head_after_call_rcu() to test whether a

 * given rcu_head structure has already been passed to call_rcu(), then

 * you must also invoke this rcu_head_init() function on it just after

 * allocating that structure.  Calls to this function must not race with

 * calls to call_rcu(), rcu_head_after_call_rcu(), or callback invocation.

 */

static inline void rcu_head_init(struct rcu_head *rhp)

{

	rhp->func = (rcu_callback_t)~0L;

}

/*

 * rcu_head_after_call_rcu - Has this rcu_head been passed to call_rcu()?

 * @rhp: The rcu_head structure to test.

 * @func: The function passed to call_rcu() along with @rhp.

 *

 * Returns @true if the @rhp has been passed to call_rcu() with @func,

 * and @false otherwise.  Emits a warning in any other case, including

 * the case where @rhp has already been invoked after a grace period.

 * Calls to this function must not race with callback invocation.  One way

 * to avoid such races is to enclose the call to rcu_head_after_call_rcu()

 * in an RCU read-side critical section that includes a read-side fetch

 * of the pointer to the structure containing @rhp.

 */

static inline bool

rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)

{

	if (READ_ONCE(rhp->func) == f)

		return true;

	WARN_ON_ONCE(READ_ONCE(rhp->func) != (rcu_callback_t)~0L);

	return false;

}

/* Transitional pre-consolidation compatibility definitions. */

static inline void synchronize_rcu_bh(void)

{

	synchronize_rcu();

}

static inline void synchronize_rcu_bh_expedited(void)

{

	synchronize_rcu_expedited();

}

static inline void call_rcu_bh(struct rcu_head *head, rcu_callback_t func)

{

	call_rcu(head, func);

}

static inline void rcu_barrier_bh(void)

{

	rcu_barrier();

}

static inline void synchronize_sched(void)

{

	synchronize_rcu();

}

static inline void synchronize_sched_expedited(void)

{

	synchronize_rcu_expedited();

}

static inline void call_rcu_sched(struct rcu_head *head, rcu_callback_t func)

{

	call_rcu(head, func);

}

static inline void rcu_barrier_sched(void)

{

	rcu_barrier();

}

static inline unsigned long get_state_synchronize_sched(void)

{

	return get_state_synchronize_rcu();

}

static inline void cond_synchronize_sched(unsigned long oldstate)

{

	cond_synchronize_rcu(oldstate);

}

#endif /* __LINUX_RCUPDATE_H */

/**

 * synchronize_rcu_mult - Wait concurrently for multiple grace periods

 * @...: List of call_rcu() functions for the flavors to wait on.

 * @...: List of call_rcu() functions for different grace periods to wait on

 *

 * This macro waits concurrently for multiple flavors of RCU grace periods.

 * For example, synchronize_rcu_mult(call_rcu, call_rcu_bh) would wait

 * on concurrent RCU and RCU-bh grace periods.  Waiting on a give SRCU

 * This macro waits concurrently for multiple types of RCU grace periods.

 * For example, synchronize_rcu_mult(call_rcu, call_rcu_tasks) would wait

 * on concurrent RCU and RCU-tasks grace periods.  Waiting on a give SRCU

 * domain requires you to write a wrapper function for that SRCU domain's

 * call_srcu() function, supplying the corresponding srcu_struct.

 *

 * If Tiny RCU, tell _wait_rcu_gp() not to bother waiting for RCU

 * or RCU-bh, given that anywhere synchronize_rcu_mult() can be called

 * is automatically a grace period.

 * If Tiny RCU, tell _wait_rcu_gp() does not bother waiting for RCU,

 * given that anywhere synchronize_rcu_mult() can be called is automatically

 * a grace period.

 */

#define synchronize_rcu_mult(...) \

	_wait_rcu_gp(IS_ENABLED(CONFIG_TINY_RCU), __VA_ARGS__)