Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

   <hardirq-safe>   ->  <hardirq-unsafe>

   <softirq-safe>   ->  <softirq-unsafe>

The first rule comes from the fact the a hardirq-safe lock could be

The first rule comes from the fact that a hardirq-safe lock could be

taken by a hardirq context, interrupting a hardirq-unsafe lock - and

thus could result in a lock inversion deadlock. Likewise, a softirq-safe

lock could be taken by an softirq context, interrupting a softirq-unsafe

when the chain is validated for the first time, is then put into a hash

table, which hash-table can be checked in a lockfree manner. If the

locking chain occurs again later on, the hash table tells us that we

dont have to validate the chain again.

don't have to validate the chain again.

Troubleshooting:

----------------

By: David Howells <dhowells@redhat.com>

    Paul E. McKenney <paulmck@linux.vnet.ibm.com>

    Will Deacon <will.deacon@arm.com>

    Peter Zijlstra <peterz@infradead.org>

Contents:

==========

DISCLAIMER

==========

This document is not a specification; it is intentionally (for the sake of

brevity) and unintentionally (due to being human) incomplete. This document is

meant as a guide to using the various memory barriers provided by Linux, but

in case of any doubt (and there are many) please ask.

To repeat, this document is not a specification of what Linux expects from

hardware.

The purpose of this document is twofold:

 (1) to specify the minimum functionality that one can rely on for any

     particular barrier, and

 (2) to provide a guide as to how to use the barriers that are available.

Note that an architecture can provide more than the minimum requirement

for any particular barrier, but if the architecure provides less than

that, that architecture is incorrect.

Note also that it is possible that a barrier may be a no-op for an

architecture because the way that arch works renders an explicit barrier

unnecessary in that case.

========

CONTENTS

========

 (*) Abstract memory access model.

 (*) Implicit kernel memory barriers.

     - Locking functions.

     - Lock acquisition functions.

     - Interrupt disabling functions.

     - Sleep and wake-up functions.

     - Miscellaneous functions.

 (*) Inter-CPU locking barrier effects.

 (*) Inter-CPU acquiring barrier effects.

     - Locks vs memory accesses.

     - Locks vs I/O accesses.

     - Acquires vs memory accesses.

     - Acquires vs I/O accesses.

 (*) Where are memory barriers needed?

 (*) The things CPUs get up to.

     - And then there's the Alpha.

     - Virtual Machine Guests.

 (*) Example uses.

	CPU 1		CPU 2

	===============	===============

	{ A == 1, B == 2, C = 3, P == &A, Q == &C }

	{ A == 1, B == 2, C == 3, P == &A, Q == &C }

	B = 4;		Q = P;

	P = &B		D = *Q;

     This acts as a one-way permeable barrier.  It guarantees that all memory

     operations after the ACQUIRE operation will appear to happen after the

     ACQUIRE operation with respect to the other components of the system.

     ACQUIRE operations include LOCK operations and smp_load_acquire()

     operations.

     ACQUIRE operations include LOCK operations and both smp_load_acquire()

     and smp_cond_acquire() operations. The later builds the necessary ACQUIRE

     semantics from relying on a control dependency and smp_rmb().

     Memory operations that occur before an ACQUIRE operation may appear to

     happen after it completes.

     This means that ACQUIRE acts as a minimal "acquire" operation and

     RELEASE acts as a minimal "release" operation.

A subset of the atomic operations described in atomic_ops.txt have ACQUIRE

and RELEASE variants in addition to fully-ordered and relaxed (no barrier

semantics) definitions.  For compound atomics performing both a load and a

store, ACQUIRE semantics apply only to the load and RELEASE semantics apply

only to the store portion of the operation.

Memory barriers are only required where there's a possibility of interaction

between two CPUs or between a CPU and a device.  If it can be guaranteed that

	CPU 1		      CPU 2

	===============	      ===============

	{ A == 1, B == 2, C = 3, P == &A, Q == &C }

	{ A == 1, B == 2, C == 3, P == &A, Q == &C }

	B = 4;

	<write barrier>

	WRITE_ONCE(P, &B)

	CPU 1		      CPU 2

	===============	      ===============

	{ A == 1, B == 2, C = 3, P == &A, Q == &C }

	{ A == 1, B == 2, C == 3, P == &A, Q == &C }

	B = 4;

	<write barrier>

	WRITE_ONCE(P, &B);

      the same variable, then those stores must be ordered, either by

      preceding both of them with smp_mb() or by using smp_store_release()

      to carry out the stores.  Please note that it is -not- sufficient

      to use barrier() at beginning of each leg of the "if" statement,

      as optimizing compilers do not necessarily respect barrier()

      in this case.

      to use barrier() at beginning of each leg of the "if" statement

      because, as shown by the example above, optimizing compilers can

      destroy the control dependency while respecting the letter of the

      barrier() law.

  (*) Control dependencies require at least one run-time conditional

      between the prior load and the subsequent store, and this

 (*) lockless_dereference();

     This can be thought of as a pointer-fetch wrapper around the

     smp_read_barrier_depends() data-dependency barrier.

ordered I/O regions to be partially ordered.  Its effects may go beyond the

CPU->Hardware interface and actually affect the hardware at some level.

See the subsection "Locks vs I/O accesses" for more information.

See the subsection "Acquires vs I/O accesses" for more information.

===============================

of arch specific code.

ACQUIRING FUNCTIONS

-------------------

LOCK ACQUISITION FUNCTIONS

--------------------------

The Linux kernel has a number of locking constructs:

	event_indicated = 1;

	wake_up_process(event_daemon);

A write memory barrier is implied by wake_up() and co. if and only if they wake

something up.  The barrier occurs before the task state is cleared, and so sits

between the STORE to indicate the event and the STORE to set TASK_RUNNING:

A write memory barrier is implied by wake_up() and co.  if and only if they

wake something up.  The barrier occurs before the task state is cleared, and so

sits between the STORE to indicate the event and the STORE to set TASK_RUNNING:

	CPU 1				CPU 2

	===============================	===============================

	clear_bit_unlock();

	__clear_bit_unlock();

These implement ACQUIRE-class and RELEASE-class operations. These should be used in

preference to other operations when implementing locking primitives, because

their implementations can be optimised on many architectures.

These implement ACQUIRE-class and RELEASE-class operations.  These should be

used in preference to other operations when implementing locking primitives,

because their implementations can be optimised on many architectures.

[!] Note that special memory barrier primitives are available for these

situations because on some CPUs the atomic instructions used imply full memory

 (*) inX(), outX():

     These are intended to talk to I/O space rather than memory space, but

     that's primarily a CPU-specific concept. The i386 and x86_64 processors do

     indeed have special I/O space access cycles and instructions, but many

     that's primarily a CPU-specific concept.  The i386 and x86_64 processors

     do indeed have special I/O space access cycles and instructions, but many

     CPUs don't have such a concept.

     The PCI bus, amongst others, defines an I/O space concept which - on such

See the subsection on "Cache Coherency" above.

VIRTUAL MACHINE GUESTS

-------------------

----------------------

Guests running within virtual machines might be affected by SMP effects even if

the guest itself is compiled without SMP support.  This is an artifact of

in particular, they do not control MMIO effects: to control

MMIO effects, use mandatory barriers.

============

EXAMPLE USES

============

/**

 * atomic_fetch_or - perform *p |= mask and return old value of *p

 * @p: pointer to atomic_t

 * @mask: mask to OR on the atomic_t

 * @p: pointer to atomic_t

 */

#ifndef atomic_fetch_or

static inline int atomic_fetch_or(atomic_t *p, int mask)

static inline int atomic_fetch_or(int mask, atomic_t *p)

{

	int old, val = atomic_read(p);

 * yet. Otherwise we look it up. We cache the result in the lock object

 * itself, so actual lookup of the hash should be once per lock object.

 */

static inline struct lock_class *

static struct lock_class *

register_lock_class(struct lockdep_map *lock, unsigned int subclass, int force)

{

	struct lockdep_subclass_key *key;

	long n_lock_acquired;

};

#if defined(MODULE)

#define LOCKTORTURE_RUNNABLE_INIT 1

#else

#define LOCKTORTURE_RUNNABLE_INIT 0

#endif

int torture_runnable = LOCKTORTURE_RUNNABLE_INIT;

int torture_runnable = IS_ENABLED(MODULE);

module_param(torture_runnable, int, 0444);

MODULE_PARM_DESC(torture_runnable, "Start locktorture at module init");

	if (!rt_task(current)) {

		/*

		 * (1) Boost priority once every ~50k operations. When the

		 * Boost priority once every ~50k operations. When the

		 * task tries to take the lock, the rtmutex it will account

		 * for the new priority, and do any corresponding pi-dance.

		 */

		if (!(torture_random(trsp) %

		if (trsp && !(torture_random(trsp) %

			      (cxt.nrealwriters_stress * factor))) {

			policy = SCHED_FIFO;

			param.sched_priority = MAX_RT_PRIO - 1;

	if (torture_cleanup_begin())

		return;

	/*

	 * Indicates early cleanup, meaning that the test has not run,

	 * such as when passing bogus args when loading the module. As

	 * such, only perform the underlying torture-specific cleanups,

	 * and avoid anything related to locktorture.

	 */

	if (!cxt.lwsa)

		goto end;

	if (writer_tasks) {

		for (i = 0; i < cxt.nrealwriters_stress; i++)

			torture_stop_kthread(lock_torture_writer,

	else

		lock_torture_print_module_parms(cxt.cur_ops,

						"End of test: SUCCESS");

end:

	torture_cleanup_end();

}

			VERBOSE_TOROUT_STRING("cxt.lrsa: Out of memory");

			firsterr = -ENOMEM;

			kfree(cxt.lwsa);

			cxt.lwsa = NULL;

			goto unwind;

		}

			cxt.lrsa[i].n_lock_acquired = 0;

		}

	}

	lock_torture_print_module_parms(cxt.cur_ops, "Start of test");

	/* Prepare torture context. */