Commit e3e72ab8 authored by Peter Zijlstra's avatar Peter Zijlstra Committed by Ingo Molnar
Browse files

atomic: Implement atomic_read_ctrl() 



Provide atomic_read_ctrl() to mirror READ_ONCE_CTRL(), such that we can
more conveniently use atomics in control dependencies.

Since we can assume atomic_read() implies a READ_ONCE(), we must only
emit an extra smp_read_barrier_depends() in order to upgrade to
READ_ONCE_CTRL() semantics.

Requested-by: default avatarDmitry Vyukov <dvyukov@google.com>
Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: default avatarWill Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Cc: oleg@redhat.com
Link: http://lkml.kernel.org/r/20150918115637.GM3604@twins.programming.kicks-ass.net


Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent 62e8a325

Documentation/memory-barriers.txt

+9 −8
Original line number Diff line number Diff line
@@ -637,7 +637,8 @@ as follows: 
	b = p;  /* BUG: Compiler and CPU can both reorder!!! */



Finally, the READ_ONCE_CTRL() includes an smp_read_barrier_depends()

that DEC Alpha needs in order to respect control depedencies.

that DEC Alpha needs in order to respect control depedencies. Alternatively

use one of atomic{,64}_read_ctrl().



So don't leave out the READ_ONCE_CTRL().
@@ -796,9 +797,9 @@ site: https://www.cl.cam.ac.uk/~pes20/ppcmem/index.html. 


In summary:



  (*) Control dependencies must be headed by READ_ONCE_CTRL().

      Or, as a much less preferable alternative, interpose

      smp_read_barrier_depends() between a READ_ONCE() and the

  (*) Control dependencies must be headed by READ_ONCE_CTRL(),

      atomic{,64}_read_ctrl(). Or, as a much less preferable alternative,

      interpose smp_read_barrier_depends() between a READ_ONCE() and the

      control-dependent write.



  (*) Control dependencies can order prior loads against later stores.
@@ -820,10 +821,10 @@ In summary: 
      and WRITE_ONCE() can help to preserve the needed conditional.



  (*) Control dependencies require that the compiler avoid reordering the

      dependency into nonexistence.  Careful use of READ_ONCE_CTRL()

      or smp_read_barrier_depends() can help to preserve your control

      dependency.  Please see the Compiler Barrier section for more

      information.

      dependency into nonexistence.  Careful use of READ_ONCE_CTRL(),

      atomic{,64}_read_ctrl() or smp_read_barrier_depends() can help to

      preserve your control dependency.  Please see the Compiler Barrier

      section for more information.



  (*) Control dependencies pair normally with other types of barriers.

include/asm-generic/atomic-long.h

+2 −1
Original line number Diff line number Diff line
@@ -35,7 +35,7 @@ typedef atomic_t atomic_long_t; 
#endif



#define ATOMIC_LONG_READ_OP(mo)						\

static inline long atomic_long_read##mo(atomic_long_t *l)		\

static inline long atomic_long_read##mo(const atomic_long_t *l)		\

{									\

	ATOMIC_LONG_PFX(_t) *v = (ATOMIC_LONG_PFX(_t) *)l;		\

									\
@@ -43,6 +43,7 @@ static inline long atomic_long_read##mo(atomic_long_t *l) \ 
}

ATOMIC_LONG_READ_OP()

ATOMIC_LONG_READ_OP(_acquire)

ATOMIC_LONG_READ_OP(_ctrl)



#undef ATOMIC_LONG_READ_OP

include/linux/atomic.h

+18 −0
Original line number Diff line number Diff line
@@ -4,6 +4,15 @@ 
#include <asm/atomic.h>

#include <asm/barrier.h>



#ifndef atomic_read_ctrl

static inline int atomic_read_ctrl(const atomic_t *v)

{

	int val = atomic_read(v);

	smp_read_barrier_depends(); /* Enforce control dependency. */

	return val;

}

#endif



/*

 * Relaxed variants of xchg, cmpxchg and some atomic operations.

 *
@@ -455,6 +464,15 @@ static inline int atomic_dec_if_positive(atomic_t *v) 
#include <asm-generic/atomic64.h>

#endif



#ifndef atomic64_read_ctrl

static inline long long atomic64_read_ctrl(const atomic64_t *v)

{

	long long val = atomic64_read(v);

	smp_read_barrier_depends(); /* Enforce control dependency. */

	return val;

}

#endif



#ifndef atomic64_andnot

static inline void atomic64_andnot(long long i, atomic64_t *v)

{

CRA Git | Maintained and supported by SUSTech CRA and CCSE