Commit 1a149d7d authored by Steven Rostedt (VMware)'s avatar Steven Rostedt (VMware)
Browse files

ring-buffer: Rewrite trace_recursive_(un)lock() to be simpler 



The current method to prevent the ring buffer from entering into a recursize
loop is to use a bitmask and set the bit that maps to the current context
(normal, softirq, irq or NMI), and if that bit was already set, it is
considered a recursive loop.

New code is being added that may require the ring buffer to be entered a
second time in the current context. The recursive locking prevents that from
happening. Instead of mapping a bitmask to the current context, just allow 4
levels of nesting in the ring buffer. This matches the 4 context levels that
it can already nest. It is highly unlikely to have more than two levels,
thus it should be fine when we add the second entry into the ring buffer. If
that proves to be a problem, we can always up the number to 8.

An added benefit is that reading preempt_count() to get the current level
adds a very slight but noticeable overhead. This removes that need.

Signed-off-by: default avatarSteven Rostedt (VMware) <rostedt@goodmis.org>
parent 12ecef0c

kernel/trace/ring_buffer.c

+17 −47
Original line number Diff line number Diff line
@@ -2538,61 +2538,29 @@ rb_wakeups(struct ring_buffer *buffer, struct ring_buffer_per_cpu *cpu_buffer) 
 * The lock and unlock are done within a preempt disable section.

 * The current_context per_cpu variable can only be modified

 * by the current task between lock and unlock. But it can

 * be modified more than once via an interrupt. To pass this

 * information from the lock to the unlock without having to

 * access the 'in_interrupt()' functions again (which do show

 * a bit of overhead in something as critical as function tracing,

 * we use a bitmask trick.

 * be modified more than once via an interrupt. There are four

 * different contexts that we need to consider.

 *

 *  bit 0 =  NMI context

 *  bit 1 =  IRQ context

 *  bit 2 =  SoftIRQ context

 *  bit 3 =  normal context.

 *  Normal context.

 *  SoftIRQ context

 *  IRQ context

 *  NMI context

 *

 * This works because this is the order of contexts that can

 * preempt other contexts. A SoftIRQ never preempts an IRQ

 * context.

 *

 * When the context is determined, the corresponding bit is

 * checked and set (if it was set, then a recursion of that context

 * happened).

 *

 * On unlock, we need to clear this bit. To do so, just subtract

 * 1 from the current_context and AND it to itself.

 *

 * (binary)

 *  101 - 1 = 100

 *  101 & 100 = 100 (clearing bit zero)

 *

 *  1010 - 1 = 1001

 *  1010 & 1001 = 1000 (clearing bit 1)

 *

 * The least significant bit can be cleared this way, and it

 * just so happens that it is the same bit corresponding to

 * the current context.

 * If for some reason the ring buffer starts to recurse, we

 * only allow that to happen at most 4 times (one for each

 * context). If it happens 5 times, then we consider this a

 * recusive loop and do not let it go further.

 */



static __always_inline int

trace_recursive_lock(struct ring_buffer_per_cpu *cpu_buffer)

{

	unsigned int val = cpu_buffer->current_context;

	int bit;



	if (in_interrupt()) {

		if (in_nmi())

			bit = RB_CTX_NMI;

		else if (in_irq())

			bit = RB_CTX_IRQ;

		else

			bit = RB_CTX_SOFTIRQ;

	} else

		bit = RB_CTX_NORMAL;



	if (unlikely(val & (1 << bit)))

	if (cpu_buffer->current_context >= 4)

		return 1;



	val |= (1 << bit);

	cpu_buffer->current_context = val;

	cpu_buffer->current_context++;

	/* Interrupts must see this update */

	barrier();



	return 0;

}
@@ -2600,7 +2568,9 @@ trace_recursive_lock(struct ring_buffer_per_cpu *cpu_buffer) 
static __always_inline void

trace_recursive_unlock(struct ring_buffer_per_cpu *cpu_buffer)

{

	cpu_buffer->current_context &= cpu_buffer->current_context - 1;

	/* Don't let the dec leak out */

	barrier();

	cpu_buffer->current_context--;

}



/**

CRA Git | Maintained and supported by SUSTech CRA and CCSE