timers: Implement optimization for same expiry time in mod_timer() (f00c0afd) · Commits · 戴 / test

The existing optimization for same expiry time in mod_timer() checks whether the timer expiry time is the same as the new requested expiry time. In the old timer wheel implementation this does not take the slack batching into account, neither does the new implementation evaluate whether the new expiry time will requeue the timer to the same bucket. To optimize that, we can calculate the resulting bucket and check if the new expiry time is different from the current expiry time. This calculation happens outside the base lock held region. If the resulting bucket is the same we can avoid taking the base lock and requeueing the timer. If the timer needs to be requeued then we have to check under the base lock whether the base time has changed between the lockless calculation and taking the lock. If it has changed we need to recalculate under the lock. This optimization takes effect for timers which are enqueued into the less granular wheel levels (1 and above). With a simple test case the functionality has been verified: Before After Match: 5.5% 86.6% Requeue: 94.5% 13.4% Recalc: <0.01% In the non optimized case the timer is requeued in 94.5% of the cases. With the index optimization in place the requeue rate drops to 13.4%. The case where the lockless index calculation has to be redone is less than 0.01%. With a real world test case (networking) we observed the following changes: Before After Match: 97.8% 99.7% Requeue: 2.2% 0.3% Recalc: <0.001% That means two percent fewer lock/requeue/unlock operations done in one of the hot path use cases of timers. Signed-off-by:

Anna-Maria Gleixner <anna-maria@linutronix.de> Signed-off-by:

Thomas Gleixner <tglx@linutronix.de> Cc: Arjan van de Ven <arjan@infradead.org> Cc: Chris Mason <clm@fb.com> Cc: Eric Dumazet <edumazet@google.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: George Spelvin <linux@sciencehorizons.net> Cc: Josh Triplett <josh@joshtriplett.org> Cc: Len Brown <lenb@kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rik van Riel <riel@redhat.com> Cc: rt@linutronix.de Link: http://lkml.kernel.org/r/20160704094342.778527749@linutronix.de Signed-off-by:

Ingo Molnar <mingo@kernel.org>

kernel/time/timer.c

+35 −16

Original line number	Original line	Diff line number	Diff line
	@@ -960,29 +960,37 @@ static inline int
	__mod_timer(struct timer_list *timer, unsigned long expires, bool pending_only)		__mod_timer(struct timer_list *timer, unsigned long expires, bool pending_only)
	{		{
	struct timer_base base, new_base;		struct timer_base base, new_base;
	unsigned long flags;		unsigned int idx = UINT_MAX;
			unsigned long clk = 0, flags;
	int ret = 0;		int ret = 0;

	/*		/*
	* TODO: Calculate the array bucket of the timer right here w/o		* This is a common optimization triggered by the networking code - if
	* holding the base lock. This allows to check not only		* the timer is re-modified to have the same timeout or ends up in the
	* timer->expires == expires below, but also whether the timer		* same array bucket then just return:
	* ends up in the same bucket. If we really need to requeue
	* the timer then we check whether base->clk have
	* advanced between here and locking the timer base. If
	* jiffies advanced we have to recalc the array bucket with the
	* lock held.
	*/		*/
			if (timer_pending(timer)) {
			if (timer->expires == expires)
			return 1;
			/*
			* Take the current timer_jiffies of base, but without holding
			* the lock!
			*/
			base = get_timer_base(timer->flags);
			clk = base->clk;

			idx = calc_wheel_index(expires, clk);

	/*		/*
	* This is a common optimization triggered by the		* Retrieve and compare the array index of the pending
	* networking code - if the timer is re-modified		* timer. If it matches set the expiry to the new value so a
	* to be the same thing then just return:		* subsequent call will exit in the expires check above.
	*/		*/
	if (timer_pending(timer)) {		if (idx == timer_get_idx(timer)) {
	if (timer->expires == expires)		timer->expires = expires;
	return 1;		return 1;
	}		}
			}

	timer_stats_timer_set_start_info(timer);		timer_stats_timer_set_start_info(timer);
	BUG_ON(!timer->function);		BUG_ON(!timer->function);
	@@ -1018,7 +1026,18 @@ __mod_timer(struct timer_list *timer, unsigned long expires, bool pending_only)
	}		}

	timer->expires = expires;		timer->expires = expires;
			/*
			* If 'idx' was calculated above and the base time did not advance
			* between calculating 'idx' and taking the lock, only enqueue_timer()
			* and trigger_dyntick_cpu() is required. Otherwise we need to
			* (re)calculate the wheel index via internal_add_timer().
			*/
			if (idx != UINT_MAX && clk == base->clk) {
			enqueue_timer(base, timer, idx);
			trigger_dyntick_cpu(base, timer);
			} else {
	internal_add_timer(base, timer);		internal_add_timer(base, timer);
			}

	out_unlock:		out_unlock:
	spin_unlock_irqrestore(&base->lock, flags);		spin_unlock_irqrestore(&base->lock, flags);

Admin message