Commit 7bd36014 authored Sep 11, 2013 by John Stultz Committed by Ingo Molnar Sep 12, 2013

timekeeping: Fix HRTICK related deadlock from ntp lock changes



Gerlando Falauto reported that when HRTICK is enabled, it is
possible to trigger system deadlocks. These were hard to
reproduce, as HRTICK has been broken in the past, but seemed
to be connected to the timekeeping_seq lock.

Since seqlock/seqcount's aren't supported w/ lockdep, I added
some extra spinlock based locking and triggered the following
lockdep output:

[   15.849182] ntpd/4062 is trying to acquire lock:
[   15.849765]  (&(&pool->lock)->rlock){..-...}, at: [<ffffffff810aa9b5>] __queue_work+0x145/0x480
[   15.850051]
[   15.850051] but task is already holding lock:
[   15.850051]  (timekeeper_lock){-.-.-.}, at: [<ffffffff810df6df>] do_adjtimex+0x7f/0x100

<snip>

[   15.850051] Chain exists of: &(&pool->lock)->rlock --> &p->pi_lock --> timekeeper_lock
[   15.850051]  Possible unsafe locking scenario:
[   15.850051]
[   15.850051]        CPU0                    CPU1
[   15.850051]        ----                    ----
[   15.850051]   lock(timekeeper_lock);
[   15.850051]                                lock(&p->pi_lock);
[   15.850051] lock(timekeeper_lock);
[   15.850051] lock(&(&pool->lock)->rlock);
[   15.850051]
[   15.850051]  *** DEADLOCK ***

The deadlock was introduced by 06c017fd ("timekeeping:
Hold timekeepering locks in do_adjtimex and hardpps") in 3.10

This patch avoids this deadlock, by moving the call to
schedule_delayed_work() outside of the timekeeper lock
critical section.

Reported-by: Gerlando Falauto <gerlando.falauto@keymile.com>
Tested-by: Lin Ming <minggr@gmail.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: stable <stable@vger.kernel.org> #3.11, 3.10
Link: http://lkml.kernel.org/r/1378943457-27314-1-git-send-email-john.stultz@linaro.org


Signed-off-by: Ingo Molnar <mingo@kernel.org>

parent 5a8e01f8

include/linux/timex.h

+1 −0

Original line number	Diff line number	Diff line
		@@ -141,6 +141,7 @@ extern int do_adjtimex(struct timex *);
		extern void hardpps(const struct timespec , const struct timespec );

		int read_current_timer(unsigned long *timer_val);
		void ntp_notify_cmos_timer(void);

		/* The clock frequency of the i8253/i8254 PIT */
		#define PIT_TICK_RATE 1193182ul

kernel/time/ntp.c

+2 −4

Original line number	Diff line number	Diff line
		@@ -516,13 +516,13 @@ static void sync_cmos_clock(struct work_struct *work)
		schedule_delayed_work(&sync_cmos_work, timespec_to_jiffies(&next));
		}

		static void notify_cmos_timer(void)
		void ntp_notify_cmos_timer(void)
		{
		schedule_delayed_work(&sync_cmos_work, 0);
		}

		#else
		static inline void notify_cmos_timer(void) { }
		void ntp_notify_cmos_timer(void) { }
		#endif


		@@ -687,8 +687,6 @@ int __do_adjtimex(struct timex txc, struct timespec ts, s32 *time_tai)
		if (!(time_status & STA_NANO))
		txc->time.tv_usec /= NSEC_PER_USEC;

		notify_cmos_timer();

		return result;
		}

kernel/time/timekeeping.c

+2 −0

Original line number	Diff line number	Diff line
		@@ -1703,6 +1703,8 @@ int do_adjtimex(struct timex *txc)
		write_seqcount_end(&timekeeper_seq);
		raw_spin_unlock_irqrestore(&timekeeper_lock, flags);

		ntp_notify_cmos_timer();

		return ret;
		}

Admin message