Commit df8d9eea authored by Rafael J. Wysocki's avatar Rafael J. Wysocki
Browse files

cpuidle: Run tick_broadcast_exit() with disabled interrupts 



Commit 335f4919 (sched/idle: Use explicit broadcast oneshot
control function) replaced clockevents_notify() invocations in
cpuidle_idle_call() with direct calls to tick_broadcast_enter()
and tick_broadcast_exit(), but it overlooked the fact that
interrupts were already enabled before calling the latter which
led to functional breakage on systems using idle states with the
CPUIDLE_FLAG_TIMER_STOP flag set.

Fix that by moving the invocations of tick_broadcast_enter()
and tick_broadcast_exit() down into cpuidle_enter_state() where
interrupts are still disabled when tick_broadcast_exit() is
called.  Also ensure that interrupts will be disabled before
running tick_broadcast_exit() even if they have been enabled by
the idle state's ->enter callback.  Trigger a WARN_ON_ONCE() in
that case, as we generally don't want that to happen for states
with CPUIDLE_FLAG_TIMER_STOP set.

Fixes: 335f4919 (sched/idle: Use explicit broadcast oneshot control function)
Reported-and-tested-by: default avatarLinus Walleij <linus.walleij@linaro.org>
Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: default avatarDaniel Lezcano <daniel.lezcano@linaro.org>
Reported-and-tested-by: default avatarSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: default avatarRafael J. Wysocki <rafael.j.wysocki@intel.com>
parent b787f68c

drivers/cpuidle/cpuidle.c

+16 −0
Original line number Diff line number Diff line
@@ -158,9 +158,18 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, 
	int entered_state;



	struct cpuidle_state *target_state = &drv->states[index];

	bool broadcast = !!(target_state->flags & CPUIDLE_FLAG_TIMER_STOP);

	ktime_t time_start, time_end;

	s64 diff;



	/*

	 * Tell the time framework to switch to a broadcast timer because our

	 * local timer will be shut down.  If a local timer is used from another

	 * CPU as a broadcast timer, this call may fail if it is not available.

	 */

	if (broadcast && tick_broadcast_enter())

		return -EBUSY;



	trace_cpu_idle_rcuidle(index, dev->cpu);

	time_start = ktime_get();
@@ -169,6 +178,13 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, 
	time_end = ktime_get();

	trace_cpu_idle_rcuidle(PWR_EVENT_EXIT, dev->cpu);



	if (broadcast) {

		if (WARN_ON_ONCE(!irqs_disabled()))

			local_irq_disable();



		tick_broadcast_exit();

	}



	if (!cpuidle_state_is_coupled(dev, drv, entered_state))

		local_irq_enable();

kernel/sched/idle.c

+2 −14
Original line number Diff line number Diff line
@@ -81,7 +81,6 @@ static void cpuidle_idle_call(void) 
	struct cpuidle_device *dev = __this_cpu_read(cpuidle_devices);

	struct cpuidle_driver *drv = cpuidle_get_cpu_driver(dev);

	int next_state, entered_state;

	unsigned int broadcast;

	bool reflect;



	/*
@@ -150,17 +149,6 @@ static void cpuidle_idle_call(void) 
		goto exit_idle;

	}



	broadcast = drv->states[next_state].flags & CPUIDLE_FLAG_TIMER_STOP;



	/*

	 * Tell the time framework to switch to a broadcast timer

	 * because our local timer will be shutdown. If a local timer

	 * is used from another cpu as a broadcast timer, this call may

	 * fail if it is not available

	 */

	if (broadcast && tick_broadcast_enter())

		goto use_default;



	/* Take note of the planned idle state. */

	idle_set_state(this_rq(), &drv->states[next_state]);
@@ -174,8 +162,8 @@ static void cpuidle_idle_call(void) 
	/* The cpu is no longer idle or about to enter idle. */

	idle_set_state(this_rq(), NULL);



	if (broadcast)

		tick_broadcast_exit();

	if (entered_state == -EBUSY)

		goto use_default;



	/*

	 * Give the governor an opportunity to reflect on the outcome

CRA Git | Maintained and supported by SUSTech CRA and CCSE