Commit 17405453 authored Aug 20, 2013 by Yoshihiro YUNOMAE Committed by Ingo Molnar Aug 20, 2013

x86/ioapic/kcrash: Prevent crash_kexec() from deadlocking on ioapic_lock



Prevent crash_kexec() from deadlocking on ioapic_lock. When
crash_kexec() is executed on a CPU, the CPU will take ioapic_lock
in disable_IO_APIC(). So if the cpu gets an NMI while locking
ioapic_lock, a deadlock will happen.

In this patch, ioapic_lock is zapped/initialized before disable_IO_APIC().

You can reproduce this deadlock the following way:

1. Add mdelay(1000) after raw_spin_lock_irqsave() in
   native_ioapic_set_affinity()@arch/x86/kernel/apic/io_apic.c

   Although the deadlock can occur without this modification, it will increase
   the potential of the deadlock problem.

2. Build and install the kernel

3. Set up the OS which will run panic() and kexec when NMI is injected
    # echo "kernel.unknown_nmi_panic=1" >> /etc/sysctl.conf
    # vim /etc/default/grub
      add "nmi_watchdog=0 crashkernel=256M" in GRUB_CMDLINE_LINUX line
    # grub2-mkconfig

4. Reboot the OS

5. Run following command for each vcpu on the guest
    # while true; do echo <CPU num> > /proc/irq/<IO-APIC-edge or IO-APIC-fasteoi>/smp_affinitity; done;
   By running this command, cpus will get ioapic_lock for setting affinity.

6. Inject NMI (push a dump button or execute 'virsh inject-nmi <domain>' if you
   use VM). After injecting NMI, panic() is called in an nmi-handler context.
   Then, kexec will normally run in panic(), but the operation will be stopped
   by deadlock on ioapic_lock in crash_kexec()->machine_crash_shutdown()->
   native_machine_crash_shutdown()->disable_IO_APIC()->clear_IO_APIC()->
   clear_IO_APIC_pin()->ioapic_read_entry().

Signed-off-by: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Gleb Natapov <gleb@redhat.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Joerg Roedel <joro@8bytes.org>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: yrl.pp-manager.tt@hitachi.com
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Seiji Aguchi <seiji.aguchi@hds.com>
Link: http://lkml.kernel.org/r/20130820070107.28245.83806.stgit@yunodevel


Signed-off-by: Ingo Molnar <mingo@kernel.org>

parent 82982d72

arch/x86/include/asm/apic.h

+2 −0

Original line number	Diff line number	Diff line
		@@ -715,4 +715,6 @@ static inline void exiting_ack_irq(void)
		ack_APIC_irq();
		}

		extern void ioapic_zap_locks(void);

		#endif /* _ASM_X86_APIC_H */

arch/x86/kernel/apic/io_apic.c

+5 −0

Original line number	Diff line number	Diff line
		@@ -1534,6 +1534,11 @@ void intel_ir_io_apic_print_entries(unsigned int apic,
		}
		}

		void ioapic_zap_locks(void)
		{
		raw_spin_lock_init(&ioapic_lock);
		}

		__apicdebuginit(void) print_IO_APIC(int ioapic_idx)
		{
		union IO_APIC_reg_00 reg_00;

arch/x86/kernel/crash.c

+3 −1

Original line number	Diff line number	Diff line
		@@ -128,7 +128,9 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
		cpu_emergency_svm_disable();

		lapic_shutdown();
		#if defined(CONFIG_X86_IO_APIC)
		#ifdef CONFIG_X86_IO_APIC
		/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
		ioapic_zap_locks();
		disable_IO_APIC();
		#endif
		#ifdef CONFIG_HPET_TIMER

Admin message