Commit 18404756 authored by Max Krasnyansky's avatar Max Krasnyansky Committed by Thomas Gleixner
Browse files

genirq: Expose default irq affinity mask (take 3) 



Current IRQ affinity interface does not provide a way to set affinity
for the IRQs that will be allocated/activated in the future.
This patch creates /proc/irq/default_smp_affinity that lets users set
default affinity mask for the newly allocated IRQs. Changing the default
does not affect affinity masks for the currently active IRQs, they
have to be changed explicitly.

Updated based on Paul J's comments and added some more documentation.

Signed-off-by: default avatarMax Krasnyansky <maxk@qualcomm.com>
Cc: pj@sgi.com
Cc: a.p.zijlstra@chello.nl
Cc: tglx@linutronix.de
Cc: rdunlap@xenotime.net
Cc: mingo@elte.hu
Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
parent c3b25b32

Documentation/IRQ-affinity.txt

+28 −9
Original line number Diff line number Diff line 
ChangeLog:

	Started by Ingo Molnar <mingo@redhat.com>

	Update by Max Krasnyansky <maxk@qualcomm.com>



SMP IRQ affinity, started by Ingo Molnar <mingo@redhat.com>



SMP IRQ affinity



/proc/irq/IRQ#/smp_affinity specifies which target CPUs are permitted

for a given IRQ source. It's a bitmask of allowed CPUs. It's not allowed

to turn off all CPUs, and if an IRQ controller does not support IRQ

affinity then the value will not change from the default 0xffffffff.



/proc/irq/default_smp_affinity specifies default affinity mask that applies

to all non-active IRQs. Once IRQ is allocated/activated its affinity bitmask

will be set to the default mask. It can then be changed as described above.

Default mask is 0xffffffff.



Here is an example of restricting IRQ44 (eth1) to CPU0-3 then restricting

the IRQ to CPU4-7 (this is an 8-CPU SMP box):

it to CPU4-7 (this is an 8-CPU SMP box):



[root@moon 44]# cd /proc/irq/44

[root@moon 44]# cat smp_affinity

ffffffff



[root@moon 44]# echo 0f > smp_affinity

[root@moon 44]# cat smp_affinity

0000000f
@@ -21,17 +30,27 @@ PING hell (195.4.7.3): 56 data bytes 
--- hell ping statistics ---

6029 packets transmitted, 6027 packets received, 0% packet loss

round-trip min/avg/max = 0.1/0.1/0.4 ms

[root@moon 44]# cat /proc/interrupts | grep 44:

 44:          0       1785       1785       1783       1783          1

1          0   IO-APIC-level  eth1

[root@moon 44]# cat /proc/interrupts | grep 'CPU\|44:'

           CPU0       CPU1       CPU2       CPU3      CPU4       CPU5        CPU6       CPU7

 44:       1068       1785       1785       1783         0          0           0         0    IO-APIC-level  eth1



As can be seen from the line above IRQ44 was delivered only to the first four

processors (0-3).

Now lets restrict that IRQ to CPU(4-7).



[root@moon 44]# echo f0 > smp_affinity

[root@moon 44]# cat smp_affinity

000000f0

[root@moon 44]# ping -f h

PING hell (195.4.7.3): 56 data bytes

..

--- hell ping statistics ---

2779 packets transmitted, 2777 packets received, 0% packet loss

round-trip min/avg/max = 0.1/0.5/585.4 ms

[root@moon 44]# cat /proc/interrupts | grep 44:

 44:       1068       1785       1785       1784       1784       1069       1070       1069   IO-APIC-level  eth1

[root@moon 44]#

[root@moon 44]# cat /proc/interrupts |  'CPU\|44:'

           CPU0       CPU1       CPU2       CPU3      CPU4       CPU5        CPU6       CPU7

 44:       1068       1785       1785       1783      1784       1069        1070       1069   IO-APIC-level  eth1



This time around IRQ44 was delivered only to the last four processors.

i.e counters for the CPU0-3 did not change.

Documentation/filesystems/proc.txt

+18 −11
Original line number Diff line number Diff line
@@ -380,28 +380,35 @@ i386 and x86_64 platforms support the new IRQ vector displays. 
Of some interest is the introduction of the /proc/irq directory to 2.4.

It could be used to set IRQ to CPU affinity, this means that you can "hook" an

IRQ to only one CPU, or to exclude a CPU of handling IRQs. The contents of the

irq subdir is one subdir for each IRQ, and one file; prof_cpu_mask

irq subdir is one subdir for each IRQ, and two files; default_smp_affinity and

prof_cpu_mask.



For example 

  > ls /proc/irq/

  0  10  12  14  16  18  2  4  6  8  prof_cpu_mask

  1  11  13  15  17  19  3  5  7  9

  1  11  13  15  17  19  3  5  7  9  default_smp_affinity

  > ls /proc/irq/0/

  smp_affinity



The contents of the prof_cpu_mask file and each smp_affinity file for each IRQ

is the same by default:

smp_affinity is a bitmask, in which you can specify which CPUs can handle the

IRQ, you can set it by doing:



  > echo 1 > /proc/irq/10/smp_affinity



This means that only the first CPU will handle the IRQ, but you can also echo

5 which means that only the first and fourth CPU can handle the IRQ.



The contents of each smp_affinity file is the same by default:



  > cat /proc/irq/0/smp_affinity

  ffffffff



It's a bitmask, in which you can specify which CPUs can handle the IRQ, you can

set it by doing:



  > echo 1 > /proc/irq/prof_cpu_mask

The default_smp_affinity mask applies to all non-active IRQs, which are the

IRQs which have not yet been allocated/activated, and hence which lack a

/proc/irq/[0-9]* directory.



This means that only the first CPU will handle the IRQ, but you can also echo 5

which means that only the first and fourth CPU can handle the IRQ.

prof_cpu_mask specifies which CPUs are to be profiled by the system wide

profiler. Default value is ffffffff (all cpus).



The way IRQs are routed is handled by the IO-APIC, and it's Round Robin

between all the CPUs which are allowed to handle it. As usual the kernel has

arch/alpha/kernel/irq.c

+2 −3
Original line number Diff line number Diff line
@@ -42,8 +42,7 @@ void ack_bad_irq(unsigned int irq) 
#ifdef CONFIG_SMP 

static char irq_user_affinity[NR_IRQS];



int

select_smp_affinity(unsigned int irq)

int irq_select_affinity(unsigned int irq)

{

	static int last_cpu;

	int cpu = last_cpu + 1;
@@ -51,7 +50,7 @@ select_smp_affinity(unsigned int irq) 
	if (!irq_desc[irq].chip->set_affinity || irq_user_affinity[irq])

		return 1;



	while (!cpu_possible(cpu))

	while (!cpu_possible(cpu) || !cpu_isset(cpu, irq_default_affinity))

		cpu = (cpu < (NR_CPUS-1) ? cpu + 1 : 0);

	last_cpu = cpu;

include/linux/interrupt.h

+5 −0
Original line number Diff line number Diff line
@@ -104,8 +104,11 @@ extern void enable_irq(unsigned int irq); 


#if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)



extern cpumask_t irq_default_affinity;



extern int irq_set_affinity(unsigned int irq, cpumask_t cpumask);

extern int irq_can_set_affinity(unsigned int irq);

extern int irq_select_affinity(unsigned int irq);



#else /* CONFIG_SMP */
@@ -119,6 +122,8 @@ static inline int irq_can_set_affinity(unsigned int irq) 
	return 0;

}



static inline int irq_select_affinity(unsigned int irq)  { return 0; }



#endif /* CONFIG_SMP && CONFIG_GENERIC_HARDIRQS */



#ifdef CONFIG_GENERIC_HARDIRQS

include/linux/irq.h

+0 −9
Original line number Diff line number Diff line
@@ -244,15 +244,6 @@ static inline void set_balance_irq_affinity(unsigned int irq, cpumask_t mask) 
}

#endif



#ifdef CONFIG_AUTO_IRQ_AFFINITY

extern int select_smp_affinity(unsigned int irq);

#else

static inline int select_smp_affinity(unsigned int irq)

{

	return 1;

}

#endif



extern int no_irq_affinity;



static inline int irq_balancing_disabled(unsigned int irq)

CRA Git | Maintained and supported by SUSTech CRA and CCSE