cpumask: make cpumask_next() out-of-line (f22ef333) · Commits · 戴 / test

Every for_each_XXX_cpu() invocation calls cpumask_next() which is an inline function: static inline unsigned int cpumask_next(int n, const struct cpumask *srcp) { /* -1 is a legal arg here. */ if (n != -1) cpumask_check(n); return find_next_bit(cpumask_bits(srcp), nr_cpumask_bits, n + 1); } However! find_next_bit() is regular out-of-line function which means "nr_cpu_ids" load and increment happen at the caller resulting in a lot of bloat x86_64 defconfig: add/remove: 3/0 grow/shrink: 8/373 up/down: 155/-5668 (-5513) x86_64 allyesconfig-ish: add/remove: 3/1 grow/shrink: 57/634 up/down: 3515/-28177 (-24662) !!! Some archs redefine find_next_bit() but it is OK: m68k inline but SMP is not supported arm out-of-line unicore32 out-of-line Function call will happen anyway, so move load and increment into callee. Link: http://lkml.kernel.org/r/20170824230010.GA1593@avx2 Signed-off-by:

Alexey Dobriyan <adobriyan@gmail.com> Signed-off-by:

Andrew Morton <akpm@linux-foundation.org> Signed-off-by:

Linus Torvalds <torvalds@linux-foundation.org>

include/linux/cpumask.h

+1 −14

Original line number	Diff line number	Diff line
		@@ -178,20 +178,7 @@ static inline unsigned int cpumask_first(const struct cpumask *srcp)
		return find_first_bit(cpumask_bits(srcp), nr_cpumask_bits);
		}

		/**
		* cpumask_next - get the next cpu in a cpumask
		* @n: the cpu prior to the place to search (ie. return will be > @n)
		* @srcp: the cpumask pointer
		*
		* Returns >= nr_cpu_ids if no further cpus set.
		*/
		static inline unsigned int cpumask_next(int n, const struct cpumask *srcp)
		{
		/* -1 is a legal arg here. */
		if (n != -1)
		cpumask_check(n);
		return find_next_bit(cpumask_bits(srcp), nr_cpumask_bits, n+1);
		}
		unsigned int cpumask_next(int n, const struct cpumask *srcp);

		/**
		* cpumask_next_zero - get the next unset cpu in a cpumask

lib/cpumask.c

+16 −0

Original line number	Diff line number	Diff line
		@@ -5,6 +5,22 @@
		#include <linux/export.h>
		#include <linux/bootmem.h>

		/**
		* cpumask_next - get the next cpu in a cpumask
		* @n: the cpu prior to the place to search (ie. return will be > @n)
		* @srcp: the cpumask pointer
		*
		* Returns >= nr_cpu_ids if no further cpus set.
		*/
		unsigned int cpumask_next(int n, const struct cpumask *srcp)
		{
		/* -1 is a legal arg here. */
		if (n != -1)
		cpumask_check(n);
		return find_next_bit(cpumask_bits(srcp), nr_cpumask_bits, n + 1);
		}
		EXPORT_SYMBOL(cpumask_next);

		/**
		* cpumask_next_and - get the next cpu in src1p & src2p
		* @n: the cpu prior to the place to search (ie. return will be > @n)

Admin message