Commit 7fd3253a authored Nov 30, 2020 by Björn Töpel Committed by Daniel Borkmann Dec 01, 2020

net: Introduce preferred busy-polling



The existing busy-polling mode, enabled by the SO_BUSY_POLL socket
option or system-wide using the /proc/sys/net/core/busy_read knob, is
an opportunistic. That means that if the NAPI context is not
scheduled, it will poll it. If, after busy-polling, the budget is
exceeded the busy-polling logic will schedule the NAPI onto the
regular softirq handling.

One implication of the behavior above is that a busy/heavy loaded NAPI
context will never enter/allow for busy-polling. Some applications
prefer that most NAPI processing would be done by busy-polling.

This series adds a new socket option, SO_PREFER_BUSY_POLL, that works
in concert with the napi_defer_hard_irqs and gro_flush_timeout
knobs. The napi_defer_hard_irqs and gro_flush_timeout knobs were
introduced in commit 6f8b12d6 ("net: napi: add hard irqs deferral
feature"), and allows for a user to defer interrupts to be enabled and
instead schedule the NAPI context from a watchdog timer. When a user
enables the SO_PREFER_BUSY_POLL, again with the other knobs enabled,
and the NAPI context is being processed by a softirq, the softirq NAPI
processing will exit early to allow the busy-polling to be performed.

If the application stops performing busy-polling via a system call,
the watchdog timer defined by gro_flush_timeout will timeout, and
regular softirq handling will resume.

In summary; Heavy traffic applications that prefer busy-polling over
softirq processing should use this option.

Example usage:

  $ echo 2 | sudo tee /sys/class/net/ens785f1/napi_defer_hard_irqs
  $ echo 200000 | sudo tee /sys/class/net/ens785f1/gro_flush_timeout

Note that the timeout should be larger than the userspace processing
window, otherwise the watchdog will timeout and fall back to regular
softirq processing.

Enable the SO_BUSY_POLL/SO_PREFER_BUSY_POLL options on your socket.

Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Jakub Kicinski <kuba@kernel.org>
Link: https://lore.kernel.org/bpf/20201130185205.196029-2-bjorn.topel@gmail.com

parent 854055c0

arch/alpha/include/uapi/asm/socket.h

+2 −0

Original line number	Diff line number	Diff line
		@@ -124,6 +124,8 @@

		#define SO_DETACH_REUSEPORT_BPF 68

		#define SO_PREFER_BUSY_POLL 69

		#if !defined(__KERNEL__)

		#if __BITS_PER_LONG == 64

arch/mips/include/uapi/asm/socket.h

+2 −0

Original line number	Diff line number	Diff line
		@@ -135,6 +135,8 @@

		#define SO_DETACH_REUSEPORT_BPF 68

		#define SO_PREFER_BUSY_POLL 69

		#if !defined(__KERNEL__)

		#if __BITS_PER_LONG == 64

arch/parisc/include/uapi/asm/socket.h

+2 −0

Original line number	Diff line number	Diff line
		@@ -116,6 +116,8 @@

		#define SO_DETACH_REUSEPORT_BPF 0x4042

		#define SO_PREFER_BUSY_POLL 0x4043

		#if !defined(__KERNEL__)

		#if __BITS_PER_LONG == 64

arch/sparc/include/uapi/asm/socket.h

+2 −0

Original line number	Diff line number	Diff line
		@@ -117,6 +117,8 @@

		#define SO_DETACH_REUSEPORT_BPF 0x0047

		#define SO_PREFER_BUSY_POLL 0x0048

		#if !defined(__KERNEL__)

fs/eventpoll.c

+1 −1

Original line number	Diff line number	Diff line
		@@ -397,7 +397,7 @@ static void ep_busy_loop(struct eventpoll *ep, int nonblock)
		unsigned int napi_id = READ_ONCE(ep->napi_id);

		if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on())
		napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep);
		napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false);
		}

		static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep)

Admin message