Commit 5b8e2f61 authored Dec 06, 2016 by Eric Dumazet Committed by David S. Miller Dec 07, 2016

net: sock_rps_record_flow() is for connected sockets



Paolo noticed a cache line miss in UDP recvmsg() to access
sk_rxhash, sharing a cache line with sk_drops.

sk_drops might be heavily incremented by cpus handling a flood targeting
this socket.

We might place sk_drops on a separate cache line, but lets try
to avoid wasting 64 bytes per socket just for this, since we have
other bottlenecks to take care of.

sock_rps_record_flow() should only access sk_rxhash for connected
flows.

Testing sk_state for TCP_ESTABLISHED covers most of the cases for
connected sockets, for a zero cost, since system calls using
sock_rps_record_flow() also access sk->sk_prot which is on the
same cache line.

A follow up patch will provide a static_key (Jump Label) since most
hosts do not even use RFS.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Paolo Abeni <pabeni@redhat.com>
Acked-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

parent d4aea20d

include/net/sock.h

+11 −1

Original line number	Diff line number	Diff line
		@@ -913,6 +913,16 @@ static inline void sock_rps_record_flow_hash(__u32 hash)
		static inline void sock_rps_record_flow(const struct sock *sk)
		{
		#ifdef CONFIG_RPS
		/* Reading sk->sk_rxhash might incur an expensive cache line miss.
		*
		* TCP_ESTABLISHED does cover almost all states where RFS
		* might be useful, and is cheaper [1] than testing :
		* IPv4: inet_sk(sk)->inet_daddr
		* IPv6: ipv6_addr_any(&sk->sk_v6_daddr)
		* OR an additional socket flag
		* [1] : sk_state and sk_prot are in the same cache line.
		*/
		if (sk->sk_state == TCP_ESTABLISHED)
		sock_rps_record_flow_hash(sk->sk_rxhash);
		#endif
		}

Admin message