Merge branch 'bpf-ring-buffer'

===============

BPF ring buffer

===============

This document describes BPF ring buffer design, API, and implementation details.

.. contents::

    :local:

    :depth: 2

Motivation

----------

There are two distinctive motivators for this work, which are not satisfied by

existing perf buffer, which prompted creation of a new ring buffer

implementation.

- more efficient memory utilization by sharing ring buffer across CPUs;

- preserving ordering of events that happen sequentially in time, even across

  multiple CPUs (e.g., fork/exec/exit events for a task).

These two problems are independent, but perf buffer fails to satisfy both.

Both are a result of a choice to have per-CPU perf ring buffer.  Both can be

also solved by having an MPSC implementation of ring buffer. The ordering

problem could technically be solved for perf buffer with some in-kernel

counting, but given the first one requires an MPSC buffer, the same solution

would solve the second problem automatically.

Semantics and APIs

------------------

Single ring buffer is presented to BPF programs as an instance of BPF map of

type ``BPF_MAP_TYPE_RINGBUF``. Two other alternatives considered, but

ultimately rejected.

One way would be to, similar to ``BPF_MAP_TYPE_PERF_EVENT_ARRAY``, make

``BPF_MAP_TYPE_RINGBUF`` could represent an array of ring buffers, but not

enforce "same CPU only" rule. This would be more familiar interface compatible

with existing perf buffer use in BPF, but would fail if application needed more

advanced logic to lookup ring buffer by arbitrary key.

``BPF_MAP_TYPE_HASH_OF_MAPS`` addresses this with current approach.

Additionally, given the performance of BPF ringbuf, many use cases would just

opt into a simple single ring buffer shared among all CPUs, for which current

approach would be an overkill.

Another approach could introduce a new concept, alongside BPF map, to represent

generic "container" object, which doesn't necessarily have key/value interface

with lookup/update/delete operations. This approach would add a lot of extra

infrastructure that has to be built for observability and verifier support. It

would also add another concept that BPF developers would have to familiarize

themselves with, new syntax in libbpf, etc. But then would really provide no

additional benefits over the approach of using a map.  ``BPF_MAP_TYPE_RINGBUF``

doesn't support lookup/update/delete operations, but so doesn't few other map

types (e.g., queue and stack; array doesn't support delete, etc).

The approach chosen has an advantage of re-using existing BPF map

infrastructure (introspection APIs in kernel, libbpf support, etc), being

familiar concept (no need to teach users a new type of object in BPF program),

and utilizing existing tooling (bpftool). For common scenario of using a single

ring buffer for all CPUs, it's as simple and straightforward, as would be with

a dedicated "container" object. On the other hand, by being a map, it can be

combined with ``ARRAY_OF_MAPS`` and ``HASH_OF_MAPS`` map-in-maps to implement

a wide variety of topologies, from one ring buffer for each CPU (e.g., as

a replacement for perf buffer use cases), to a complicated application

hashing/sharding of ring buffers (e.g., having a small pool of ring buffers

with hashed task's tgid being a look up key to preserve order, but reduce

contention).

Key and value sizes are enforced to be zero. ``max_entries`` is used to specify

the size of ring buffer and has to be a power of 2 value.

There are a bunch of similarities between perf buffer

(``BPF_MAP_TYPE_PERF_EVENT_ARRAY``) and new BPF ring buffer semantics:

- variable-length records;

- if there is no more space left in ring buffer, reservation fails, no

  blocking;

- memory-mappable data area for user-space applications for ease of

  consumption and high performance;

- epoll notifications for new incoming data;

- but still the ability to do busy polling for new data to achieve the

  lowest latency, if necessary.

BPF ringbuf provides two sets of APIs to BPF programs:

- ``bpf_ringbuf_output()`` allows to *copy* data from one place to a ring

  buffer, similarly to ``bpf_perf_event_output()``;

- ``bpf_ringbuf_reserve()``/``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()``

  APIs split the whole process into two steps. First, a fixed amount of space

  is reserved. If successful, a pointer to a data inside ring buffer data

  area is returned, which BPF programs can use similarly to a data inside

  array/hash maps. Once ready, this piece of memory is either committed or

  discarded. Discard is similar to commit, but makes consumer ignore the

  record.

``bpf_ringbuf_output()`` has disadvantage of incurring extra memory copy,

because record has to be prepared in some other place first. But it allows to

submit records of the length that's not known to verifier beforehand. It also

closely matches ``bpf_perf_event_output()``, so will simplify migration

significantly.

``bpf_ringbuf_reserve()`` avoids the extra copy of memory by providing a memory

pointer directly to ring buffer memory. In a lot of cases records are larger

than BPF stack space allows, so many programs have use extra per-CPU array as

a temporary heap for preparing sample. bpf_ringbuf_reserve() avoid this needs

completely. But in exchange, it only allows a known constant size of memory to

be reserved, such that verifier can verify that BPF program can't access memory

outside its reserved record space. bpf_ringbuf_output(), while slightly slower

due to extra memory copy, covers some use cases that are not suitable for

``bpf_ringbuf_reserve()``.

The difference between commit and discard is very small. Discard just marks

a record as discarded, and such records are supposed to be ignored by consumer

code. Discard is useful for some advanced use-cases, such as ensuring

all-or-nothing multi-record submission, or emulating temporary

``malloc()``/``free()`` within single BPF program invocation.

Each reserved record is tracked by verifier through existing

reference-tracking logic, similar to socket ref-tracking. It is thus

impossible to reserve a record, but forget to submit (or discard) it.

``bpf_ringbuf_query()`` helper allows to query various properties of ring

buffer.  Currently 4 are supported:

- ``BPF_RB_AVAIL_DATA`` returns amount of unconsumed data in ring buffer;

- ``BPF_RB_RING_SIZE`` returns the size of ring buffer;

- ``BPF_RB_CONS_POS``/``BPF_RB_PROD_POS`` returns current logical possition

  of consumer/producer, respectively.

Returned values are momentarily snapshots of ring buffer state and could be

off by the time helper returns, so this should be used only for

debugging/reporting reasons or for implementing various heuristics, that take

into account highly-changeable nature of some of those characteristics.

One such heuristic might involve more fine-grained control over poll/epoll

notifications about new data availability in ring buffer. Together with

``BPF_RB_NO_WAKEUP``/``BPF_RB_FORCE_WAKEUP`` flags for output/commit/discard

helpers, it allows BPF program a high degree of control and, e.g., more

efficient batched notifications. Default self-balancing strategy, though,

should be adequate for most applications and will work reliable and efficiently

already.

Design and Implementation

-------------------------

This reserve/commit schema allows a natural way for multiple producers, either

on different CPUs or even on the same CPU/in the same BPF program, to reserve

independent records and work with them without blocking other producers. This

means that if BPF program was interruped by another BPF program sharing the

same ring buffer, they will both get a record reserved (provided there is

enough space left) and can work with it and submit it independently. This

applies to NMI context as well, except that due to using a spinlock during

reservation, in NMI context, ``bpf_ringbuf_reserve()`` might fail to get

a lock, in which case reservation will fail even if ring buffer is not full.

The ring buffer itself internally is implemented as a power-of-2 sized

circular buffer, with two logical and ever-increasing counters (which might

wrap around on 32-bit architectures, that's not a problem):

- consumer counter shows up to which logical position consumer consumed the

  data;

- producer counter denotes amount of data reserved by all producers.

Each time a record is reserved, producer that "owns" the record will

successfully advance producer counter. At that point, data is still not yet

ready to be consumed, though. Each record has 8 byte header, which contains the

length of reserved record, as well as two extra bits: busy bit to denote that

record is still being worked on, and discard bit, which might be set at commit

time if record is discarded. In the latter case, consumer is supposed to skip

the record and move on to the next one. Record header also encodes record's

relative offset from the beginning of ring buffer data area (in pages). This

allows ``bpf_ringbuf_commit()``/``bpf_ringbuf_discard()`` to accept only the

pointer to the record itself, without requiring also the pointer to ring buffer

itself. Ring buffer memory location will be restored from record metadata

header. This significantly simplifies verifier, as well as improving API

usability.

Producer counter increments are serialized under spinlock, so there is

a strict ordering between reservations. Commits, on the other hand, are

completely lockless and independent. All records become available to consumer

in the order of reservations, but only after all previous records where

already committed. It is thus possible for slow producers to temporarily hold

off submitted records, that were reserved later.

Reservation/commit/consumer protocol is verified by litmus tests in

Documentation/litmus_tests/bpf-rb/_.

One interesting implementation bit, that significantly simplifies (and thus

speeds up as well) implementation of both producers and consumers is how data

area is mapped twice contiguously back-to-back in the virtual memory. This

allows to not take any special measures for samples that have to wrap around

at the end of the circular buffer data area, because the next page after the

last data page would be first data page again, and thus the sample will still

appear completely contiguous in virtual memory. See comment and a simple ASCII

diagram showing this visually in ``bpf_ringbuf_area_alloc()``.

Another feature that distinguishes BPF ringbuf from perf ring buffer is

a self-pacing notifications of new data being availability.

``bpf_ringbuf_commit()`` implementation will send a notification of new record

being available after commit only if consumer has already caught up right up to

the record being committed. If not, consumer still has to catch up and thus

will see new data anyways without needing an extra poll notification.

Benchmarks (see tools/testing/selftests/bpf/benchs/bench_ringbuf.c_) show that

this allows to achieve a very high throughput without having to resort to

tricks like "notify only every Nth sample", which are necessary with perf

buffer. For extreme cases, when BPF program wants more manual control of

notifications, commit/discard/output helpers accept ``BPF_RB_NO_WAKEUP`` and

``BPF_RB_FORCE_WAKEUP`` flags, which give full control over notifications of

data availability, but require extra caution and diligence in using this API.

	int (*map_direct_value_meta)(const struct bpf_map *map,

	int (*map_direct_value_meta)(const struct bpf_map *map,

				     u64 imm, u32 *off);

				     u64 imm, u32 *off);

	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);

	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);

	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,

			     struct poll_table_struct *pts);

};

};

struct bpf_map_memory {

struct bpf_map_memory {

	ARG_PTR_TO_LONG,	/* pointer to long */

	ARG_PTR_TO_LONG,	/* pointer to long */

	ARG_PTR_TO_SOCKET,	/* pointer to bpf_sock (fullsock) */

	ARG_PTR_TO_SOCKET,	/* pointer to bpf_sock (fullsock) */

	ARG_PTR_TO_BTF_ID,	/* pointer to in-kernel struct */

	ARG_PTR_TO_BTF_ID,	/* pointer to in-kernel struct */

	ARG_PTR_TO_ALLOC_MEM,	/* pointer to dynamically allocated memory */

	ARG_PTR_TO_ALLOC_MEM_OR_NULL,	/* pointer to dynamically allocated memory or NULL */

	ARG_CONST_ALLOC_SIZE_OR_ZERO,	/* number of allocated bytes requested */

};

};

/* type of values returned from helper functions */

/* type of values returned from helper functions */

	RET_PTR_TO_SOCKET_OR_NULL,	/* returns a pointer to a socket or NULL */

	RET_PTR_TO_SOCKET_OR_NULL,	/* returns a pointer to a socket or NULL */

	RET_PTR_TO_TCP_SOCK_OR_NULL,	/* returns a pointer to a tcp_sock or NULL */

	RET_PTR_TO_TCP_SOCK_OR_NULL,	/* returns a pointer to a tcp_sock or NULL */

	RET_PTR_TO_SOCK_COMMON_OR_NULL,	/* returns a pointer to a sock_common or NULL */

	RET_PTR_TO_SOCK_COMMON_OR_NULL,	/* returns a pointer to a sock_common or NULL */

	RET_PTR_TO_ALLOC_MEM_OR_NULL,	/* returns a pointer to dynamically allocated memory or NULL */

};

};

/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs

/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs

	PTR_TO_XDP_SOCK,	 /* reg points to struct xdp_sock */

	PTR_TO_XDP_SOCK,	 /* reg points to struct xdp_sock */

	PTR_TO_BTF_ID,		 /* reg points to kernel struct */

	PTR_TO_BTF_ID,		 /* reg points to kernel struct */

	PTR_TO_BTF_ID_OR_NULL,	 /* reg points to kernel struct or NULL */

	PTR_TO_BTF_ID_OR_NULL,	 /* reg points to kernel struct or NULL */

	PTR_TO_MEM,		 /* reg points to valid memory region */

	PTR_TO_MEM_OR_NULL,	 /* reg points to valid memory region or NULL */

};

};

/* The information passed from prog-specific *_is_valid_access

/* The information passed from prog-specific *_is_valid_access

extern const struct bpf_func_proto bpf_jiffies64_proto;

extern const struct bpf_func_proto bpf_jiffies64_proto;

extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;

extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;

extern const struct bpf_func_proto bpf_event_output_data_proto;

extern const struct bpf_func_proto bpf_event_output_data_proto;

extern const struct bpf_func_proto bpf_ringbuf_output_proto;

extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;

extern const struct bpf_func_proto bpf_ringbuf_submit_proto;

extern const struct bpf_func_proto bpf_ringbuf_discard_proto;

extern const struct bpf_func_proto bpf_ringbuf_query_proto;

const struct bpf_func_proto *bpf_tracing_func_proto(

const struct bpf_func_proto *bpf_tracing_func_proto(

	enum bpf_func_id func_id, const struct bpf_prog *prog);

	enum bpf_func_id func_id, const struct bpf_prog *prog);

#if defined(CONFIG_BPF_JIT)

#if defined(CONFIG_BPF_JIT)

BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)

BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)

#endif

#endif

BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops)

BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)

BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)

BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)

BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)

		u32 btf_id; /* for PTR_TO_BTF_ID */

		u32 btf_id; /* for PTR_TO_BTF_ID */

		u32 mem_size; /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */

		/* Max size from any of the above. */

		/* Max size from any of the above. */

		unsigned long raw;

		unsigned long raw;

	};

	};

	 * offset, so they can share range knowledge.

	 * offset, so they can share range knowledge.

	 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we

	 * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we

	 * came from, when one is tested for != NULL.

	 * came from, when one is tested for != NULL.

	 * For PTR_TO_MEM_OR_NULL this is used to identify memory allocation

	 * for the purpose of tracking that it's freed.

	 * For PTR_TO_SOCKET this is used to share which pointers retain the

	 * For PTR_TO_SOCKET this is used to share which pointers retain the

	 * same reference to the socket, to determine proper reference freeing.

	 * same reference to the socket, to determine proper reference freeing.

	 */

	 */

	BPF_MAP_TYPE_SK_STORAGE,

	BPF_MAP_TYPE_SK_STORAGE,

	BPF_MAP_TYPE_DEVMAP_HASH,

	BPF_MAP_TYPE_DEVMAP_HASH,

	BPF_MAP_TYPE_STRUCT_OPS,

	BPF_MAP_TYPE_STRUCT_OPS,

	BPF_MAP_TYPE_RINGBUF,

};

};

/* Note that tracing related programs such as

/* Note that tracing related programs such as

 *		**bpf_sk_cgroup_id**\ ().

 *		**bpf_sk_cgroup_id**\ ().

 *	Return

 *	Return

 *		The id is returned or 0 in case the id could not be retrieved.

 *		The id is returned or 0 in case the id could not be retrieved.

 *

 * void *bpf_ringbuf_output(void *ringbuf, void *data, u64 size, u64 flags)

 * 	Description

 * 		Copy *size* bytes from *data* into a ring buffer *ringbuf*.

 * 		If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of

 * 		new data availability is sent.

 * 		IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of

 * 		new data availability is sent unconditionally.

 * 	Return

 * 		0, on success;

 * 		< 0, on error.

 *

 * void *bpf_ringbuf_reserve(void *ringbuf, u64 size, u64 flags)

 * 	Description

 * 		Reserve *size* bytes of payload in a ring buffer *ringbuf*.

 * 	Return

 * 		Valid pointer with *size* bytes of memory available; NULL,

 * 		otherwise.

 *

 * void bpf_ringbuf_submit(void *data, u64 flags)

 * 	Description

 * 		Submit reserved ring buffer sample, pointed to by *data*.

 * 		If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of

 * 		new data availability is sent.

 * 		IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of

 * 		new data availability is sent unconditionally.

 * 	Return

 * 		Nothing. Always succeeds.

 *

 * void bpf_ringbuf_discard(void *data, u64 flags)

 * 	Description

 * 		Discard reserved ring buffer sample, pointed to by *data*.

 * 		If BPF_RB_NO_WAKEUP is specified in *flags*, no notification of

 * 		new data availability is sent.

 * 		IF BPF_RB_FORCE_WAKEUP is specified in *flags*, notification of

 * 		new data availability is sent unconditionally.

 * 	Return

 * 		Nothing. Always succeeds.

 *

 * u64 bpf_ringbuf_query(void *ringbuf, u64 flags)

 *	Description

 *		Query various characteristics of provided ring buffer. What

 *		exactly is queries is determined by *flags*:

 *		  - BPF_RB_AVAIL_DATA - amount of data not yet consumed;

 *		  - BPF_RB_RING_SIZE - the size of ring buffer;

 *		  - BPF_RB_CONS_POS - consumer position (can wrap around);

 *		  - BPF_RB_PROD_POS - producer(s) position (can wrap around);

 *		Data returned is just a momentary snapshots of actual values

 *		and could be inaccurate, so this facility should be used to

 *		power heuristics and for reporting, not to make 100% correct

 *		calculation.

 *	Return

 *		Requested value, or 0, if flags are not recognized.

 */

 */

#define __BPF_FUNC_MAPPER(FN)		\

#define __BPF_FUNC_MAPPER(FN)		\

	FN(unspec),			\

	FN(unspec),			\

	FN(seq_printf),			\

	FN(seq_printf),			\

	FN(seq_write),			\

	FN(seq_write),			\

	FN(sk_cgroup_id),		\

	FN(sk_cgroup_id),		\

	FN(sk_ancestor_cgroup_id),

	FN(sk_ancestor_cgroup_id),	\

	FN(ringbuf_output),		\

	FN(ringbuf_reserve),		\

	FN(ringbuf_submit),		\

	FN(ringbuf_discard),		\

	FN(ringbuf_query),

/* integer value in 'imm' field of BPF_CALL instruction selects which helper

/* integer value in 'imm' field of BPF_CALL instruction selects which helper

 * function eBPF program intends to call

 * function eBPF program intends to call

	BPF_F_GET_BRANCH_RECORDS_SIZE	= (1ULL << 0),

	BPF_F_GET_BRANCH_RECORDS_SIZE	= (1ULL << 0),

};

};

/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and

 * BPF_FUNC_bpf_ringbuf_output flags.

 */

enum {

	BPF_RB_NO_WAKEUP		= (1ULL << 0),

	BPF_RB_FORCE_WAKEUP		= (1ULL << 1),

};

/* BPF_FUNC_bpf_ringbuf_query flags */

enum {

	BPF_RB_AVAIL_DATA = 0,

	BPF_RB_RING_SIZE = 1,

	BPF_RB_CONS_POS = 2,

	BPF_RB_PROD_POS = 3,

};

/* BPF ring buffer constants */

enum {

	BPF_RINGBUF_BUSY_BIT		= (1U << 31),

	BPF_RINGBUF_DISCARD_BIT		= (1U << 30),

	BPF_RINGBUF_HDR_SZ		= 8,

};

/* Mode for BPF_FUNC_skb_adjust_room helper. */

/* Mode for BPF_FUNC_skb_adjust_room helper. */

enum bpf_adj_room_mode {

enum bpf_adj_room_mode {

	BPF_ADJ_ROOM_NET,

	BPF_ADJ_ROOM_NET,