Merge branch 'tracing-core-for-linus' of...

			     Event Tracing

		Documentation written by Theodore Ts'o

			Updated by Li Zefan

		Updated by Li Zefan and Tom Zanussi

1. Introduction

===============

See The example provided in samples/trace_events

4. Event formats

================

Each trace event has a 'format' file associated with it that contains

a description of each field in a logged event.  This information can

be used to parse the binary trace stream, and is also the place to

find the field names that can be used in event filters (see section 5).

It also displays the format string that will be used to print the

event in text mode, along with the event name and ID used for

profiling.

Every event has a set of 'common' fields associated with it; these are

the fields prefixed with 'common_'.  The other fields vary between

events and correspond to the fields defined in the TRACE_EVENT

definition for that event.

Each field in the format has the form:

     field:field-type field-name; offset:N; size:N;

where offset is the offset of the field in the trace record and size

is the size of the data item, in bytes.

For example, here's the information displayed for the 'sched_wakeup'

event:

# cat /debug/tracing/events/sched/sched_wakeup/format

name: sched_wakeup

ID: 60

format:

	field:unsigned short common_type;	offset:0;	size:2;

	field:unsigned char common_flags;	offset:2;	size:1;

	field:unsigned char common_preempt_count;	offset:3;	size:1;

	field:int common_pid;	offset:4;	size:4;

	field:int common_tgid;	offset:8;	size:4;

	field:char comm[TASK_COMM_LEN];	offset:12;	size:16;

	field:pid_t pid;	offset:28;	size:4;

	field:int prio;	offset:32;	size:4;

	field:int success;	offset:36;	size:4;

	field:int cpu;	offset:40;	size:4;

print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,

	   REC->prio, REC->success, REC->cpu

This event contains 10 fields, the first 5 common and the remaining 5

event-specific.  All the fields for this event are numeric, except for

'comm' which is a string, a distinction important for event filtering.

5. Event filtering

==================

Trace events can be filtered in the kernel by associating boolean

'filter expressions' with them.  As soon as an event is logged into

the trace buffer, its fields are checked against the filter expression

associated with that event type.  An event with field values that

'match' the filter will appear in the trace output, and an event whose

values don't match will be discarded.  An event with no filter

associated with it matches everything, and is the default when no

filter has been set for an event.

5.1 Expression syntax

---------------------

A filter expression consists of one or more 'predicates' that can be

combined using the logical operators '&&' and '||'.  A predicate is

simply a clause that compares the value of a field contained within a

logged event with a constant value and returns either 0 or 1 depending

on whether the field value matched (1) or didn't match (0):

	  field-name relational-operator value

Parentheses can be used to provide arbitrary logical groupings and

double-quotes can be used to prevent the shell from interpreting

operators as shell metacharacters.

The field-names available for use in filters can be found in the

'format' files for trace events (see section 4).

The relational-operators depend on the type of the field being tested:

The operators available for numeric fields are:

==, !=, <, <=, >, >=

And for string fields they are:

==, !=

Currently, only exact string matches are supported.

Currently, the maximum number of predicates in a filter is 16.

5.2 Setting filters

-------------------

A filter for an individual event is set by writing a filter expression

to the 'filter' file for the given event.

For example:

# cd /debug/tracing/events/sched/sched_wakeup

# echo "common_preempt_count > 4" > filter

A slightly more involved example:

# cd /debug/tracing/events/sched/sched_signal_send

# echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter

If there is an error in the expression, you'll get an 'Invalid

argument' error when setting it, and the erroneous string along with

an error message can be seen by looking at the filter e.g.:

# cd /debug/tracing/events/sched/sched_signal_send

# echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter

-bash: echo: write error: Invalid argument

# cat filter

((sig >= 10 && sig < 15) || dsig == 17) && comm != bash

^

parse_error: Field not found

Currently the caret ('^') for an error always appears at the beginning of

the filter string; the error message should still be useful though

even without more accurate position info.

5.3 Clearing filters

--------------------

To clear the filter for an event, write a '0' to the event's filter

file.

To clear the filters for all events in a subsystem, write a '0' to the

subsystem's filter file.

5.3 Subsystem filters

---------------------

For convenience, filters for every event in a subsystem can be set or

cleared as a group by writing a filter expression into the filter file

at the root of the subsytem.  Note however, that if a filter for any

event within the subsystem lacks a field specified in the subsystem

filter, or if the filter can't be applied for any other reason, the

filter for that event will retain its previous setting.  This can

result in an unintended mixture of filters which could lead to

confusing (to the user who might think different filters are in

effect) trace output.  Only filters that reference just the common

fields can be guaranteed to propagate successfully to all events.

Here are a few subsystem filter examples that also illustrate the

above points:

Clear the filters on all events in the sched subsytem:

# cd /sys/kernel/debug/tracing/events/sched

# echo 0 > filter

# cat sched_switch/filter

none

# cat sched_wakeup/filter

none

Set a filter using only common fields for all events in the sched

subsytem (all events end up with the same filter):

# cd /sys/kernel/debug/tracing/events/sched

# echo common_pid == 0 > filter

# cat sched_switch/filter

common_pid == 0

# cat sched_wakeup/filter

common_pid == 0

Attempt to set a filter using a non-common field for all events in the

sched subsytem (all events but those that have a prev_pid field retain

their old filters):

# cd /sys/kernel/debug/tracing/events/sched

# echo prev_pid == 0 > filter

# cat sched_switch/filter

prev_pid == 0

# cat sched_wakeup/filter

common_pid == 0

		function tracer guts

		====================

Introduction

------------

Here we will cover the architecture pieces that the common function tracing

code relies on for proper functioning.  Things are broken down into increasing

complexity so that you can start simple and at least get basic functionality.

Note that this focuses on architecture implementation details only.  If you

want more explanation of a feature in terms of common code, review the common

ftrace.txt file.

Prerequisites

-------------

Ftrace relies on these features being implemented:

 STACKTRACE_SUPPORT - implement save_stack_trace()

 TRACE_IRQFLAGS_SUPPORT - implement include/asm/irqflags.h

HAVE_FUNCTION_TRACER

--------------------

You will need to implement the mcount and the ftrace_stub functions.

The exact mcount symbol name will depend on your toolchain.  Some call it

"mcount", "_mcount", or even "__mcount".  You can probably figure it out by

running something like:

	$ echo 'main(){}' | gcc -x c -S -o - - -pg | grep mcount

	        call    mcount

We'll make the assumption below that the symbol is "mcount" just to keep things

nice and simple in the examples.

Keep in mind that the ABI that is in effect inside of the mcount function is

*highly* architecture/toolchain specific.  We cannot help you in this regard,

sorry.  Dig up some old documentation and/or find someone more familiar than

you to bang ideas off of.  Typically, register usage (argument/scratch/etc...)

is a major issue at this point, especially in relation to the location of the

mcount call (before/after function prologue).  You might also want to look at

how glibc has implemented the mcount function for your architecture.  It might

be (semi-)relevant.

The mcount function should check the function pointer ftrace_trace_function

to see if it is set to ftrace_stub.  If it is, there is nothing for you to do,

so return immediately.  If it isn't, then call that function in the same way

the mcount function normally calls __mcount_internal -- the first argument is

the "frompc" while the second argument is the "selfpc" (adjusted to remove the

size of the mcount call that is embedded in the function).

For example, if the function foo() calls bar(), when the bar() function calls

mcount(), the arguments mcount() will pass to the tracer are:

	"frompc" - the address bar() will use to return to foo()

	"selfpc" - the address bar() (with _mcount() size adjustment)

Also keep in mind that this mcount function will be called *a lot*, so

optimizing for the default case of no tracer will help the smooth running of

your system when tracing is disabled.  So the start of the mcount function is

typically the bare min with checking things before returning.  That also means

the code flow should usually kept linear (i.e. no branching in the nop case).

This is of course an optimization and not a hard requirement.

Here is some pseudo code that should help (these functions should actually be

implemented in assembly):

void ftrace_stub(void)

{

	return;

}

void mcount(void)

{

	/* save any bare state needed in order to do initial checking */

	extern void (*ftrace_trace_function)(unsigned long, unsigned long);

	if (ftrace_trace_function != ftrace_stub)

		goto do_trace;

	/* restore any bare state */

	return;

do_trace:

	/* save all state needed by the ABI (see paragraph above) */

	unsigned long frompc = ...;

	unsigned long selfpc = <return address> - MCOUNT_INSN_SIZE;

	ftrace_trace_function(frompc, selfpc);

	/* restore all state needed by the ABI */

}

Don't forget to export mcount for modules !

extern void mcount(void);

EXPORT_SYMBOL(mcount);

HAVE_FUNCTION_TRACE_MCOUNT_TEST

-------------------------------

This is an optional optimization for the normal case when tracing is turned off

in the system.  If you do not enable this Kconfig option, the common ftrace

code will take care of doing the checking for you.

To support this feature, you only need to check the function_trace_stop

variable in the mcount function.  If it is non-zero, there is no tracing to be

done at all, so you can return.

This additional pseudo code would simply be:

void mcount(void)

{

	/* save any bare state needed in order to do initial checking */

+	if (function_trace_stop)

+		return;

	extern void (*ftrace_trace_function)(unsigned long, unsigned long);

	if (ftrace_trace_function != ftrace_stub)

...

HAVE_FUNCTION_GRAPH_TRACER

--------------------------

Deep breath ... time to do some real work.  Here you will need to update the

mcount function to check ftrace graph function pointers, as well as implement

some functions to save (hijack) and restore the return address.

The mcount function should check the function pointers ftrace_graph_return

(compare to ftrace_stub) and ftrace_graph_entry (compare to

ftrace_graph_entry_stub).  If either of those are not set to the relevant stub

function, call the arch-specific function ftrace_graph_caller which in turn

calls the arch-specific function prepare_ftrace_return.  Neither of these

function names are strictly required, but you should use them anyways to stay

consistent across the architecture ports -- easier to compare & contrast

things.

The arguments to prepare_ftrace_return are slightly different than what are

passed to ftrace_trace_function.  The second argument "selfpc" is the same,

but the first argument should be a pointer to the "frompc".  Typically this is

located on the stack.  This allows the function to hijack the return address

temporarily to have it point to the arch-specific function return_to_handler.

That function will simply call the common ftrace_return_to_handler function and

that will return the original return address with which, you can return to the

original call site.

Here is the updated mcount pseudo code:

void mcount(void)

{

...

	if (ftrace_trace_function != ftrace_stub)

		goto do_trace;

+#ifdef CONFIG_FUNCTION_GRAPH_TRACER

+	extern void (*ftrace_graph_return)(...);

+	extern void (*ftrace_graph_entry)(...);

+	if (ftrace_graph_return != ftrace_stub ||

+	    ftrace_graph_entry != ftrace_graph_entry_stub)

+		ftrace_graph_caller();

+#endif

	/* restore any bare state */

...

Here is the pseudo code for the new ftrace_graph_caller assembly function:

#ifdef CONFIG_FUNCTION_GRAPH_TRACER

void ftrace_graph_caller(void)

{

	/* save all state needed by the ABI */

	unsigned long *frompc = &...;

	unsigned long selfpc = <return address> - MCOUNT_INSN_SIZE;

	prepare_ftrace_return(frompc, selfpc);

	/* restore all state needed by the ABI */

}

#endif

For information on how to implement prepare_ftrace_return(), simply look at

the x86 version.  The only architecture-specific piece in it is the setup of

the fault recovery table (the asm(...) code).  The rest should be the same

across architectures.

Here is the pseudo code for the new return_to_handler assembly function.  Note

that the ABI that applies here is different from what applies to the mcount

code.  Since you are returning from a function (after the epilogue), you might

be able to skimp on things saved/restored (usually just registers used to pass

return values).

#ifdef CONFIG_FUNCTION_GRAPH_TRACER

void return_to_handler(void)

{

	/* save all state needed by the ABI (see paragraph above) */

	void (*original_return_point)(void) = ftrace_return_to_handler();

	/* restore all state needed by the ABI */

	/* this is usually either a return or a jump */

	original_return_point();

}

#endif

HAVE_FTRACE_NMI_ENTER

---------------------

If you can't trace NMI functions, then skip this option.

<details to be filled>

HAVE_FTRACE_SYSCALLS

---------------------

<details to be filled>

HAVE_FTRACE_MCOUNT_RECORD

-------------------------

See scripts/recordmcount.pl for more info.

<details to be filled>

HAVE_DYNAMIC_FTRACE

---------------------

<details to be filled>

means that the list of tracers can always grow).

Implementation Details

----------------------

See ftrace-design.txt for details for arch porters and such.

The File System

---------------

F:	fs/fscache/

F:	include/linux/fscache*.h

FTRACE

TRACING

M:	Steven Rostedt <rostedt@goodmis.org>

M:	Frederic Weisbecker <fweisbec@gmail.com>

M:	Ingo Molnar <mingo@redhat.com>

T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tracing/core

S:	Maintained

F:	Documentation/trace/ftrace.txt

F:	arch/*/*/*/ftrace.h

F:	arch/*/kernel/ftrace.c

F:	include/*/ftrace.h

F:	include/*/ftrace.h include/trace/ include/linux/trace*.h

F:	kernel/trace/

FUJITSU FR-V (FRV) PORT

	depends on TRACING_SUPPORT

	select TRACING

	select RING_BUFFER

	select RING_BUFFER_ALLOW_SWAP

	help

	  OProfile is a profiling system capable of profiling the

	  whole system, include the kernel, kernel modules, libraries,