doc: kernel: smp: Add section about IPI cascades

IPI, and this code will only be used for testing purposes or on

systems without power consumption requirements.

IPI Cascades

============

The kernel can not control the order in which IPIs are processed by the CPUs

in the system. In general, this is not an issue and a single set of IPIs is

sufficient to trigger a reschedule on the N CPUs that results with them

scheduling the highest N priority ready threads to execute. When CPU masking

is used, there may be more than one valid set of threads (not to be confused

with an optimal set of threads) that can be scheduled on the N CPUs and a

single set of IPIs may be insufficient to result in any of these valid sets.

.. note::

    When CPU masking is not in play, the optimal set of threads is the same

    as the valid set of threads. However when CPU masking is in play, there

    may be more than one valid set--one of which may be optimal.

    To better illustrate the distinction, consider a 2-CPU system with ready

    threads T1 and T2 at priorities 1 and 2 respectively. Let T2 be pinned to

    CPU0 and T1 not be pinned. If CPU0 is executing T2 and CPU1 executing T1,

    then this set is is both valid and optimal. However, if CPU0 is executing

    T1 and CPU1 is idling, then this too would be valid though not optimal.

In those cases where a single set of IPIs is not sufficient to generate a valid

set, the resulting set of executing threads are expected to be close to a valid

set, and subsequent IPIs can generally be expected to correct the situation

soon. However, for cases where neither the approximation nor the delay are

acceptable, enabling :kconfig:option:`CONFIG_SCHED_IPI_CASCADE` will allow the

kernel to generate cascading IPIs until the kernel has selected a valid set of

ready threads for the CPUs.

There are three types of costs/penalties associated with the IPI cascades--and

for these reasons they are disabled by default. The first is a cost incurred

by the CPU producing the IPI when a new thread preempts the old thread as checks

must be done to compare the old thread against the threads executing on the

other CPUs. The second is a cost incurred by the CPUs receiving the IPIs as

they must be processed. The third is the apparent sputtering of a thread as it

"winks in" and then "winks out" due to cascades stemming from the

aforementioned first cost.

SMP Kernel Internals

********************