Merge tag 'linux-kselftest-kunit-5.7-rc1' of...

==============

KUnit is a lightweight unit testing and mocking framework for the Linux kernel.

These tests are able to be run locally on a developer's workstation without a VM

or special hardware.

KUnit is heavily inspired by JUnit, Python's unittest.mock, and

Googletest/Googlemock for C++. KUnit provides facilities for defining unit test

cases, grouping related test cases into test suites, providing common

infrastructure for running tests, and much more.

KUnit consists of a kernel component, which provides a set of macros for easily

writing unit tests. Tests written against KUnit will run on kernel boot if

built-in, or when loaded if built as a module. These tests write out results to

the kernel log in `TAP <https://testanything.org/>`_ format.

To make running these tests (and reading the results) easier, KUnit offers

:doc:`kunit_tool <kunit-tool>`, which builds a `User Mode Linux

<http://user-mode-linux.sourceforge.net>`_ kernel, runs it, and parses the test

results. This provides a quick way of running KUnit tests during development,

without requiring a virtual machine or separate hardware.

Get started now: :doc:`start`

Why KUnit?

possible if the code under test is very small and does not have any external

dependencies outside of the test's control like hardware.

Outside of KUnit, there are no testing frameworks currently

available for the kernel that do not require installing the kernel on a test

machine or in a VM and all require tests to be written in userspace running on

the kernel; this is true for Autotest, and kselftest, disqualifying

any of them from being considered unit testing frameworks.

KUnit provides a common framework for unit tests within the kernel.

KUnit tests can be run on most architectures, and most tests are architecture

independent. All built-in KUnit tests run on kernel startup.  Alternatively,

KUnit and KUnit tests can be built as modules and tests will run when the test

module is loaded.

KUnit addresses the problem of being able to run tests without needing a virtual

machine or actual hardware with User Mode Linux. User Mode Linux is a Linux

architecture, like ARM or x86; however, unlike other architectures it compiles

to a standalone program that can be run like any other program directly inside

of a host operating system; to be clear, it does not require any virtualization

support; it is just a regular program.

.. note::

Alternatively, kunit and kunit tests can be built as modules and tests will

run when the test module is loaded.

        KUnit can also run tests without needing a virtual machine or actual

        hardware under User Mode Linux. User Mode Linux is a Linux architecture,

        like ARM or x86, which compiles the kernel as a Linux executable. KUnit

        can be used with UML either by building with ``ARCH=um`` (like any other

        architecture), or by using :doc:`kunit_tool <kunit-tool>`.

KUnit is fast. Excluding build time, from invocation to completion KUnit can run

several dozen tests in only 10 to 20 seconds; this might not sound like a big

*   :doc:`start` - for new users of KUnit

*   :doc:`usage` - for a more detailed explanation of KUnit features

*   :doc:`api/index` - for the list of KUnit APIs used for testing

*   :doc:`kunit-tool` - for more information on the kunit_tool helper script

*   :doc:`faq` - for answers to some common questions about KUnit

<http://user-mode-linux.sourceforge.net/>`_), running KUnit tests, parsing

the test results and displaying them in a user friendly manner.

kunit_tool addresses the problem of being able to run tests without needing a

virtual machine or actual hardware with User Mode Linux. User Mode Linux is a

Linux architecture, like ARM or x86; however, unlike other architectures it

compiles the kernel as a standalone Linux executable that can be run like any

other program directly inside of a host operating system. To be clear, it does

not require any virtualization support: it is just a regular program.

What is a kunitconfig?

======================

KUnit has the same dependencies as the Linux kernel. As long as you can build

the kernel, you can run KUnit.

KUnit Wrapper

=============

Included with KUnit is a simple Python wrapper that helps format the output to

easily use and read KUnit output. It handles building and running the kernel, as

well as formatting the output.

Running tests with the KUnit Wrapper

====================================

Included with KUnit is a simple Python wrapper which runs tests under User Mode

Linux, and formats the test results.

The wrapper can be run with:

:doc:`kunit-tool` page.

Creating a .kunitconfig

=======================

The Python script is a thin wrapper around Kbuild. As such, it needs to be

configured with a ``.kunitconfig`` file. This file essentially contains the

regular Kernel config, with the specific test targets as well.

-----------------------

If you want to run a specific set of tests (rather than those listed in the

KUnit defconfig), you can provide Kconfig options in the ``.kunitconfig`` file.

This file essentially contains the regular Kernel config, with the specific

test targets as well. The ``.kunitconfig`` should also contain any other config

options required by the tests.

A good starting point for a ``.kunitconfig`` is the KUnit defconfig:

.. code-block:: bash

	cd $PATH_TO_LINUX_REPO

	cp arch/um/configs/kunit_defconfig .kunitconfig

Verifying KUnit Works

---------------------

You can then add any other Kconfig options you wish, e.g.:

.. code-block:: none

        CONFIG_LIST_KUNIT_TEST=y

:doc:`kunit_tool <kunit-tool>` will ensure that all config options set in

``.kunitconfig`` are set in the kernel ``.config`` before running the tests.

It'll warn you if you haven't included the dependencies of the options you're

using.

.. note::

   Note that removing something from the ``.kunitconfig`` will not trigger a

   rebuild of the ``.config`` file: the configuration is only updated if the

   ``.kunitconfig`` is not a subset of ``.config``. This means that you can use

   other tools (such as make menuconfig) to adjust other config options.

Running the tests

-----------------

To make sure that everything is set up correctly, simply invoke the Python

wrapper from your kernel repo:

	Because it is building a lot of sources for the first time, the

	``Building KUnit kernel`` step may take a while.

Running tests without the KUnit Wrapper

=======================================

If you'd rather not use the KUnit Wrapper (if, for example, you need to

integrate with other systems, or use an architecture other than UML), KUnit can

be included in any kernel, and the results read out and parsed manually.

.. note::

   KUnit is not designed for use in a production system, and it's possible that

   tests may reduce the stability or security of the system.

Configuring the kernel

----------------------

In order to enable KUnit itself, you simply need to enable the ``CONFIG_KUNIT``

Kconfig option (it's under Kernel Hacking/Kernel Testing and Coverage in

menuconfig). From there, you can enable any KUnit tests you want: they usually

have config options ending in ``_KUNIT_TEST``.

KUnit and KUnit tests can be compiled as modules: in this case the tests in a

module will be run when the module is loaded.

Running the tests

-----------------

Build and run your kernel as usual. Test output will be written to the kernel

log in `TAP <https://testanything.org/>`_ format.

.. note::

   It's possible that there will be other lines and/or data interspersed in the

   TAP output.

Writing your first test

=======================

.. TODO(brendanhiggins@google.com): Add an actual example of an architecture

   dependent KUnit test.

KUnit debugfs representation

============================

When kunit test suites are initialized, they create an associated directory

in /sys/kernel/debug/kunit/<test-suite>.  The directory contains one file

- results: "cat results" displays results of each test case and the results

  of the entire suite for the last test run.

The debugfs representation is primarily of use when kunit test suites are

run in a native environment, either as modules or builtin.  Having a way

to display results like this is valuable as otherwise results can be

intermixed with other events in dmesg output.  The maximum size of each

results file is KUNIT_LOG_SIZE bytes (defined in include/kunit/test.h).

struct kunit;

/* Size of log associated with test. */

#define KUNIT_LOG_SIZE	512

/*

 * TAP specifies subtest stream indentation of 4 spaces, 8 spaces for a

 * sub-subtest.  See the "Subtests" section in

 * https://node-tap.org/tap-protocol/

 */

#define KUNIT_SUBTEST_INDENT		"    "

#define KUNIT_SUBSUBTEST_INDENT		"        "

/**

 * struct kunit_case - represents an individual test case.

 *

	/* private: internal use only. */

	bool success;

	char *log;

};

static inline char *kunit_status_to_string(bool status)

{

	return status ? "ok" : "not ok";

}

/**

 * KUNIT_CASE - A helper for creating a &struct kunit_case

 *

	int (*init)(struct kunit *test);

	void (*exit)(struct kunit *test);

	struct kunit_case *test_cases;

	/* private - internal use only */

	struct dentry *debugfs;

	char *log;

};

/**

	/* private: internal use only. */

	const char *name; /* Read only after initialization! */

	char *log; /* Points at case log after initialization */

	struct kunit_try_catch try_catch;

	/*

	 * success starts as true, and may only be set to false during a

	struct list_head resources; /* Protected by lock. */

};

void kunit_init_test(struct kunit *test, const char *name);

void kunit_init_test(struct kunit *test, const char *name, char *log);

int kunit_run_tests(struct kunit_suite *suite);

size_t kunit_suite_num_test_cases(struct kunit_suite *suite);

unsigned int kunit_test_case_num(struct kunit_suite *suite,

				 struct kunit_case *test_case);

int __kunit_test_suites_init(struct kunit_suite **suites);

void __kunit_test_suites_exit(struct kunit_suite **suites);

/**

 * kunit_test_suites() - used to register one or more &struct kunit_suite

 *			 with KUnit.

	static struct kunit_suite *suites[] = { __VA_ARGS__, NULL};	\

	static int kunit_test_suites_init(void)				\

	{								\

		unsigned int i;						\

		for (i = 0; suites[i] != NULL; i++)			\

			kunit_run_tests(suites[i]);			\

		return 0;						\

		return __kunit_test_suites_init(suites);		\

	}								\

	late_initcall(kunit_test_suites_init);				\

	static void __exit kunit_test_suites_exit(void)			\

	{								\

		return;							\

		return __kunit_test_suites_exit(suites);		\

	}								\

	module_exit(kunit_test_suites_exit)

#define kunit_test_suite(suite)	kunit_test_suites(&suite)

#define kunit_suite_for_each_test_case(suite, test_case)		\

	for (test_case = suite->test_cases; test_case->run_case; test_case++)

bool kunit_suite_has_succeeded(struct kunit_suite *suite);

/*

 * Like kunit_alloc_resource() below, but returns the struct kunit_resource

 * object that contains the allocation. This is mostly for testing purposes.

void kunit_cleanup(struct kunit *test);

void kunit_log_append(char *log, const char *fmt, ...);

/*

 * printk and log to per-test or per-suite log buffer.  Logging only done

 * if CONFIG_KUNIT_DEBUGFS is 'y'; if it is 'n', no log is allocated/used.

 */

#define kunit_log(lvl, test_or_suite, fmt, ...)				\

	do {								\

		printk(lvl fmt, ##__VA_ARGS__);				\

		kunit_log_append((test_or_suite)->log,	fmt "\n",	\

				 ##__VA_ARGS__);			\

	} while (0)

#define kunit_printk(lvl, test, fmt, ...)				\

	printk(lvl "\t# %s: " fmt, (test)->name, ##__VA_ARGS__)

	kunit_log(lvl, test, KUNIT_SUBTEST_INDENT "# %s: " fmt,		\

		  (test)->name,	##__VA_ARGS__)

/**

 * kunit_info() - Prints an INFO level message associated with @test.