Merge tag 'media/v4.20-7' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media

    * - ``V4L2_BUF_TYPE_META_CAPTURE``

      - 13

      - Buffer for metadata capture, see :ref:`metadata`.

    * - ``V4L2_BUF_TYPE_META_OUTPUT``

      - 14

      - Buffer for metadata output, see :ref:`metadata`.

******************

Metadata refers to any non-image data that supplements video frames with

additional information. This may include statistics computed over the image

or frame capture parameters supplied by the image source. This interface is

intended for transfer of metadata to userspace and control of that operation.

additional information. This may include statistics computed over the image,

frame capture parameters supplied by the image source or device specific

parameters for specifying how the device processes images. This interface is

intended for transfer of metadata between the userspace and the hardware and

control of that operation.

The metadata interface is implemented on video capture device nodes. The device

can be dedicated to metadata or can implement both video and metadata capture

as specified in its reported capabilities.

The metadata interface is implemented on video device nodes. The device can be

dedicated to metadata or can support both video and metadata as specified in its

reported capabilities.

Querying Capabilities

=====================

Device nodes supporting the metadata interface set the ``V4L2_CAP_META_CAPTURE``

flag in the ``device_caps`` field of the

Device nodes supporting the metadata capture interface set the

``V4L2_CAP_META_CAPTURE`` flag in the ``device_caps`` field of the

:c:type:`v4l2_capability` structure returned by the :c:func:`VIDIOC_QUERYCAP`

ioctl. That flag means the device can capture metadata to memory.

ioctl. That flag means the device can capture metadata to memory. Similarly,

device nodes supporting metadata output interface set the

``V4L2_CAP_META_OUTPUT`` flag in the ``device_caps`` field of

:c:type:`v4l2_capability` structure. That flag means the device can read

metadata from memory.

At least one of the read/write or streaming I/O methods must be supported.

supported as well.

To use the :ref:`format` ioctls applications set the ``type`` field of the

:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_META_CAPTURE`` and use the

:c:type:`v4l2_meta_format` ``meta`` member of the ``fmt`` union as needed per

the desired operation. Both drivers and applications must set the remainder of

the :c:type:`v4l2_format` structure to 0.

:c:type:`v4l2_format` structure to ``V4L2_BUF_TYPE_META_CAPTURE`` or to

``V4L2_BUF_TYPE_META_OUTPUT`` and use the :c:type:`v4l2_meta_format` ``meta``

member of the ``fmt`` union as needed per the desired operation. Both drivers

and applications must set the remainder of the :c:type:`v4l2_format` structure

to 0.

.. c:type:: v4l2_meta_format

.. toctree::

    :maxdepth: 1

    pixfmt-meta-intel-ipu3

    pixfmt-meta-d4xx

    pixfmt-meta-uvc

    pixfmt-meta-vsp1-hgo

.. -*- coding: utf-8; mode: rst -*-

.. _v4l2-meta-fmt-params:

.. _v4l2-meta-fmt-stat-3a:

******************************************************************

V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')

******************************************************************

.. c:type:: ipu3_uapi_stats_3a

3A statistics

=============

For IPU3 ImgU, the 3A statistics accelerators collect different statistics over

an input bayer frame. Those statistics, defined in data struct :c:type:`ipu3_uapi_stats_3a`,

are obtained from "ipu3-imgu 3a stat" metadata capture video node, which are then

passed to user space for statistics analysis using :c:type:`v4l2_meta_format` interface.

The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and

Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,

and AE (Auto-exposure) histogram.

struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above.

.. code-block:: c

	struct ipu3_uapi_stats_3a {

		struct ipu3_uapi_awb_raw_buffer awb_raw_buffer;

		struct ipu3_uapi_ae_raw_buffer_aligned ae_raw_buffer[IPU3_UAPI_MAX_STRIPES];

		struct ipu3_uapi_af_raw_buffer af_raw_buffer;

		struct ipu3_uapi_awb_fr_raw_buffer awb_fr_raw_buffer;

		struct ipu3_uapi_4a_config stats_4a_config;

		__u32 ae_join_buffers;

		__u8 padding[28];

		struct ipu3_uapi_stats_3a_bubble_info_per_stripe stats_3a_bubble_per_stripe;

		struct ipu3_uapi_ff_status stats_3a_status;

	};

.. c:type:: ipu3_uapi_params

Pipeline parameters

===================

IPU3 pipeline has a number of image processing stages, each of which takes a

set of parameters as input. The major stages of pipelines are shown here:

Raw pixels -> Bayer Downscaling -> Optical Black Correction ->

Linearization -> Lens Shading Correction -> White Balance / Exposure /

Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color

Correction Matrix -> Gamma correction -> Color Space Conversion ->

Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color

Correction -> XNR3 -> TNR -> DDR

The table below presents a description of the above algorithms.

======================== =======================================================

Name			 Description

======================== =======================================================

Optical Black Correction Optical Black Correction block subtracts a pre-defined

			 value from the respective pixel values to obtain better

			 image quality.

			 Defined in :c:type:`ipu3_uapi_obgrid_param`.

Linearization		 This algo block uses linearization parameters to

			 address non-linearity sensor effects. The Lookup table

			 table is defined in

			 :c:type:`ipu3_uapi_isp_lin_vmem_params`.

SHD			 Lens shading correction is used to correct spatial

			 non-uniformity of the pixel response due to optical

			 lens shading. This is done by applying a different gain

			 for each pixel. The gain, black level etc are

			 configured in :c:type:`ipu3_uapi_shd_config_static`.

BNR			 Bayer noise reduction block removes image noise by

			 applying a bilateral filter.

			 See :c:type:`ipu3_uapi_bnr_static_config` for details.

ANR			 Advanced Noise Reduction is a block based algorithm

			 that performs noise reduction in the Bayer domain. The

			 convolution matrix etc can be found in

			 :c:type:`ipu3_uapi_anr_config`.

Demosaicing		 Demosaicing converts raw sensor data in Bayer format

			 into RGB (Red, Green, Blue) presentation. Then add

			 outputs of estimation of Y channel for following stream

			 processing by Firmware. The struct is defined as

			 :c:type:`ipu3_uapi_dm_config`. (TODO)

Color Correction	 Color Correction algo transforms sensor specific color

			 space to the standard "sRGB" color space. This is done

			 by applying 3x3 matrix defined in

			 :c:type:`ipu3_uapi_ccm_mat_config`.

Gamma correction	 Gamma correction :c:type:`ipu3_uapi_gamma_config` is a

			 basic non-linear tone mapping correction that is

			 applied per pixel for each pixel component.

CSC			 Color space conversion transforms each pixel from the

			 RGB primary presentation to YUV (Y: brightness,

			 UV: Luminance) presentation. This is done by applying

			 a 3x3 matrix defined in

			 :c:type:`ipu3_uapi_csc_mat_config`

CDS			 Chroma down sampling

			 After the CSC is performed, the Chroma Down Sampling

			 is applied for a UV plane down sampling by a factor

			 of 2 in each direction for YUV 4:2:0 using a 4x2

			 configurable filter :c:type:`ipu3_uapi_cds_params`.

CHNR			 Chroma noise reduction

			 This block processes only the chrominance pixels and

			 performs noise reduction by cleaning the high

			 frequency noise.

			 See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.

TCC			 Total color correction as defined in struct

			 :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.

XNR3			 eXtreme Noise Reduction V3 is the third revision of

			 noise reduction algorithm used to improve image

			 quality. This removes the low frequency noise in the

			 captured image. Two related structs are  being defined,

			 :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory

			 and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector

			 memory.

TNR			 Temporal Noise Reduction block compares successive

			 frames in time to remove anomalies / noise in pixel

			 values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and

			 :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP

			 vector and data memory respectively.

======================== =======================================================

A few stages of the pipeline will be executed by firmware running on the ISP

processor, while many others will use a set of fixed hardware blocks also

called accelerator cluster (ACC) to crunch pixel data and produce statistics.

ACC parameters of individual algorithms, as defined by

:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user

space through struct :c:type:`ipu3_uapi_flags` embedded in

:c:type:`ipu3_uapi_params` structure. For parameters that are configured as

not enabled by the user space, the corresponding structs are ignored by the

driver, in which case the existing configuration of the algorithm will be

preserved.

Both 3A statistics and pipeline parameters described here are closely tied to

the underlying camera sub-system (CSS) APIs. They are usually consumed and

produced by dedicated user space libraries that comprise the important tuning

tools, thus freeing the developers from being bothered with the low level

hardware and algorithm details.

It should be noted that IPU3 DMA operations require the addresses of all data

structures (that includes both input and output) to be aligned on 32 byte

boundaries.

The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"

video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.

.. code-block:: c

	struct ipu3_uapi_params {

		/* Flags which of the settings below are to be applied */

		struct ipu3_uapi_flags use;

		/* Accelerator cluster parameters */

		struct ipu3_uapi_acc_param acc_param;

		/* ISP vector address space parameters */

		struct ipu3_uapi_isp_lin_vmem_params lin_vmem_params;

		struct ipu3_uapi_isp_tnr3_vmem_params tnr3_vmem_params;

		struct ipu3_uapi_isp_xnr3_vmem_params xnr3_vmem_params;

		/* ISP data memory (DMEM) parameters */

		struct ipu3_uapi_isp_tnr3_params tnr3_dmem_params;

		struct ipu3_uapi_isp_xnr3_params xnr3_dmem_params;

		/* Optical black level compensation */

		struct ipu3_uapi_obgrid_param obgrid_param;

	};

Intel IPU3 ImgU uAPI data types

===============================

.. kernel-doc:: drivers/staging/media/ipu3/include/intel-ipu3.h

    * - ``V4L2_CAP_STREAMING``

      - 0x04000000

      - The device supports the :ref:`streaming <mmap>` I/O method.

    * - ``V4L2_CAP_META_OUTPUT``

      - 0x08000000

      - The device supports the :ref:`metadata` output interface.

    * - ``V4L2_CAP_TOUCH``

      - 0x10000000

      - This is a touch device.