Commit b8bf2681 authored by Greg Kroah-Hartman's avatar Greg Kroah-Hartman
Browse files

Merge tag 'fpga-dfl-for-5.4' of... 

Merge tag 'fpga-dfl-for-5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/mdf/linux-fpga

 into char-misc-next

Moritz writes:

FPGA DFL Changes for 5.4

This pull-request contains the FPGA DFL changes for 5.4

- The first three patches are cleanup patches making use of dev_groups and
  making the init callback optional.
- One patch adds userclock sysfs entries that are DFL specific
- One patch exposes AFU port disable/enable functions
- One patch adds error reporting
- One patch adds AFU SignalTap support
- One patch adds FME global error reporting
- The final patch is a documentation patch that decribes the
  virtualization interfaces

This patchset requires the 'dev_groups_all_drivers' tag from drivers
core for the dev_groups refactoring as well as the DFL changes already
in char-misc-next.

Signed-off-by: default avatarMoritz Fischer <mdf@kernel.org>

* tag 'fpga-dfl-for-5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/mdf/linux-fpga:
  Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
  fpga: dfl: fme: add global error reporting support
  fpga: dfl: afu: add STP (SignalTap) support
  fpga: dfl: afu: add error reporting support.
  fpga: dfl: afu: expose __afu_port_enable/disable function.
  fpga: dfl: afu: add userclock sysfs interfaces.
  fpga: dfl: afu: convert platform_driver to use dev_groups
  fpga: dfl: fme: convert platform_driver to use dev_groups
  fpga: dfl: make init callback optional
  driver core: add dev_groups to all drivers
parents ec13c78d 77a0ef48

Documentation/ABI/testing/sysfs-platform-dfl-fme

+62 −0
Original line number Diff line number Diff line
@@ -44,3 +44,65 @@ Description: Read-only. It returns socket_id to indicate which socket 
		this FPGA belongs to, only valid for integrated solution.

		User only needs this information, in case standard numa node

		can't provide correct information.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/pcie0_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-Write. Read this file for errors detected on pcie0 link.

		Write this file to clear errors logged in pcie0_errors. Write

		fails with -EINVAL if input parsing fails or input error code

		doesn't match.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/pcie1_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-Write. Read this file for errors detected on pcie1 link.

		Write this file to clear errors logged in pcie1_errors. Write

		fails with -EINVAL if input parsing fails or input error code

		doesn't match.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/nonfatal_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. It returns non-fatal errors detected.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/catfatal_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. It returns catastrophic and fatal errors detected.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/inject_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-Write. Read this file to check errors injected. Write this

		file to inject errors for testing purpose. Write fails with

		-EINVAL if input parsing fails or input inject error code isn't

		supported.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/fme_errors

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-Write. Read this file to get errors detected on FME.

		Write this file to clear errors logged in fme_errors. Write

		fials with -EINVAL if input parsing fails or input error code

		doesn't match.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/first_error

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the first error detected by

		hardware.



What:		/sys/bus/platform/devices/dfl-fme.0/errors/next_error

Date:		August 2019

KernelVersion:  5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the second error detected by

		hardware.

Documentation/ABI/testing/sysfs-platform-dfl-port

+53 −0
Original line number Diff line number Diff line
@@ -46,3 +46,56 @@ Contact: Wu Hao <hao.wu@intel.com> 
Description:	Read-write. Read or set AFU latency tolerance reporting value.

		Set ltr to 1 if the AFU can tolerate latency >= 40us or set it

		to 0 if it is latency sensitive.



What:		/sys/bus/platform/devices/dfl-port.0/userclk_freqcmd

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Write-only. User writes command to this interface to set

		userclock to AFU.



What:		/sys/bus/platform/devices/dfl-port.0/userclk_freqsts

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the status of issued command

		to userclck_freqcmd.



What:		/sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Write-only. User writes command to this interface to set

		userclock counter.



What:		/sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the status of issued command

		to userclck_freqcntrcmd.



What:		/sys/bus/platform/devices/dfl-port.0/errors/errors

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-Write. Read this file to get errors detected on port and

		Accelerated Function Unit (AFU). Write error code to this file

		to clear errors. Write fails with -EINVAL if input parsing

		fails or input error code doesn't match. Write fails with

		-EBUSY or -ETIMEDOUT if error can't be cleared as hardware

		in low power state (-EBUSY) or not respoding (-ETIMEDOUT).



What:		/sys/bus/platform/devices/dfl-port.0/errors/first_error

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the first error detected by

		hardware.



What:		/sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req

Date:		August 2019

KernelVersion:	5.4

Contact:	Wu Hao <hao.wu@intel.com>

Description:	Read-only. Read this file to get the first malformed request

		captured by hardware.

Documentation/fpga/dfl.rst

+105 −0
Original line number Diff line number Diff line
@@ -87,6 +87,8 @@ The following functions are exposed through ioctls: 
- Get driver API version (DFL_FPGA_GET_API_VERSION)

- Check for extensions (DFL_FPGA_CHECK_EXTENSION)

- Program bitstream (DFL_FPGA_FME_PORT_PR)

- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)

- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)



More functions are exposed through sysfs

(/sys/class/fpga_region/regionX/dfl-fme.n/):
@@ -102,6 +104,10 @@ More functions are exposed through sysfs 
     one FPGA device may have more than one port, this sysfs interface indicates

     how many ports the FPGA device has.



 Global error reporting management (errors/)

     error reporting sysfs interfaces allow user to read errors detected by the

     hardware, and clear the logged errors.





FIU - PORT

==========
@@ -143,6 +149,10 @@ More functions are exposed through sysfs: 
 Read Accelerator GUID (afu_id)

     afu_id indicates which PR bitstream is programmed to this AFU.



 Error reporting (errors/)

     error reporting sysfs interfaces allow user to read port/afu errors

     detected by the hardware, and clear the logged errors.





DFL Framework Overview

======================
@@ -218,6 +228,101 @@ the compat_id exposed by the target FPGA region. This check is usually done by 
userspace before calling the reconfiguration IOCTL.





FPGA virtualization - PCIe SRIOV

================================

This section describes the virtualization support on DFL based FPGA device to

enable accessing an accelerator from applications running in a virtual machine

(VM). This section only describes the PCIe based FPGA device with SRIOV support.



Features supported by the particular FPGA device are exposed through Device

Feature Lists, as illustrated below:



::



    +-------------------------------+  +-------------+

    |              PF               |  |     VF      |

    +-------------------------------+  +-------------+

        ^            ^         ^              ^

        |            |         |              |

  +-----|------------|---------|--------------|-------+

  |     |            |         |              |       |

  |  +-----+     +-------+ +-------+      +-------+   |

  |  | FME |     | Port0 | | Port1 |      | Port2 |   |

  |  +-----+     +-------+ +-------+      +-------+   |

  |                  ^         ^              ^       |

  |                  |         |              |       |

  |              +-------+ +------+       +-------+   |

  |              |  AFU  | |  AFU |       |  AFU  |   |

  |              +-------+ +------+       +-------+   |

  |                                                   |

  |            DFL based FPGA PCIe Device             |

  +---------------------------------------------------+



FME is always accessed through the physical function (PF).



Ports (and related AFUs) are accessed via PF by default, but could be exposed

through virtual function (VF) devices via PCIe SRIOV. Each VF only contains

1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)

created via PCIe SRIOV interface, to virtual machines.



The driver organization in virtualization case is illustrated below:

::



    +-------++------++------+             |

    | FME   || FME  || FME  |             |

    | FPGA  || FPGA || FPGA |             |

    |Manager||Bridge||Region|             |

    +-------++------++------+             |

    +-----------------------+  +--------+ |             +--------+

    |          FME          |  |  AFU   | |             |  AFU   |

    |         Module        |  | Module | |             | Module |

    +-----------------------+  +--------+ |             +--------+

          +-----------------------+       |       +-----------------------+

          | FPGA Container Device |       |       | FPGA Container Device |

          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |

          +-----------------------+       |       +-----------------------+

            +------------------+          |         +------------------+

            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |

            +------------------+   Host   | Machine +------------------+

   -------------------------------------- | ------------------------------

             +---------------+            |          +---------------+

             | PCI PF Device |            |          | PCI VF Device |

             +---------------+            |          +---------------+



FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device

is detected. It:



* Finishes enumeration on both FPGA PCIe PF and VF device using common

  interfaces from DFL framework.

* Supports SRIOV.



The FME device driver plays a management role in this driver architecture, it

provides ioctls to release Port from PF and assign Port to PF. After release

a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV

sysfs interface.



To enable accessing an accelerator from applications running in a VM, the

respective AFU's port needs to be assigned to a VF using the following steps:



#. The PF owns all AFU ports by default. Any port that needs to be

   reassigned to a VF must first be released through the

   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.



#. Once N ports are released from PF, then user can use command below

   to enable SRIOV and VFs. Each VF owns only one Port with AFU.



   ::



      echo N > $PCI_DEVICE_PATH/sriov_numvfs



#. Pass through the VFs to VMs



#. The AFU under VF is accessible from applications in VM (using the

   same driver inside the VF).



Note that an FME can't be assigned to a VF, thus PR and other management

functions are only available via the PF.



Device enumeration

==================

This section introduces how applications enumerate the fpga device from

drivers/base/dd.c

+14 −0
Original line number Diff line number Diff line
@@ -554,9 +554,16 @@ re_probe: 
			goto probe_failed;

	}



	if (device_add_groups(dev, drv->dev_groups)) {

		dev_err(dev, "device_add_groups() failed\n");

		goto dev_groups_failed;

	}



	if (test_remove) {

		test_remove = false;



		device_remove_groups(dev, drv->dev_groups);



		if (dev->bus->remove)

			dev->bus->remove(dev);

		else if (drv->remove)
@@ -584,6 +591,11 @@ re_probe: 
		 drv->bus->name, __func__, dev_name(dev), drv->name);

	goto done;



dev_groups_failed:

	if (dev->bus->remove)

		dev->bus->remove(dev);

	else if (drv->remove)

		drv->remove(dev);

probe_failed:

	if (dev->bus)

		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
@@ -1114,6 +1126,8 @@ static void __device_release_driver(struct device *dev, struct device *parent) 


		pm_runtime_put_sync(dev);



		device_remove_groups(dev, drv->dev_groups);



		if (dev->bus && dev->bus->remove)

			dev->bus->remove(dev);

		else if (drv->remove)

drivers/fpga/Makefile

+2 −1
Original line number Diff line number Diff line
@@ -39,8 +39,9 @@ obj-$(CONFIG_FPGA_DFL_FME_BRIDGE) += dfl-fme-br.o 
obj-$(CONFIG_FPGA_DFL_FME_REGION)	+= dfl-fme-region.o

obj-$(CONFIG_FPGA_DFL_AFU)		+= dfl-afu.o



dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o

dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o dfl-fme-error.o

dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o

dfl-afu-objs += dfl-afu-error.o



# Drivers for FPGAs which implement DFL

obj-$(CONFIG_FPGA_DFL_PCI)		+= dfl-pci.o

CRA Git | Maintained and supported by SUSTech CRA and CCSE