remoteproc: add framework for controlling remote processors

Remote Processor Framework

1. Introduction

Modern SoCs typically have heterogeneous remote processor devices in asymmetric

multiprocessing (AMP) configurations, which may be running different instances

of operating system, whether it's Linux or any other flavor of real-time OS.

OMAP4, for example, has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP.

In a typical configuration, the dual cortex-A9 is running Linux in a SMP

configuration, and each of the other three cores (two M3 cores and a DSP)

is running its own instance of RTOS in an AMP configuration.

The remoteproc framework allows different platforms/architectures to

control (power on, load firmware, power off) those remote processors while

abstracting the hardware differences, so the entire driver doesn't need to be

duplicated. In addition, this framework also adds rpmsg virtio devices

for remote processors that supports this kind of communication. This way,

platform-specific remoteproc drivers only need to provide a few low-level

handlers, and then all rpmsg drivers will then just work

(for more information about the virtio-based rpmsg bus and its drivers,

please read Documentation/rpmsg.txt).

2. User API

  int rproc_boot(struct rproc *rproc)

    - Boot a remote processor (i.e. load its firmware, power it on, ...).

      If the remote processor is already powered on, this function immediately

      returns (successfully).

      Returns 0 on success, and an appropriate error value otherwise.

      Note: to use this function you should already have a valid rproc

      handle. There are several ways to achieve that cleanly (devres, pdata,

      the way remoteproc_rpmsg.c does this, or, if this becomes prevalent, we

      might also consider using dev_archdata for this). See also

      rproc_get_by_name() below.

  void rproc_shutdown(struct rproc *rproc)

    - Power off a remote processor (previously booted with rproc_boot()).

      In case @rproc is still being used by an additional user(s), then

      this function will just decrement the power refcount and exit,

      without really powering off the device.

      Every call to rproc_boot() must (eventually) be accompanied by a call

      to rproc_shutdown(). Calling rproc_shutdown() redundantly is a bug.

      Notes:

      - we're not decrementing the rproc's refcount, only the power refcount.

        which means that the @rproc handle stays valid even after

        rproc_shutdown() returns, and users can still use it with a subsequent

        rproc_boot(), if needed.

      - don't call rproc_shutdown() to unroll rproc_get_by_name(), exactly

        because rproc_shutdown() _does not_ decrement the refcount of @rproc.

        To decrement the refcount of @rproc, use rproc_put() (but _only_ if

        you acquired @rproc using rproc_get_by_name()).

  struct rproc *rproc_get_by_name(const char *name)

    - Find an rproc handle using the remote processor's name, and then

      boot it. If it's already powered on, then just immediately return

      (successfully). Returns the rproc handle on success, and NULL on failure.

      This function increments the remote processor's refcount, so always

      use rproc_put() to decrement it back once rproc isn't needed anymore.

      Note: currently rproc_get_by_name() and rproc_put() are not used anymore

      by the rpmsg bus and its drivers. We need to scrutinize the use cases

      that still need them, and see if we can migrate them to use the non

      name-based boot/shutdown interface.

  void rproc_put(struct rproc *rproc)

    - Decrement @rproc's power refcount and shut it down if it reaches zero

      (essentially by just calling rproc_shutdown), and then decrement @rproc's

      validity refcount too.

      After this function returns, @rproc may _not_ be used anymore, and its

      handle should be considered invalid.

      This function should be called _iff_ the @rproc handle was grabbed by

      calling rproc_get_by_name().

3. Typical usage

#include <linux/remoteproc.h>

/* in case we were given a valid 'rproc' handle */

int dummy_rproc_example(struct rproc *my_rproc)

{

	int ret;

	/* let's power on and boot our remote processor */

	ret = rproc_boot(my_rproc);

	if (ret) {

		/*

		 * something went wrong. handle it and leave.

		 */

	}

	/*

	 * our remote processor is now powered on... give it some work

	 */

	/* let's shut it down now */

	rproc_shutdown(my_rproc);

}

4. API for implementors

  struct rproc *rproc_alloc(struct device *dev, const char *name,

				const struct rproc_ops *ops,

				const char *firmware, int len)

    - Allocate a new remote processor handle, but don't register

      it yet. Required parameters are the underlying device, the

      name of this remote processor, platform-specific ops handlers,

      the name of the firmware to boot this rproc with, and the

      length of private data needed by the allocating rproc driver (in bytes).

      This function should be used by rproc implementations during

      initialization of the remote processor.

      After creating an rproc handle using this function, and when ready,

      implementations should then call rproc_register() to complete

      the registration of the remote processor.

      On success, the new rproc is returned, and on failure, NULL.

      Note: _never_ directly deallocate @rproc, even if it was not registered

      yet. Instead, if you just need to unroll rproc_alloc(), use rproc_free().

  void rproc_free(struct rproc *rproc)

    - Free an rproc handle that was allocated by rproc_alloc.

      This function should _only_ be used if @rproc was only allocated,

      but not registered yet.

      If @rproc was already successfully registered (by calling

      rproc_register()), then use rproc_unregister() instead.

  int rproc_register(struct rproc *rproc)

    - Register @rproc with the remoteproc framework, after it has been

      allocated with rproc_alloc().

      This is called by the platform-specific rproc implementation, whenever

      a new remote processor device is probed.

      Returns 0 on success and an appropriate error code otherwise.

      Note: this function initiates an asynchronous firmware loading

      context, which will look for virtio devices supported by the rproc's

      firmware.

      If found, those virtio devices will be created and added, so as a result

      of registering this remote processor, additional virtio drivers might get

      probed.

      Currently, though, we only support a single RPMSG virtio vdev per remote

      processor.

  int rproc_unregister(struct rproc *rproc)

    - Unregister a remote processor, and decrement its refcount.

      If its refcount drops to zero, then @rproc will be freed. If not,

      it will be freed later once the last reference is dropped.

      This function should be called when the platform specific rproc

      implementation decides to remove the rproc device. it should

      _only_ be called if a previous invocation of rproc_register()

      has completed successfully.

      After rproc_unregister() returns, @rproc is _not_ valid anymore and

      it shouldn't be used. More specifically, don't call rproc_free()

      or try to directly free @rproc after rproc_unregister() returns;

      none of these are needed, and calling them is a bug.

      Returns 0 on success and -EINVAL if @rproc isn't valid.

5. Implementation callbacks

These callbacks should be provided by platform-specific remoteproc

drivers:

/**

 * struct rproc_ops - platform-specific device handlers

 * @start:	power on the device and boot it

 * @stop:	power off the device

 * @kick:	kick a virtqueue (virtqueue id given as a parameter)

 */

struct rproc_ops {

	int (*start)(struct rproc *rproc);

	int (*stop)(struct rproc *rproc);

	void (*kick)(struct rproc *rproc, int vqid);

};

Every remoteproc implementation should at least provide the ->start and ->stop

handlers. If rpmsg functionality is also desired, then the ->kick handler

should be provided as well.

The ->start() handler takes an rproc handle and should then power on the

device and boot it (use rproc->priv to access platform-specific private data).

The boot address, in case needed, can be found in rproc->bootaddr (remoteproc

core puts there the ELF entry point).

On success, 0 should be returned, and on failure, an appropriate error code.

The ->stop() handler takes an rproc handle and powers the device down.

On success, 0 is returned, and on failure, an appropriate error code.

The ->kick() handler takes an rproc handle, and an index of a virtqueue

where new message was placed in. Implementations should interrupt the remote

processor and let it know it has pending messages. Notifying remote processors

the exact virtqueue index to look in is optional: it is easy (and not

too expensive) to go through the existing virtqueues and look for new buffers

in the used rings.

6. Binary Firmware Structure

At this point remoteproc only supports ELF32 firmware binaries. However,

it is quite expected that other platforms/devices which we'd want to

support with this framework will be based on different binary formats.

When those use cases show up, we will have to decouple the binary format

from the framework core, so we can support several binary formats without

duplicating common code.

When the firmware is parsed, its various segments are loaded to memory

according to the specified device address (might be a physical address

if the remote processor is accessing memory directly).

In addition to the standard ELF segments, most remote processors would

also include a special section which we call "the resource table".

The resource table contains system resources that the remote processor

requires before it should be powered on, such as allocation of physically

contiguous memory, or iommu mapping of certain on-chip peripherals.

Remotecore will only power up the device after all the resource table's

requirement are met.

In addition to system resources, the resource table may also contain

resource entries that publish the existence of supported features

or configurations by the remote processor, such as trace buffers and

supported virtio devices (and their configurations).

Currently the resource table is just an array of:

/**

 * struct fw_resource - describes an entry from the resource section

 * @type: resource type

 * @id: index number of the resource

 * @da: device address of the resource

 * @pa: physical address of the resource

 * @len: size, in bytes, of the resource

 * @flags: properties of the resource, e.g. iommu protection required

 * @reserved: must be 0 atm

 * @name: name of resource

 */

struct fw_resource {

	u32 type;

	u32 id;

	u64 da;

	u64 pa;

	u32 len;

	u32 flags;

	u8 reserved[16];

	u8 name[48];

} __packed;

Some resources entries are mere announcements, where the host is informed

of specific remoteproc configuration. Other entries require the host to

do something (e.g. reserve a requested resource) and possibly also reply

by overwriting a member inside 'struct fw_resource' with info about the

allocated resource.

Different resource entries use different members of this struct,

with different meanings. This is pretty limiting and error-prone,

so the plan is to move to variable-length TLV-based resource entries,

where each resource will begin with a type and length fields, followed by

its own specific structure.

Here are the resource types that are currently being used:

/**

 * enum fw_resource_type - types of resource entries

 *

 * @RSC_CARVEOUT:   request for allocation of a physically contiguous

 *		    memory region.

 * @RSC_DEVMEM:     request to iommu_map a memory-based peripheral.

 * @RSC_TRACE:	    announces the availability of a trace buffer into which

 *		    the remote processor will be writing logs. In this case,

 *		    'da' indicates the device address where logs are written to,

 *		    and 'len' is the size of the trace buffer.

 * @RSC_VRING:	    request for allocation of a virtio vring (address should

 *		    be indicated in 'da', and 'len' should contain the number

 *		    of buffers supported by the vring).

 * @RSC_VIRTIO_DEV: announces support for a virtio device, and serves as

 *		    the virtio header. 'da' contains the virtio device

 *		    features, 'pa' holds the virtio guest features (host

 *		    will write them here after they're negotiated), 'len'

 *		    holds the virtio status, and 'flags' holds the virtio

 *		    device id (currently only VIRTIO_ID_RPMSG is supported).

 */

enum fw_resource_type {

	RSC_CARVEOUT	= 0,

	RSC_DEVMEM	= 1,

	RSC_TRACE	= 2,

	RSC_VRING	= 3,

	RSC_VIRTIO_DEV	= 4,

	RSC_VIRTIO_CFG	= 5,

};

Most of the resource entries share the basic idea of address/length

negotiation with the host: the firmware usually asks for memory

of size 'len' bytes, and the host needs to allocate it and provide

the device/physical address (when relevant) in 'da'/'pa' respectively.

If the firmware is compiled with hard coded device addresses, and

can't handle dynamically allocated 'da' values, then the 'da' field

will contain the expected device addresses (today we actually only support

this scheme, as there aren't yet any use cases for dynamically allocated

device addresses).

We also expect that platform-specific resource entries will show up

at some point. When that happens, we could easily add a new RSC_PLAFORM

type, and hand those resources to the platform-specific rproc driver to handle.

7. Virtio and remoteproc

The firmware should provide remoteproc information about virtio devices

that it supports, and their configurations: a RSC_VIRTIO_DEV resource entry

should specify the virtio device id, and subsequent RSC_VRING resource entries

should indicate the vring size (i.e. how many buffers do they support) and

where should they be mapped (i.e. which device address). Note: the alignment

between the consumer and producer parts of the vring is assumed to be 4096.

At this point we only support a single virtio rpmsg device per remote

processor, but the plan is to remove this limitation. In addition, once we

move to TLV-based resource table, the plan is to have a single RSC_VIRTIO

entry per supported virtio device, which will include the virtio header,

the vrings information and the virtio config space.

Of course, RSC_VIRTIO resource entries are only good enough for static

allocation of virtio devices. Dynamic allocations will also be made possible

using the rpmsg bus (similar to how we already do dynamic allocations of

rpmsg channels; read more about it in rpmsg.txt).

F:	drivers/base/regmap/

F:	include/linux/regmap.h

REMOTE PROCESSOR (REMOTEPROC) SUBSYSTEM

M:	Ohad Ben-Cohen <ohad@wizery.com>

S:	Maintained

F:	drivers/remoteproc/

F:	Documentation/remoteproc.txt

F:	include/linux/remoteproc.txt

RFKILL

M:	Johannes Berg <johannes@sipsolutions.net>

L:	linux-wireless@vger.kernel.org

source "drivers/iommu/Kconfig"

source "drivers/remoteproc/Kconfig"

source "drivers/virt/Kconfig"

source "drivers/devfreq/Kconfig"

obj-$(CONFIG_HWSPINLOCK)	+= hwspinlock/

obj-$(CONFIG_NFC)		+= nfc/

obj-$(CONFIG_IOMMU_SUPPORT)	+= iommu/

obj-$(CONFIG_REMOTEPROC)	+= remoteproc/

# Virtualization drivers

obj-$(CONFIG_VIRT_DRIVERS)	+= virt/

# REMOTEPROC gets selected by whoever wants it

config REMOTEPROC

	tristate