Porting GuideLine
Hardware
The porting of OpenAMP to a new multicore system requires some hardware for inter-processor communication.
Shared memory for
the RPMsg buffer
the VirtIO Rings
the Resource Table (optional if the RPMsg host is not the Linux kernel)
Optional, but recommended, mailboxes acting as a doorbell interrupt on processors to notify of message reception
Memory and interrupt assignments are critical design choices for any port. For a broader overview, refer to System Wide Considerations.
Porting Options
OpenAMP consists of two major components: Remoteproc and RPMsg. These can be ported independently or together.
For the the RPmsg and VirtIO transport, there is a concept of a driver and device role which affect the layers.
The VIRTIO_DEV_DEVICE role and its alias RPMSG_REMOTE refer to the VirtIO instance of the remote processor.
The VIRTIO_DEV_DRIVER role and its alias RPMSG_HOST refer to the VirtIO instance of the main processor.
The support for either of these roles can be enabled or disabled through the main CMake options, as defined in OpenAMP’s cmake/options.cmake
set the VIRTIO_DRIVER_SUPPORT cmake configuration to define WITH_VIRTIO_DRIVER
set the VIRTIO_DEVICE_SUPPORT cmake configuration to define WITH_VIRTIO_DIVICE
option (WITH_VIRTIO_DRIVER "Build with virtio driver (front end) enabled" ON)
option (WITH_VIRTIO_DEVICE "Build with virtio device (back end) enabled" ON)
These definitions affect what is compiled into the main/driver or remote/device image and can be used to limit code size if required. By default both are enabled.
One can think of the implementation options as four features, three of which are part of Remoteproc.
Full Remoteproc as life cycle management, implemented in driver role on main processor
The resource table, implemented as part of the full Remoteproc for main processor or independently in device role on remote processor
Remoteproc VirtIO transport layer for VirtIO protocol
RPMsg, implementing IPC and VirtIO transport layer if not implemented by Remoteproc
The resulting porting implementation approaches include:
On the main processor
Remoteproc as driver role to manage remote processor, with optional associated resource table management
Remoteproc VirtIO to implement the VirtIO driver transport layer
On the remote processor
Optional Remoteproc as device role for the resource table management
Remoteproc VirtIO device role to implement the VirtIO device transport layer
Additional details and examples are provided in the subsequent sections.
Remoteproc Framework
The Remoteproc framework as the name suggests is responsible for remote processor management. For both roles this includes resource table management and/or setup of the VirtIO transport and as the driver role also the remote processor lifecycle management.
Remoteproc Driver Role
In the driver role, Remoteproc includes management of the remote processor lifecycle and resource table management. For instance, if no address is specified for the VirtIO vrings and buffers, the main processor allocates the memory regions and updates the resource table accordingly.
Most OpenAMP Samples and Demos provide examples of lifecycle management for Remoteproc in the driver role.
The load_fw is an example dedicated to demonstrating Remoteproc lifecycle management.
Remoteproc Device Role
Use of Remoteproc framework for the remote or device role is optional and when used is for resource discovery, parsing the resource table to determine the resources allocated. The resources may be statically defined in the table, or dynamically assigned by the main or driver processor and updated in the table.
Dynamic examples are:
ZynqMP R5 machine implementation example.
Management using a static resource examples are:
Remoteproc VirtIO Framework
When using Remoteproc, the VirtIO transport layer is created through it (as compared to through RPMsg’s VirtIO).
Remoteproc VirtIO Driver Role
For the main processor driver role, Remoteproc is used
to obtain and configure resource table information.
create the main processor or driver side VirtIO device
The intent of OpenAMP Remoteproc is to remain compatible with Linux’s Remoteproc.
The Multi Services Example example uses Linux Remoteproc for the driver role implementation. For details refer to RPMsg Multi Services Demo.
Remoteproc VirtIO Device Role
For the remote processor device role, Remoteproc is used
to get resource table information populated by the VirtIO driver
create remote or device side VirtIO device
The Multi Services Example Remote shows Remoteproc VirtIO device creation using rproc_virtio_create_vdev and obtaining the resource table using Zephyr’s rsc_table_get function.
RPMsg Framework
The RPMsg Framework provides the IPC which is based on VirtIO.
When using Remoteproc, the VirtIO used for RPMsg is setup by it for the main processor as the driver role and device role for the remote.
When the implementation does not require Remoteproc, OpenAMP provides VirtIO through the RPMsg Virtio Framework.
The Zephyr OpenAMP IPC Sample is one demonstrating the use of VirtIO through RPMsg framework using static vrings as the transport layer.
The main processor VirtIO device is setup as driver role, using RPMSG_HOST passed to rpmsg_init_vdev.
Refer Zephyr OpenAMP IPC Sample Main for an example driver role implementation.
The remote processor VirtIO device is setup as device role, using RPMSG_REMOTE passed to rpmsg_init_vdev.
Refer Zephyr OpenAMP IPC Sample Remote for an example device role implementation.
Hardware Abstraction
The OpenAMP Framework uses libmetal to provide abstractions that allows for porting of the OpenAMP Framework to various software environments (operating systems and bare metal environments) and machines (processors/platforms). To port OpenAMP for your platform, you will need to:
add your system environment support to libmetal,
optionally implement a platform specific Remoteproc driver.
define your shared memory layout and specify it in a resource table.
Add System/Machine Support in Libmetal
User will need to add system/machine support to lib/system/<SYS>/ directory in libmetal repository. OpenAMP requires the following libmetal primitives:
alloc
Memory allocation and memory free as defined in alloc.h, which call the functions of equivalent names with double underscore which are the ported functions (__metal_allocate_memory, __metal_free_memory).
-
static inline void *metal_allocate_memory(unsigned int size)
allocate requested memory size return a pointer to the allocated memory
- Parameters:
size – [in] size in byte of requested memory
- Returns:
memory pointer, or 0 if it failed to allocate
-
static inline void metal_free_memory(void *ptr)
free the memory previously allocated
- Parameters:
ptr – [in] pointer to memory
io
Memory mapping as used by io.h, in order to access vrings and carved out memory.
metal_sys_io_mem_map and metal_machine_io_mem_map functions.
mutex
Mutex functions as used by mutex.h which call the functions of equivalent names with double underscore which are the ported functions (e.g. __metal_mutex_init).
-
static inline void metal_mutex_init(metal_mutex_t *mutex)
Initialize a libmetal mutex.
- Parameters:
mutex – [in] Mutex to initialize.
-
static inline void metal_mutex_deinit(metal_mutex_t *mutex)
Deinitialize a libmetal mutex.
- Parameters:
mutex – [in] Mutex to deinitialize.
-
static inline int metal_mutex_try_acquire(metal_mutex_t *mutex)
Try to acquire a mutex.
- Parameters:
mutex – [in] Mutex to mutex.
- Returns:
0 on failure to acquire, non-zero on success.
-
static inline void metal_mutex_acquire(metal_mutex_t *mutex)
Acquire a mutex.
- Parameters:
mutex – [in] Mutex to mutex.
-
static inline void metal_mutex_release(metal_mutex_t *mutex)
Release a previously acquired mutex.
See also
- Parameters:
mutex – [in] Mutex to mutex.
-
static inline int metal_mutex_is_acquired(metal_mutex_t *mutex)
Checked if a mutex has been acquired.
See also
- Parameters:
mutex – [in] mutex to check.
sleep
At the moment, OpenAMP only requires microseconds sleep as when OpenAMP fails to get a buffer to send messages, it will call this function to sleep and then try again.
The __metal_sleep_usec to be implemented by the port is wrapped in sleep.h.
-
static inline int metal_sleep_usec(unsigned int usec)
delay in microseconds delay the next execution in the calling thread fo usec microseconds.
- Parameters:
usec – [in] microsecond intervals
- Returns:
0 on success, non-zero for failures
init
Init is ported for libmetal initialization for sys.h.
metal_sys_init and metal_sys_finish functions.
Please refer to lib/system/generic/ when adding RTOS support to libmetal.
libmetal uses C11/C++11 stdatomics interface for atomic operations, if you use a different compiler to GNU gcc, you may need to implement the atomic operations defined in lib/compiler/gcc/atomic.h.
Platform Specific Remoteproc Driver
An OpenAMP port could need a platform specific Remoteproc driver to use Remoteproc life cycle management (LCM) APIs. The Remoteproc driver platform specific functions are defined in lib/include/openamp/remoteproc.h and provided through the remoteproc_ops data structure.
The Remoteproc LCM APIs use these platform specific implementation of init, remove, mmap, handle_rsc, config, start, stop, shutdown and notify. These functions are passed to Remoteproc via the remoteproc_ops structure which contains function pointers to each.
-
struct remoteproc_ops
Remoteproc operations to manage a remoteproc instance.
Remoteproc operations need to be implemented by each remoteproc driver
Public Members
-
struct remoteproc *(*init)(struct remoteproc *rproc, const struct remoteproc_ops *ops, void *arg)
Initialize the remoteproc instance.
-
void (*remove)(struct remoteproc *rproc)
Remove the remoteproc instance.
-
void *(*mmap)(struct remoteproc *rproc, metal_phys_addr_t *pa, metal_phys_addr_t *da, size_t size, unsigned int attribute, struct metal_io_region **io)
Memory map the memory with physical address or destination address as input.
-
int (*handle_rsc)(struct remoteproc *rproc, void *rsc, size_t len)
Handle the vendor specific resource.
-
int (*config)(struct remoteproc *rproc, void *data)
Configure the remoteproc to make it ready to load and run the executable.
-
int (*start)(struct remoteproc *rproc)
Kick the remoteproc to run the application.
-
int (*stop)(struct remoteproc *rproc)
Stop the remoteproc from running the application, the resource such as memory may not be off.
-
int (*shutdown)(struct remoteproc *rproc)
Shutdown the remoteproc and release its resources.
-
int (*notify)(struct remoteproc *rproc, uint32_t id)
Notify the remote.
-
struct remoteproc_mem *(*get_mem)(struct remoteproc *rproc, const char *name, metal_phys_addr_t pa, metal_phys_addr_t da, void *va, size_t size, struct remoteproc_mem *buf)
Get remoteproc memory I/O region by either name, virtual address, physical address or device address.
- Param rproc:
Pointer to remoteproc instance
- Param name:
Memory name
- Param pa:
Physical address
- Param da:
Device address
- Param va:
Virtual address
- Param size:
Memory size
- Param buf:
Pointer to remoteproc_mem struct object to store result
- Return:
remoteproc memory pointed by buf if success, otherwise NULL
-
struct remoteproc *(*init)(struct remoteproc *rproc, const struct remoteproc_ops *ops, void *arg)
The remoteproc_init API receives this structure, and its function pointers, which are then used by the other APIs.
For example Cortex-R5 remoteproc_ops are defined in zynqmp_r5_a53_rproc.c.
Use Remoteproc to Manage Remote Processor
With the Remoteproc driver functions required by the framework ported, the user can call the Remoteproc APIs to run an application on a remote processor, as described in the Remote User APIs section of the Remoteproc design.
The following code snippets from the Load FW System Reference Example demonstrate the use of the Remote User APIs.
Remoteproc Init
From platform_info.c
static struct remoteproc * platform_create_proc(unsigned int cpu_id)
{
const struct remoteproc_ops *ops;
struct remoteproc * rproc;
if (NODE_RPU_0 <= LOAD_FW_TARGET && LOAD_FW_TARGET <= NODE_RPU_1)
ops = &zynqmp_rpu_rproc_ops;
else if (NODE_APU_0 <= LOAD_FW_TARGET && LOAD_FW_TARGET <= NODE_APU_1)
ops = &zynqmp_apu_rproc_ops;
else
return NULL;
rproc = remoteproc_init(&rproc_inst, ops, &cpu_id);
if (!rproc)
return NULL;
return &rproc_inst;
}
Lifecycle APIs
From load_fw.c
int load_exectuable_block(struct remoteproc *rproc,
const struct image_store_ops *store_ops, void *store,
const char *img_path)
{
int ret;
(void)img_path;
if (rproc == NULL)
return -EINVAL;
/* Configure remoteproc to get ready to load executable */
remoteproc_config(rproc, NULL);
/* Load remoteproc executable */
LPRINTF("Start to load executable with remoteproc_load() \r\n");
ret = remoteproc_load(rproc, NULL, store, store_ops, NULL);
if (ret) {
LPRINTF("failed to load firmware\r\n");
return ret;
}
/* Start the processor */
ret = remoteproc_start(rproc);
if (ret) {
LPRINTF("failed to start processor\r\n");
return ret;
}
LPRINTF("successfully started the processor\r\n");
#ifndef RPU_BOOT_LINUX
/* ... */
asm volatile("wfi");
LPRINTF("going to stop the processor\r\n");
remoteproc_stop(rproc);
/* application may want to do some cleanup before shutdown */
LPRINTF("going to shutdown the processor\r\n");
remoteproc_shutdown(rproc);
#endif /* RPU_BOOT_LINUX */
return 0;
}
#ifndef RPU_BOOT_LINUX
Platform Specific RPMsg
In OpenAMP, RPMsg uses VirtIO to manage shared buffers. The OpenAMP library provides a Remoteproc VirtIO backend implementation, and a VirtIO and RPMsg implementation.
You can also implement your own VirtIO backend using the OpenAMP VirtIO and RPMsg components provided by OpenAMP.
If you choose to create your own backend, you can use the Remoteproc VirtIO backend as a reference.
For an example of setting up Remoteproc Virtio for a remote device refer to zynqmp_r5 platform_info.c.