Please note as of Wednesday, August 15th, 2018 this wiki has been set to read only. If you are a TI Employee and require Edit ability please contact x0211426 from the company directory.

DM81xx AM38xx PCI Express Endpoint Driver User Guide

From Texas Instruments Wiki
Jump to: navigation, search
TIBanner.png
DM81xx AM38xx PCI Express Endpoint Driver User Guide
Hemant Pedanekar


Introduction

This document is applicable to DM816x/AM389x and DM814x/AM387x family of devices referred hereafter as DM816x and DM814x respectively. Descriptions common across both the device families use the term DM81xx. The code snippets and examples in this document will use the terms TI81XX/ti81xx.

DM81xx devices have PCI Express hardware module which can either be configured to act as a Root Complex or a PCIe Endpoint. This document caters to the Endpoint mode of operation and describes the Driver needed to configure and operate on DM81xx PCI Express device as Endpoint.

This driver is referred as EP Driver and has to be loaded on DM81xx EP running Linux kernel from PSP package. It is expected that a userspace application would use this driver to configure the PCIe module and do transfers with peers.

NOTE: This document is applicable to PSP GIT kernel available from Arago git post PSP 04.00.01.13 release for DM816x and 04.01.00.06 release for DM814x or kernel from release 04.00.02.14 and 04.01.00.07 or later for respective platforms. Jump to Package Contents and Download section for details about getting all associated components.

Scope of the Document

At present, this document covers following areas:

  • Details about the EP Driver
  • Summary of applications and kernel modules provided as sample
  • Build configuration and instructions
  • Execution

Following items will be incorporated in future versions of this document:

  • Completion of driver and sample code details - for now, please refer the respective source code.
  • More details about setups
  • Design and expected/recommended use cases of the EP driver
  • EDMA usage details
  • Throughput data

Assumptions and Conventions

Following points/prerequisites are assumed throughout this document:

  1. If using PSP release 04.00.01.13, the PSP kernel tarball package from Arago git is extracted and build environment is set up as described in PSP User Guide associated with 04.00.01.13 release package.
  2. Else. if using PSP release 04.00.02.14 (DM816x) or 04.01.00.07 (DM814x), the kernel tarball from release package is extracted and build environment is set up as described in User Guides packaged with respective releases.
  3. All kernel modules/driver source and header file paths are relative to the kernel source directory where the package is extracted, except otherwise stated.
  4. References to kernel modules provided as sample assume that the patches for the same are applied to kernel extacted in #1 above.
  5. References to sample applications assume the application patch is applied in an empty directory created by the user on the build host.
  6. All application files are referred with path relative to the above directory.
  7. The term "DM81xx EP" refers to local DM816x or DM814x EP where the EP driver instance runs.

Terminology

  • PCIe - PCI Express
  • EP - PCIe Endpoint
  • RC - PCIe Root Complex

Package Contents and Download

The EP Driver package consists of main driver along with sample test applications and supplementary drivers (custom samples).

  • Out of these, the EP driver is available in kernel source for DM816x release 04.00.02,14 or DM814x release 04.01.00.07 and onwards. Users of earlier release may download the kernel tarball including first version of EP driver directly from DM81xx Arago tree on ti81xx-master branch form here.
  • The sample applications for RC side as well as EP side are in attached tarball. Click here to download. Further discussions assume that this file is downloaded on build host as epdrv_sample_apps.tar.gz.
  • EDMA sample driver for DM81xx EP side is provided as patch file here to be applied to kernel snapshot from ti81xx-master branch
  • Sample driver for DM81xx RC side is provide as a patch on the same kernel snapshot (after applying EDMA patch).

Supported Setups

Currently following setups are tested:

  • DM814x RC <-> DM816x EP
  • DM816x RC <-> DM816x EP
  • DM816x RC <-> DM814x EP

For details about doing these setups, please refer following documents:
DM81xx PCIe RC Driver User Guide
DM81xx PCIe Boot Driver User Guide

Note: There may be some porting effort and code changes required when non DM81xx RC is involved.

Not Supported / Known Issues

  • Following warning is observed while building kernel and RC module for DM81xx after applying RC driver patch
warning: 'virt_to_bus' is deprecated (declared at /db/psp_git/users/a0393588/temp/linux-omap-2.6/arch/arm/include/asm/memory.h:222
  • This doesn't have any impact on execution and can be ignored
  • Must follow the sequence of loading various components as described in steps in Execution section below, other sequences are not yet validated and may not work
  • At present, only polling option is being used on EP and RC side. MSI support is currently not tested.
  • Running RC application on EP and vice verse may lead to issues and should be avoided.
  • ioctl values for various drivers currently overlap with existing drivers in kernel. Particularly, debugging tools such as strace will treat the ioctls for the drivers as one of the existing ioctl already supported in kernel: eg., TI81XX_EDMA_SET_BUF_INFO is reported as SNDCTL_DSP_CHANNELS

EP Driver Summary

The EP driver controls the DM816x/DM814x Endpoint device it is running on.

It creates a device node named "ti81xx_pcie_ep" under "/dev" to enable application to open and configure the device through character device interface.

Driver Calls

open() Application have to first open the device to get the file descriptor for calling subsequent functions.

ioctl() Various IOCTLs can be used by the application to access the PCIe hardware on local DM81xx EP.

mmap() MMAP interface can be used by the application using EP driver to map the reserved memory for PCIe as well as the h/w window of 256MB starting from 0x20000000.

close() Close the driver and cleanup outbound mappings.

IOCTLs Supported

TI81XX_SET_INBOUND Set inbound translation for a given BAR on DM81xx EP.

TI81XX_SET_OUTBOUND_SIZE Set the size for outbound window. The same size is applicable to all windows configured.

TI81XX_SET_OUTBOUND Create outbound window for specified address. This address has to be specified by the application such that a valid PCIe address is generated over the PCIe link.

TI81XX_ACCESS_REGS Read of write the specified PCIe module register - this could be from PCIe module's application space or local configuration space.

TI81XX_SEND_MSI Send MSI targeted to specified address. Presently not supported.

TI81XX_GET_PCIE_MEM_INFO Application can use this to query the size and physical address of the reserved memory on DM81xx EP for PCIe transfers. This memory space can then be divided by the application across different BARs (for inbound mapping) and to perform CPU/EDMA read/write transfers from EP. To access this memory in userspace, the application can use mmap facility provided by the EP driver (for CPU transfers) or by EDMA sample driver for EDMA trnasfers.

TI81XX_GET_OUTBOUND_STATUS Return a 32-bit mask inidicating available outbound windows. A value '1' for specific bit position indicates corresponding window in free and not enabled for outbound.

TI81XX_CLR_OUTBOUND_MAP Clear outbound mapping for specified address. Corresponding windo will be marked as free.

TI81XX_GET_INTR_CNTR Get the current count of received MSI interrupts. This count is cumulative from the initialization of opening of driver.

TI81XX_GET_BAR_INFO Returns the information associated with particular BAR such as address assigned to it (by RC) and size of the BAR.

EP Driver Build Configuration

The default DM81xx kernel build doesn't enable EP driver. Follow steps below to enable and build EP driver:

NOTE: Currently, EP driver can be enabled only if PCI Bus Support is disabled in kernel configuration. This is done to avoid any conflict when mistakenly running EP driver on a Root Complex.

Using the correct kernel

  • Ensure that the kernel from downloaded from ti81xx-master branch is used and extracted on build host.

Ensuring default kernel configuration

  • Configure the kernel with default options. E.g., for DM816x EVM,
make ARCH=arm ti8168_evm_defconfig
  • When building for DM814x EVM as EP,
make ARCH=arm ti8148_evm_defconfig

Enter kernel configuration menu

make ARCH=arm menuconfig

You will be presented with menu such as below:

   General setup  --->
   [*] Enable loadable module support  --->
   -*- Enable the block layer  ---> 
      System Type  --->
       Bus support  --->
   ...
   ...
   Device Drivers  --->

Disable PCI Bus Support

  • Remove PCI bus support, which will in turn remove PCIe Root Complex Driver.
  • Scroll till "Bus Support" listed above using DOWN arrow and press SPACE to enter the option menu:
   [*] PCI support
   [*] Message Signaled Interrupts (MSI and MSI-X)
   [ ] PCI Debugging
   < > PCI Stub driver
   [ ] PCI IOV support
   < > PCCard (PCMCIA/CardBus) support  --->

Disable PCI support by pressing 'n' key on "PCI support" option. This will automatically disable other dependent configurations.

   [ ] PCI support
   < > PCCard (PCMCIA/CardBus) support  --->

Now press RIGH arrow to highlight "Exit" and press ENTER to return to initial menu.

Enable EP Driver
Scroll with DOWN arrow key till "Device Drivers" option and press SPACE. This will present following menu:

   Generic Driver Options  --->
   < > Connector - unified userspace <-> kernelspace linker  --->
   ...
   ...
       Character devices  --->

Scroll till "Character devices" and press SPACE, which will show a menu similar to following:

   -*- Virtual terminal
   ...
   ...
   < > TI81XX PCIe Endpoint Driver (NEW)

Using DOWN arrow key, go till PCIe Endpoint Driver option as shown above.

Press 'y' to build the driver into kernel

   ...
   <*> TI81XX PCIe Endpoint Driver

or 'm' to build as loadable module.

   ...
   <M> TI81XX PCIe Endpoint Driver

NOTE: At present, only modular build is supported/validated so it is recommended to build the driver as module.

Use RIGHT arrow key to highlight "Exit" and press ENTER successively till you are prompted to save the kernel configuration. Select "Yes" and press enter to exit the configuration menu.

Building EP Driver

Once the EP driver is enabled in kernel build configuration as described in earlier section, you can build it as follows:

Building statically with kernel
In this case, rebuild of kernel is required and driver will be built into kernel

make ARCH=arm uImage

Building as loadable module

NOTE: Even when loading the EP driver as module, you need to build the kernel at least once and make sure that it is used for EP before loading the module otherwise the EP driver may not get loaded successfully. In addition, to enable reserving memory at boot time and for successful EP driver build, PCIe EP driver must be selected in build configuration before building the kernel.

Build the kernel wil PCI support disabled and EP driver selected to be built as module (refer earlier section for configuration):

make ARCH=arm uImage

Build kernel modules

make ARCH=arm modules

This will build EP driver module located inside drivers/char as:

drivers/char/ti81xx_pcie_epdrv.ko

Resource Reservation

The EP drive expects a contiguous block of memory reserved at boot time to facilitate applications to do mmap when doing data transfer using PCIe. The desired size should vary depending upon the application requirement.

By default there is no memory reserved for EP driver unless a kernel commandline argument "pcie_mem" with non-zero value specifying size of the memory to be reserved is passed.

NOTE1: PCIe EP driver must be included in kernel configuration to be able to reserve memory for PCIe.

NOTE2: The EP driver doesn't require any memory to be reserved for its own functionality, but this memory is required to provide mmap regions mapping to PCIe BARs (inbound transfers) and/or EDMA regions (outbound transfers) to allow application to perform PCIe communication and transfers.

An example for reserving 8 MB of RAM for PCIe EP, append following to kernel bootargs:

pcie_mem=8M

Note that, though the user can specify the size of memory to be reserved for PCIe, the start address of this chunk is not guaranteed to be same across different booting of the kernel and may vary due to parameters such as different boot arguments, kernel configuration and changes in kernel. The EP drivers provides an IOCTL to enable application (using EP driver) to query the physical base address and size of the reserved PCIe memory block.

WARNING: If PCIe memory is not reserved during boot time, the EP driver would still load successfully but the application on EP might fail if it expects reserved RAM available to transfer data using mmap or DMA. To ensure that the driver had reserved memory available, the application must use TI81XX_GET_PCIE_MEM_INFO ioctl to check if reserved memory size fits application's requirement.

Inbound Mapping

EP driver, on its own, doesn't create inbound mapping for any BAR. It is the application's responsibility to allot a contiguous region from PCIe reserved memory to create inbound mapping using IOCTL TI81XX_SET_INBOUND.

Note: It is recommended but not forced on the application to use the region from PCIe reserved memory to create inbound mapping. It may as well use a different mechanism to reserve memory (outside of EP driver) and pass the physical address etc to EP driver for setting up inbound translation.

Outbound Mapping

Unlike inbound, outbound mapping doesn't directly depend on the PCIe memory reserved, since it is internal masters (Cortex A8 CPU or EDMA) who need to be aware of (or provided with) memory to be used as source/destination for outbound and is independent of the PCIe and EP driver.

In this case also, a contiguous space for transfer is needed and it may be convenient to use a part of memory already reserved for PCIe. For example, in case the memory reserved is 8 MB, designate buffers in the lower 4 MB to carry out EDMA reads/writes to/drom.

NOTE: EP driver, on its own, doesn't provide any EDMA transfer functionality and it is the application's responsibility to set up outbound transfer parameters such as outbound mapping (using IOCTL tba), determining source/destination address for transfer, etc. before proceeding to use EDMA driver.

EP Driver Files

drivers/char/ti81xx_pcie_epdrv.h
drivers/char/ti81xx_pcie_epdrv.c

Kernel Files/Functions Added/Updated

  • arch/arm/plat-omap/devices.c
    • Adds mechanism to reserve RAM for PCIe EP operations at boot time with size

specified by "pcie_mem" boot argument.

  • arch/arm/plat-omap/include/plat/common.h
  • arch/arm/plat-omap/common.c
    • Reserve memory during boot

Sample Components

The various sample components to utilize the EP driver on DM81xx EP are attached to this page.

These include:

  • EP side: Sample kernel module for EDMA
  • RC side: Sample kernel module for DM81xx RC which provides RC side application with access over PCI
  • RC & EP side: Sample Applications

The details are coverrd in individual sections below.

EDMA Kernel Module

This module is provided as a sample driver to access EDMA functionality from user spoace. The main purpose of this driver is to enable the EP side application to initiate EDMA transfers to the peer over PCIe. Note that since this is a sample driver, it is not integrated into kernel and needs to be applied as patch to the kernel source from release.

Driver Files

drivers/char/ti81xx_edma.h drivers/char/ti81xx_edma.c

Driver Calls

Please refer driver files and EP application code for reference.

IOCTLs supported

Please refer driver files and EP application code for reference.

Building The Module

Apply the attached patch 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch by copying it to kernel source directory. Ensure that the kernel is already configured for DM81xx EP as mentioned earlier.

The download link mentioned earlier has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch.gz
# patch -p 1 -i 0001-char-ti81xx-Add-EDMA-driver-to-provide-minimal-acces.patch

NOTE: The above step directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module.

Build module with following command

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_edma.ko

RC Kernel Module

This sample driver needs to be applied to the RC kernel. This means it must have PCI bus support with MSI enabled.

Note: At present, only DM816x/DM814x RC is supported. Also, to apply this patch cleanly, the kernel extracted earlier (for building EP driver) needs to be used.

Driver Files

drivers/char/ti81xx_pcie_rcdrv.h drivers/char/ti81xx_pcie_rcdrv.c

Driver Calls

Please refer driver files and RC application code for reference.

IOCTLs supported

Please refer driver files and RC application code for reference.

Building The Module

Apply the attached patch 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch by copying it to kernel source directory. Ensure that the patch for EDMA sample module is already applied (as mentioned above), this will avoid conflicts.

The download link mentioned earlier has the patch in gzipped format so it needs to be extracted to get the patch file. You can then use 'patch' utility to apply the patch as:

# gunzip 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch.gz
# patch -p 1 -i 0002-char-ti81xx-Add-driver-for-controlling-TI81XX-EP-fro.patch

NOTE: The above step directly modifies the drivers/char/Makefile to build this sample driver always as loadable kernel module when PCI MSI (CONFIG_PCI_MSI) support is enabled.

Ensuring default kernel configuration

  • Configure the kernel with default options. E.g., for DM816x EVM,
make ARCH=arm ti8168_evm_defconfig
  • In case using DM814x EVM as RC, use,
make ARCH=arm ti8148_evm_defconfig

Build the kernel for RC

make ARCH=arm uImage

Build RC module

make ARCH=arm modules

The module will be built as

drivers/char/ti81xx_pcie_rcdrv.ko

Sample Applications

The attached tarball epdrv_sample_apps.tar.gz contains RC and EP side sample applications which use various kernel modules described above to communicate and perform data trnasfers between them across PCIe.

Currently the EP application requires minimum 8MB of memory reserved for PCIe transfers, thus, the kernel command line must be passed pcie_mem=8M at least. If the memory reserved is less, then the application will exit.

Files

./Makefile
./lib
./lib/Makefile
./lib/ti81xx_ep_lib.c
./lib/ti81xx_mgmt_lib.c
./lib/ti81xx_pci_info.c
./lib/ti81xx_trans.c
./include
./include/debug_msg.h
./include/ti81xx_ep_lib.h
./include/ti81xx_mgmt_lib.h
./include/ti81xx_pci_info.h
./include/ti81xx_trans.h
./application
./application/makefile
./application/ti81xx_ep_app.c
./application/ti81xx_rc_app.c

Building Applications

Create an empty directory and extract the attached tarball epdrv_sample_apps.tar.gz into that directory.

E.g., assuming the directory created is named 'apps' and eprv_sample_apps.tar.gz is copied to it,

# cd apps
# tar -xzvf eprv_sample_apps.tar.gz

Build the applications for DM816x/DM814x RC and DM816x/DM814x EP using following command

# make KERNSRC=<absolute-path-to-DM81xx-kernel-source>

Remember to pass correct path to kernel source where the kernel tarball mentioned earlier is extracted.

The above command will build EP and RC applications at following paths respectively:

application/ep_app
application/rc_app

To clean the built files, you can use

# make clean

Note: Currently only default mode of applications is supported (DEMO mode). The other 2 modes - throughput and sanity mode are not supported.

Execution

Following sequence needs to be executed to run the demo with above components (commands with 'EP#' to be executed on EP size while those with 'RC#' to be executed on RC side):

  • Ensure the desire PCIe setup is done - in this case, we assume DM816x <-> DM816x setup.
  • Power on DM816x EP
  • Power on DM816x RC
  • Boot EP from RC (refer Boot Driver User Guide)
  • Make sure that at least 8 MB of PCIe memory is reserved by appending pcie_mem=8M (or more) to the 'bootargs' to the EP kernel. You may also need to ensure that the 'mem' argument passed to kernel ensures sufficient memory is available.
  • Ensure that EP Driver, EDMA driver and EP sample application are present in a directory on EP filesystem.
  • Once shell is available on EP side, navigate to the directory containing the above files. Load EP Driver and EDMA sample driver:
EP# insmod ti81xx_pcie_epdrv.ko
EP# insmod ti81xx_edma.ko
  • After successful load of these modules, run EP application:
EP# ./ep_app
  • You may need to add executable permission to the application before above step:
EP# chmod a+x ep_app
  • The EP application will not wait for communication with peer.
  • On DM816x RC side, make sure that the RC driver and application are present in working directory.
  • Navigate to that directory and load the RC driver
RC# insmod ti81xx_pcie_rcdrv.ko
  • Run RC application (again, you may need to run chmod a+x rc_app before this step)
RC# ./rc_app

Since these applications are built with DEMO mode, they will start communicating each other and do data transfer in either directions using CPU (read/write from RC) and EDMA (read/write from EP).

During execution of the applications, on RC and EP consoles, you would see chunks with 'C', 'D', 'L' characters. The meaning is explained below:

on RC, data represented by 'C' is sent by memcpy from EP
on RC, data represented by 'D' is sent by EDMA from EP
on EP, data represented by 'C' is sent by memcpy from RC
on EP, data represented by 'L' is data that has been read by EDMA from a buffer on RC populated by application on RC.
  • To exit the DEMO, terminate RC and EP apps - this can be done by pressing Ctrl + C combination at shell on both RC and EP. Note that there are no log files generated for DEMO mode.