Processor SDK RTOS PCIe

From Texas Instruments Wiki
Jump to: navigation, search


RTOS Software Developer Guide PCIe



Overview

User Interface

Application

Debug


Overview

PCIe module supports dual operation mode: End Point (EP or Type0) or Root Complex (RC or Type1). This driver focuses on EP mode but it also provides access to some basic RC configuration/functionality. For RC this is the lowest level; additional software is needed to perform generic enumeration of third party devices.

The PCIe subsystem has two address spaces. The first (Address Space 0) is dedicated for local application registers, local configuration accesses and remote configuration accesses. The second (Address Space 1) is dedicated for data transfer. This PCIe driver focuses on configuration of the interface and sending/receiving interrupts via Address Space 0. Data is transferred outside the scope of the LLD using CPU or EDMA through the data area.

There are two revisions of the pcie hardware. The first, v0, in KeyStone devices (C66x, K2x). The second, v1, is in AM57xx devices. The LLD abstacts the configuration of standards-based PCIe registers (Type 0, Type 1 and Capabilities registers) so same user code can be used on both device types. However, there are different interfaces for the registers not based on PCIe standards (port logic and ti-conf which generally covers interrupts and address translation). That portion of user code needs to differ between C66x/K2x and AM57xx devices.

The example includes configuration of one SoC as EP and a second as RC. It then performs a simple CPU-based memory transfer (EDMA can be used via the same addresses used by the CPU), and interrupt generation (from EP) and reception (to RC). It also shows differences in user code required to support both C66x/K2x and AM57xx devices.

Differences in Operation between C66x/K2x and AM57xx devices

C66x/K2x and AM57xx are functionally identical, except that interrupts are handled by example and lld only on AM57xx.

Modes of Operation

The LLD is intended to bring up the PCIe peripheral, open memory mappings, and send/receive interrupts.

Root Complex (RC)

The PCIe peripheral can be used as a root complex. One or more other endpoints can be connected (more than one requires a PCIe switch on the board). The LLD configures the peripheral in RC mode. It doesn't attempt to emulate the full enumeration capability of a Linux driver. Instead the user will need to supply code specific to each endpoint they intend to support.

Endpoint (EP)

The PCIe peripheral can be used as an endpoint. This is the more intended usecase for the LLD. Once the link is initialized, the LLD can provide data addresses and send interrupts to the RC.

Interrupts

The example for AM57XX provides code to send interrupts from an endpoint, and the LLD/example together contain code to receive/demux the interrupts (both MSI and Legacy) on an RC.

Driver Configuration

Board-specific configuration

PCIe's board specific configuration is in the example in PDK_INSTALL_DIR/packages/ti/drv/pcie/example/sample/am57x/src/pcie_sample_board.c. Calling sequence is in example and repeated below.

PCIe configuration structure

The pcie_soc.c binds the driver to the hardware on the board. It is passed into the driver via the call to Pcie_init().

API Call Flow

The API call flow is covered in pcie_sample.c.

The overall components are:

  1. Initialize the driver
  2. Initialize the SERDES and Power the peripheral (see example code for order for each device)
  3. Configure RC or EP symmetrically to talk to another similar device
  4. Perform data IO and interrupt.

Examples

Name
Description
EVM Configuration
Expected Results
PCIE_idkAM57[12]x*ExampleProject

2-device PCIe connection

IMPORTANT: Cable must be MODIFIED in order to avoid damaging the clock drivers in the SoC!


Connect two like AM57xx  IDK EVMs J22 using a MODIFIED male-male crossover PCIe x1 or x4 cable and optional straight-through extender. For AM572x we used a one-lane cross cable; for AM571x we used a 4-lane cross cable.

Projects available for C66, A15, and M4.

AM572x or AM571x EVM : One board is EP/other is RC; link is established and data/interrupts exchanged. All printed output goes to the serial console.

PCIE_evmK2G*ExampleProject

2-device PCIe connection

IMPORTANT: Cable must be MODIFIED in order to avoid damaging the clock drivers in the SoC!


Connect two K2G  EVMs J5 using a MODIFIED male-male crossover PCIe x1 cable and optional straight-through extender. We used a one-lane cross cable.

Ensure that jumper J15 is positioned to the left hand two pins as viewed with "J15" silkscreen right side up (pin 0 isn't clearly marked). In the right hand position, the PCIe PLL will NOT get clock.

Projects available for C66 and A15.

K2G EVM : One board is EP/other is RC; link is established and data is exchanged. All printed output goes to serial console.

PCIE_idkAM571x_*armEdmaPktExampleProject
(Available starting in Processor-SDK 3.1)

Packet Exchange over PCIe Benchmark

IMPORTANT: Cable must be MODIFIED in order to avoid damaging the clock drivers in the SoC!


Connect two like >=REV1.3A AM571x  IDK EVMs J22 using a MODIFIED male-male crossover PCIe x4 cable and optional straight-through extender. We used a 4-lane cross cable.

Projects available for A15 only. >=REV1.3A EVM required (for x4 PCIe connectors).

AM571X >=REV1.3A IDK : One board is EP/other is RC; link is established and data is exchanged. Produces same output as standard ExampleProject, except EP also prints packet exchange benchmark results.

Remaining PCIE_*ExampleProject

2-device PCIe connection

Connect two like C66x/K2x (except K2G, see previous row) EVMs using an AMC breakout card. For K2L, it is necessary to configure the mux via the BMC console with "interface_muxsel pcie" command.

Projects available for A15 and/or C66 as present in each device.

6678, 6657, K2E, K2H, K2L : One board is EP/other is RC; link is established and data exchanged. For A15 projects, all printed output goes to serial console. For C66 projects, all printed output goes to CCS console.

Quick setup of xds100 for two EVMs

  1. create new target configuration using XDS100v2 and AM572x (or AM571x) from the basic tab.
  2. Select Advanced tab.
  3. Highlight the XDS100v2, and click the "new" button and add second XDS100v2.
  4. Highlight the newly added XDS100v2, click the "add" button and select a second Am572x.
  5. open command prompt, and run ti\ccs_base\common\uscif\xds100serial to get your serial numbers
  6. Highlight first XDS100v2, select "Debug Probe Selection" to "Select by Serial number" and enter one of the 2 serial numbers
  7. Repeat second XDS100v2, setting to to the second serial number.

General instructions for configuring multiple EVMs with any emulator type are available in Multi-Emulator_Debug_with_CCS

Detailed instructions to run example

Ensure 2 Like EVMs are connected with a x1 PCIe male/male cross cable (for AM5XX) or a breakout card (for C667x, C665x, K2x)

Build project(s) appropriate for your EVM. Projects for A15 and C66 are provided based on core types available on each device.

Load via jtag either the ARM or DSP projects (but don't mix and match) onto the first arm or dsp core of each the 2 EVMs. Same .out file supports both RC and EP. Use an "expressions" window to set PcieModeGbl to PCIE_RC_MODE on one EVM (it makes that EVM RC). Leave the second EVM alone (pcie_EP_MODE). Run the loaded cores. See table above to determine whether output is expected on serial console or CCS console.

Sample example output

Note that output will vary slightly based on device type. The following is from A57XX. The output from the RC and EP are interleaved since this is run from a 2*XDS1000 double config as described in #Quick setup of xds100 for two EVMs

**********************************************
*             PCIe Test Start                *
*                RC mode                     *
**********************************************

Version #: 0x02020003; string PCIE LLD Revision: 02.02.00.03:Dec 24 2015:17:38:37

PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
**********************************************
*             PCIe Test Start                *
*                EP mode                     *
**********************************************

Version #: 0x02020003; string PCIE LLD Revision: 02.02.00.03:Dec 24 2015:17:38:37

PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
Link is up.
Link is up.
End Point received data.
End Point sent data to Root Complex, completing the loopback.
EP sending interrupts to RC
Root Complex received data.
RC waiting for 10 of each of 2 types of interrupts
RC got all 20 interrupts
Test passed.
End of Test.



Debug FAQ

  1. If example fails to get link up
    1. Confirm that male/male cross cable or breakout board is correctly connected.
    2. If running from ARM cores, confirm that immediately after reset/reload that both devices have PcieModeGbl=PCIE_EP_MODE. If the PCIE_RC_MODE seems to survive reset/reload, it seems to mean watch window failed to refresh. Click the "Refresh" button for the watch window and it should flip back to EP, where you can reset it to RC. Simply running will cause both sides to run as EP, which leads to test failure.
    3. Confirm that one side of the example has PcieModeGbl=PCIE_RC_MODE and the other is PCIE_EP_MODE.
      1. Note that when changing to RC you must click somewhere outside the expression value to make the modification for RC to "take effect". Simply pressing F8 after modifying the value will run without actually modifying the variable! The modification will be done when the ARM or DSP is stopped, so everything looks right, except that the log will show "PCIe test start EP mode" twice instead of "PCIe test start EP mode" once and "PCIe test start RC mode" once.

Additional References

Additional documentation can be found in:

Document Location
Hardware Peripheral Users Guide
  • C66x/K2x: User Guide
  • AM57XX: TRM Chapter 24.9 titled "PCIe Controller"
API Reference Manual

$(TI_PDK_INSTALL_DIR)\packages\ti\drv\pcie\docs\doxygen\html\index.html

Release Notes $(TI_PDK_INSTALL_DIR)\packages\ti\drv\pcie\docs\ReleaseNotes_PCIE_LLD.pdf