SMP Debug

From Texas Instruments Wiki
Jump to: navigation, search

An SMP system needs to be debugged as a system, focusing on the logical control flow regardless of where it is executed, instead of the typical core-centric approach of CCS. Thus, execution (steps/runs) should be simultaneous on all cores and breakpoints should halt all cores regardless of which core executed the code in question. This wiki explains how to use the CCS features that provide this type of behaviour.

Quickstart Guide

First create a sync group that groups the cores in the SMP system, and then force those cores to always use hardware breakpoints

  1. Multi-select each core in the SMP system, right click and select "Sync Group"
  2. Select Tools->Debugger Options->Misc/Other Options
  3. Scroll to near the bottom and unselect “Allow software breakpoints to be used”
  4. Click "Remember My Settings"
  5. Repeat steps 3 and 4 for each core in the SMP system

After this, any run/step/halt issued to any of those cores will run/step/halt all of them. Any halt on any of those cores will cross trigger the other cores to halt, and adjust the selection to the core responsible for the halt. Any symbols loaded will be visible on all cores, and any breakpoints set will be set across all cores. Effectively, the cores can now be debugged as a logical unit instead of individually.

If hardware breakpoints are not an option, step 3 may be skipped as long as shared memory is defined (see below), or the application is loaded to non-shared memory.

Sync Groups

A sync group is like a normal eclipse group of two or more cores except that the IDE will treat the cores of a sync group as one single debug entity. Once sync grouped, the IDE will run, halt, step and reset all the cores as a group, automatically apply loaded debug symbol across all of them, and set/clear any breakpoints across all of them.

Setup

A sync group is formed just like a regular group. Multi-select (using ctrl+click) the cores you want to group, right click, and select “Sync Group” from the context menu. Unlike a regular group, hardware resources are required to physically keep the execution of the cores in sync, so the option may not available if the cores do not have the capability, if the necessary resources are already in use (by another sync group or cross triggering), or if the cores are in a state where the resources cannot be reprogrammed (running). If the group is formed prior to connecting, the resources will be programmed once a connection is initiated. However, the sync group will revert into a regular group if that fails. Both regular and sync groups are stored in the workspace and will be re-applied on the next launch.

SyncGroup - Setup.png

Additional options may need to be configured to properly deal with breakpoints set across cores. Please see Symbols and Breakpoints below.

Execution

Once a sync group is formed, a run issued on any core will start every core running at the same instant. A halt on any core will synchronously halt every core. This applies equally to halts from a breakpoint as well as halts requested by you. The halt will not be instantaneous, but will be within a few clock cycles of the core that initiated the halt. If a breakpoint is programmed to perform an action and re-run, every core will be briefly halted, and then all of them will be synchronously re-run once the breakpoint’s action is complete.

A source step issued on any core will step only that core while running every other core, but the other cores will halt as soon as the step finishes. An assembly step into however will synchronously issue one assembly step on each core.

On a halt, the IDE will automatically adjust the focus in the debug view to the core that caused the halt, assuming the selection was on another core in the same group. Cores that were halted in response to another core will indicate the cause of the halt is due to “Cross Triggering” and will collapse so that their callstack is not visible unless explicitly expanded. These features allow you to run and step the group, using either hotkeys or toolbar buttons, but always see the state of the core that caused the halt of the entire SMP system. For instance, if the application’s logical flow were to jump between cores during a function call, the selection in the debug view would also change cores if that function were stepped over.

SyncGroup - Reason for Halt.png

Symbols and Breakpoints

Once a sync group is formed, symbols loaded to any core will be made available on every core. However, if a program is loaded, it is only written to the one core. It is assumed that the cores are either sharing memory, or there is another way that the data is being made visible to other cores.

Likewise, a breakpoint set on any core is automatically set on every other core. If a breakpoint fails to set, then the all associated breakpoints are disabled too. The breakpoint view will show the breakpoint multiple times, once per core. The layout of this view can optionally be modified to group breakpoints by core so that one can easily distinguish between the same breakpoint set on multiple cores, and an actual physically different breakpoint. To do this, click on the down arrow in the upper right of the view, select Group By and then Debug Contexts

BreakpointsGroupByContexts - Setup.png

BreakpointsGroupByContexts - View.png

Breakpoints and symbols are only shared between similar cores. For instance, if an ARM and a DSP were sync grouped, they would not share breakpoints and symbols. If you are working on similar cores, yet want them to work like they are not similar (ie you want to keep their execution in sync only) then that can be configured per-core in the options. Select Tools->Debugger Options->Misc/Other Options, scroll to the bottom and select “Synchronize execution only”. Then click “Remember My Settings” to save the decision to the launch. This has to be done for every core.

Breakpoints can only be set across cores if the memory is not physically shared, hardware breakpoints are used, or the emulation driver supports shared memory breakpoints and shared memory has been configured in the IDE.

To always use hardware breakpoints select Tools->Debugger Options->Misc/Other Options, scroll to near the bottom and unselect “Allow software breakpoints to be used”. Again, this option is per core and “Remember My Settings” will save the setting to the launch.

Options - Menu.png

Options - PropertyPage.png

To use software breakpoints in shared memory, see Shared Memory below.

Sync Operations outside of a Sync Group

Sometimes a full sync group is needed, and a few limited synchronous capabilities used time to time is sufficient.

Sync Execution

If a normal group is formed instead of a sync group, a run will be applied to the entire group. However, if the run is issued from the group node, instead of a core node, then it will be. This also applies to steps and halts.

Global Breakpoints

By right clicking on a core, there will be an option to Enable Global Breakpoints. When selected on two or more cores, then halts on those cores will be synchronized as if they were in a sync group. Cores with the option enabled should be decorated with “Global breakpoint on”.

EnableGlobalBreakpoints - Setup.png

EnableGlobalBreakpoints - View.png

Like sync groups, the feature requires hardware resources and so may not be available for the same reasons. Likewise, the status in the debug view will indicate if the halt was due to global breakpoints or not by decorating the status with “Cross Triggering”.

Global breakpoints can also be enabled on cores belonging to a sync group. When so done, that sync group will cross trigger halts on cores not belonging to the sync group. In all other respects, the cores are not part of the sync group. Although the option available per core, because a sync group always cross triggers itself, if any members of a sync group have global breakpoints enabled then they effectively all do.

Shared Memory

If the application is loaded to memory that's shared between cores, then the IDE must be told what regions of memory are shared with what other cores in order to use software breakpoints. This is done with the memory map which is programmable via GEL. First, ensure the memory map is on by running the command “GEL_MapOn()”. Next, add shared memory blocks using the command GEL_MapAddStr which takes 5 parameters:

  • Address – the start of the memory block
  • Page – the page of the memory block (or 0 on unpaged architectures)
  • Length – the length (in addresses) of the memory block
  • Attribute – A string that’s either “RAM” or “ROM” followed by “|SH#” where # is the shared memory block number. This number must match a subsequent number used for an identically sized memory block on another core.
  • Wait state – 0

For instance, executing GEL_MapAddStr( 0x8000, 0, 0x1000, "RAM|SH1", 0 ) will inform the IDE that the memory at address 0x8000 to 0x8FFF is in shared memory block one. If evaluated on another core, then the IDE will know those two cores have shared access to that memory block. Shared memory blocks are associated with each other via the id number (one in this example) and not the address. That allows for shared memory blocks to be located at differing addresses per core. However, memory blocks cannot be shared on the same core at two locations, so the id must be unique per core. Ideally, these commands would be in the default GEL script in the StartUp() function.

Once setup, breakpoints set within a shared memory block will be set such that they do not interfere with each other. This feature can even be used outside of a sync group.

Currently, only the Cortex M3 driver and the 55 driver support shared memory.

CIO Console Output

If your code produces output on the CCS console, make sure you have enabled the "Enable CIO function use" option under Tools menu->Debugger Options->Generic Debug Options. This needs to be enabled on every SMP core, as the output may be coming from any of them depending on where the code is scheduled to run.

The CIO buffers are not necessarily thread safe. You must use a thread safe version, or manually protect entry into these functions (and always flush output), or only call into them from one core.

Known issues

SDSCM00045670 Incorrect error message printed if a reset is issued from a sync group node

Effectively if a reset is issued from the group node of a sync group, a misleading error message is printed even though the reset was successful. This can be avoided by issuing the reset from a core instead of the group node.

This was fixed in CCS 5.4

SDSCM00045684 Unable to step past shared memory breakpoints

Effectively if a breakpoint is set in shared memory, only one core will be able to step past it. Other cores will appear to hit it repeated over and over again. If this happens with the CIO breakpoint, errors about an invalid CIO command will be printed.

This can be avoided by using hardware breakpoints, or by marking the shared memory region as “ROM” on all cores but one, even though it’s really RAM.

This was fixed in CCS 5.4

Troubleshooting

  • Breakpoint cannot be stepped past
    • Check that you aren't using software breakpoints without first having defined any shared memory regions. The problem also manifests as the opcode at the breakpoint looking like a BKPT or similar instruction and not the original opcode.
  • Invalid CIO command error messages
    • CIO is implemented via breakpoints. As such, this could be the same issue as above
    • In addition, make sure that the CIO buffer (at the label __CIO_BUFFER_) is not shared between cores, or that you've taken appropriate measures to protect access to it (used an SMP-safe library version, or a mutual exclusion mechanism etc)