Linux Aware Debug (CCSv4.x)

From Texas Instruments Wiki
(Redirected from Linux Aware Debug)
Jump to: navigation, search

Overview

Embedded Linux is a popular embedded operating system that continues to enjoy rapid growth among the embedded community. As the number of users (and the complexity of the applications) continues to grow, so do the demands for tools to support debugging of embedded applications running on the Linux environment. Code Composer Studio v4 (CCSv4) provides such Linux aware debug support in the form of Linux (Stop-Mode) Thread Level Debug and Linux Module Debug capability.

The stop-mode debug capabilities of CCS are most often used for low-level development activities such as kernel-porting and driver development. Higher level application development is better suited to a run-mode debugger such as gdb.

Dependencies

The following dependencies apply to Linux Aware Debug:

  • CCS versions: CCS 4.x (CCSv5 works a bit differently) 
  • Devices: Cortex A8 based devices (tested on OMAP 3430/3530 EVMs and beagleboard). ARM926 based devices (requires CCSv4.1.3 or later).
  • Linux versions: 2.6.19 or later

It may operate on other kernel versions but it has not been tested and is not supported.

Setup

The example below describes setting up CCSv4 Linux aware debugging for an OMAP3530 EVM that has Linux 2.6 flashed on the board and boots up upon power up. Note that the target setup should not include the default gel initialization scripts as they assume the mmu is disabled and no OS is present. In the setup editor, select the advanced tab and expand the tree. Delete any entries under "initialization script" for every core.

First, launch a debug session for the desired target. Once a debug session is open, load the debug symbols for the Linux kernel ('Target->Load Symbols'):

CCSv4 LinuxAware02.PNG

Connect to the target (if not already done so). In this example Linux is flashed on the target and boots up upon power up. Connecting to the target will halt execution. Upon connection the 'Debug' view will update the call stack and attempt to open any associated source files in the editor. The first attempt to open a source file will fail with an option to specify the source file. Click the 'Locate the Source File...' button and use the file browse dialog to find and select the source file in the source tree:

CCSv4 LinuxAware13.PNG

Once the correct file is specified, the file will open in the editor. Any subsequent kernel source files to open should automatically be found:

CCSv4 LinuxAware05.PNG

Configure the OS Debugging options by selecting 'Tools->Debugger Options' and scrolling down to the 'OS Aware Debug Options' group. Adjust the options as you desire, and then click 'Remember My Settings' if you would like it applied on subsequent launches. The options can also be changed on the fly while Linux aware debugging is enabled, and the changes will be applied on the next target halt. The options are described in more detail in the following sections, but the default options should be sufficient:

CCSv4 LinuxAwareOptions.PNG

Once the kernel symbols are loaded, and the options are configured, Linux aware debugging can be enabled by right-clicking on the target in the 'Debug' view and selecting the 'Enable OS Debugging...' option in the context menu:

CCSv4 LinuxAware06.PNG

Note that OS debugging can only be enabled when the target is halted and connected.

CCSv4 has now been configured for Linux aware debugging. Note the visibility of all the active Linux processes in the 'Debug' view once Linux aware debug as been enabled:

CCSv4 LinuxAware07.PNG

Thread Level Debug

Once Linux aware debugging has been enabled, by default all the active processes will be visible in the 'Debug' view:

CCSv4 LinuxAware08.PNG

Each of these process nodes can be expanded to see threads associated with it, and each thread can each be expanded to see the call stack of that thread. Each process can have it's own set of symbols loaded via 'Target->Load Symbols...' or 'Target->Add Symbols...', but it will also see symbols loaded on the CPU context (including the symbols loaded from the vmlinux image originally). Providing process symbols are loaded, the callstack will unwind into user code from kernel code. All debug views will update to reflect the context of the selected thread and stack frame.

The list of active process is updated every time the target is halted. Any new processes that have been started since the last update will appear at the bottom of the list. In the example below, target execution is resumed and the 'top' application is started from a shell. Upon a target halt, the list of processes in the 'Debug' view is updated to include the new 'top' process:

CCSv4 LinuxAware09.PNG

If execution is again resumed and the 'top' application is terminated and then the list of processes is updated (via another target halt), it can be seen that the 'top' process has been terminated:

CCSv4 LinuxAware10.PNG

Terminated processes can be removed from the view by pressing the double "X" button to the left of the run button.

The Linux aware debugging support in CCS4.0 is stop mode only, meaning it is not possible to halt only one thread or process and leave other processes running. Halting one will halt them all. The thread with the label "Current-Suspended" is the thread that is currently scheduled and running.

Thread Level Breakpoints

Any breakpoint set will apply only to the thread that is selected in the 'debug' view. Setting a breakpoint when the CPU is selected (the 'main' thread at the top) will create a breakpoint that applies for all threads (standard breakpoint). Only software breakpoints can be thread-aware. As such, hardware breakpoints can only be set when the CPU is selected. Additionally, since user-space memory is dependent on what process is active, software breakpoints cannot be set in user space from the CPU context either.

If a breakpoint is thread-aware, the 'Breakpoints' view will display which thread it is associated with. In the screenshot below, note that the first breakpoint entry is a standard breakpoint and the second breakpoint entry is a thread level breakpoint with the thread context information circled in red:

CCSv4 LinuxAware11.PNG

When the breakpoint is reached during target execution, the target will halt and the debugger will check to see if it is a thread level breakpoint. If so, it will then determine if the thread that reached the breakpoint is the same as the thread the breakpoint was set for. If the thread the breakpoint was set for is the thread that reached the breakpoint, the target will remain halted. If it is not, the debugger will then resume target execution and output a message to the console:

Cortex_A8_0: Thread-aware breakpoint at <address> was hit by another thread.  Target is being re-run.

In the screenshot below, there are two breakpoints set at the same location. Only the thread level breakpoint for the 'top' process is enabled. During execution, another thread reached the breakpoint first and once the target halted, the debugger determined that the thread that reached the breakpoint is not the thread the breakpoint was set for, and resumed execution. Note the message reported to the console. The next time the breakpoint was reached, it was by the thread that the breakpoint was set for. Hence the target remained halted:

CCSv4 LinuxAware12.PNG

Note that the target will always halt when a thread level breakpoint is reached. Hence it is always intrusive, whether or not the breakpoint applies to the thread that reached it. However in the cases when a halt and rerun occurs, the list of processes in the 'Debug' view is not updated.

Process Visibility Options

If only working with module or kernel code, and information about running processes is not needed, the above behaviour can be disabled by unchecking the option 'Display processes and threads in the debug view'. Linux aware debugging can still be used for module symbol management, and to enable the use of software breakpoints from the CPU context.

If you're developing with a slow emulator, you may find that the time it takes to determine if processes or threads have changed is significant. If this is the case, you can uncheck the option 'Update process list on every halt' at any time. If unchecked, the debugger will assume the processes have not changed since the last halt. You can check this option at any time to resume the process list update on the next halt. You can also force the debugger to refresh the process list by evaluating the gel expression 'GEL_RefreshOSProcessList()' from the console.

Module Debug

The Module Debug capability provides automatic loading, relocation and unloading of the module symbols. The debugger monitors the insertion and removal of the user specified modules and manages the module symbols behind the scene. Breakpoints set in module code can be set on the source line, or at a module symbol before the module is inserted. Since the module is not yet inserted, there will be no symbols loaded for it, and the breakpoint will not enable as it's location won't resolve to an address. When the target is run, and the debugger detects a module insertion, symbols will load automatically and enable the breakpoints so that they are hit when the target is re-run.

Patching the Kernel

To use this feature effectively, the user has to patch the Linux kernel as follows, all the changes apply to the file module.c in 2.6_kernel/kernel:

  • At the beginning of the file, insert the function TI_CCS_Notify_Module_List_Changed
void TI_CCS_Notify_Module_List_Changed() {
 static unsigned long count = 0;
 ++count;
}
  • Go to the end of the function sys_delete_module in the file module.c, insert a line to call TI_CCS_Notify_Module_List_Changed

// For the Linux kernel version up to 2.6.28
asmlinkage long sys_delete_module(const char __user *name_user, unsigned int flags)
{
 ...
 
out:
 mutex_unlock(&module_mutex);
 
 TI_CCS_Notify_Module_List_Changed();
 
 return ret;
}
// For the Linux kernel version 2.6.28.1 and above
SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
 unsigned int, flags)
{
 
 ...
 
 out:
 mutex_unlock(&module_mutex);
 TI_CCS_Notify_Module_List_Changed();
out_stop:
 stop_machine_destroy();
 return ret;
}
  • Go to the function sys_init_module in the file module.c, insert 3 lines to call TI_CCS_Notify_Module_List_Changed
// For the Linux kernel version up to 2.6.28
/* This is where the real work happens */
 
asmlinkage long sys_init_module(void __user *umod, unsigned long len, const char __user *uargs)
{
 ...
 
 /* Drop lock so they can recurse */
 mutex_unlock(&module_mutex);
 
 blocking_notifier_call_chain(&module_notify_list, MODULE_STATE_COMING, mod);
 
 // FIRST CALL
 TI_CCS_Notify_Module_List_Changed();
 
 /* Start the module */
 if (mod->init != NULL)
 ret = mod->init();
 if (ret < 0) {
 /* Init routine failed: abort. Try to protect us from buggy refcounters. */
 mod->state = MODULE_STATE_GOING;
 synchronize_sched();
 if (mod->unsafe)
 printk(KERN_ERR "%s: module is now stuck!\n", mod->name);
 else {
 module_put(mod);
 mutex_lock(&module_mutex);
 free_module(mod);
 mutex_unlock(&module_mutex);
 }
 
 // SECOND CALL
 TI_CCS_Notify_Module_List_Changed();
 
 return ret;
 }
 
 ...
 
 mutex_unlock(&module_mutex);
 
 // THIRD CALL
 TI_CCS_Notify_Module_List_Changed();
 
 return 0;
}

// For the Linux kernel version 2.6.28.1 and above
/* This is where the real work happens */
SYSCALL_DEFINE3(init_module, void __user *, umod,
 unsigned long, len, const char __user *, uargs)
{
 ...
 
 /* Drop lock so they can recurse */
 mutex_unlock(&module_mutex);
 
 blocking_notifier_call_chain(&module_notify_list, MODULE_STATE_COMING, mod);
 
 do_mod_ctors(mod);
 // FIRST CALL
 TI_CCS_Notify_Module_List_Changed();
 
 /* Start the module */
 if (mod->init != NULL)
 ret = do_one_initcall(mod->init);
 if (ret < 0) {
 /* Init routine failed: abort. Try to protect us from
 buggy refcounters. */
 mod->state = MODULE_STATE_GOING;
 synchronize_sched();
 module_put(mod);
 blocking_notifier_call_chain(&module_notify_list,
 MODULE_STATE_GOING, mod);
 mutex_lock(&module_mutex);
 free_module(mod);
 mutex_unlock(&module_mutex);
 wake_up(&module_wq);
 
 // SECOND CALL
 TI_CCS_Notify_Module_List_Changed();
 return ret;
 }
 
 ...
 
 mutex_unlock(&module_mutex);
 
 // THIRD CALL
 TI_CCS_Notify_Module_List_Changed();
 
 return 0;
}

After the changes are made, the kernel must be rebuilt and placed on the target. The option to 'Automatically load module symbols' can now be changed to 'When modules are loaded or unloaded'.

Now, when Linux aware debugging is enabled, the debugger will automatically set a hidden breakpoint on the call to 'TI_CCS_Notify_Module_List_Changed'. Whenever hit, the debugger will load and/or unload the corresponding module symbol files as defined in LinuxDriver.ini and re-run the target. Note that this is intrusive on the target execution whenever modules are loaded or unloaded.

Working With an Unpatched Kernel

There is an alternative way to use this feature if the kernel cannot be patched. The option to 'Automatically load module symbols' can be changed to 'Whenever halted in the module load/unload routines' and then breakpoints must then be manually set on the same lines of code that would have been modified to call TI_CCS_Notify_Module_List_Changed() if the kernel were patched. When those breakpoints are hit, the debugger will detect if modules have changed and load and/or unload the corresponding module symbol files as defined in LinuxDriver.ini. The target will not automatically be re-run in this case, unless the breakpoints are manually configured to not halt (such as by giving them a condition of '0').

Define Module Symbols

The debugger uses the LinuxDriver.ini file in <INSTALL DIR>\ccsv4\DebugServer\bin\win32 to determine the list of the modules to monitor and their corresponding symbol files. For example, if the user wants to debug the modules vfat and ntfs and their symbol files are located at c:\Linux\src\modules\vfat\vfat.ko and d:\Kernel\src\ntfs\ntfs.ko respectively, the following lines have to be added to the file LinuxDriver.ini.

vfat=c:\Linux\src\modules\vfat\vfat.ko
ntfs=d:\Kernel\src\ntfs\ntfs.ko

After the file LinuxDriver.ini is modified, the user has to restart the OS Debugging.

Limitations and Known Issues

There are several usability and performance issues to be aware of:

  • List of processes in the 'Debug' view are not sorted in alphabetical order upon subsequent updates (i.e. new processes are added to the bottom of the original list)
  • If the user sets thread-aware breakpoints on a process that terminates, the debugger will not be able to clean up the breakpoints as there will no longer be an mmu context for that process. Linux may still have a cache of the application in memory, and if the process is restarted, the breakpoint will be embedded in memory and it will not be possible to remove it. To resolve this issue, login as root and issue the following command in the Linux console to clear the Linux page cache: echo 1 > /proc/sys/vm/drop_caches
  • Certain revisions of OMAP34x and 35x devices (below rev 3.0) have a silicon bug involving how the emulator interacts with the target when in low-power mode. Once in low-power mode, CCSv4 is unable to connect or halt on the Cortex A8. When working with such target, it is recommended to disable or prevent the device from entering low-power mode. (An application with an infinite spin loop can be created and run to always keep the processor awake).