Template:Multimedia dsp gst plugin internal

= DSP C66x Gstreamer Plugin Internals = TI's Processor SDK Linux supplies ARM based GStreamer plugin that abstracts C66x DSP offload. The primary goal of this DSP GStreamer plugin is to demonstrate how C66x can be used in GStreamer framework, in combination with other GStreamer plugins. The plugin, under the hood, uses OpenCL to dispatch to the C66x cores. This plugin provides sample DSP kernels and can be used as a reference to develop user's own DSP kernels.

Overview of Existing Source Code
Source code of the DSP plugin can be found from http://git.ti.com/processor-sdk/gst-plugin-dsp66.

As shown in the figure below, the GST plugin code (gstdsp66*.c and gstdsp66*.h files) is directly under the ./src folder. It is implemented in C following GST framework requirements, and therefore it is compatible with the gstreamer version used in Processor-SDK-Linux.

Dispatch of work load to DSP is done via call to functions in independent shared objects, which are implemented in OpenCL code organized under the kernels folder. The kernels folder currently has a sub-folder of oclconv, which provides sample DSP kernels for image processing. As long as the APIs between the GST plugin code (in ./src folder) and OpenCL code (in ./src/kernels/oclconv folder) are the same, this shared object can be compiled and installed separately. This approach allows easier modification, implementation and maintenance once the APIs are fixed.



The image processing functions in oclconv are implemented via calls to DSP optimized imglib and vlib library functions, or implemented in OpenCL C.
 * Kernels implemented with OpenCL C: Median2x2
 * Kernels implemented with imglib function calls from OpenCL C: Median3x3, Sobel3x3, Conv5x5
 * Kernels implemented with vlib function calls from OpenCL C: Canny

Adding Custom DSP Kernels
Using the existing oclconv as the template, more folders can be added under ./src/kernels folder to create shared libraries with additional wrappers (for functions invoked from GST plugin context) and OCL (host side and DSP) kernels. Makefile in ./src/kernels folder will attempt make in all sub-folders. Each sub-folder will provide independent shared library object that can be invoked from gstdsp66 context (e.g., function calls in ./src/gstdsp66videokernel.c file). Individual shared object libraries can be independently recompiled and updated in the target file system.

Modifying the Existing Plugin
The DSP plugin also allows easy modifications and additions, and below are some examples.

Currently the DSP plugin provides five sample image process operations: 1) Median2x2; 2) Median3x3; 3) Sobel3x3; 4) Conv5x5; and 5) Canny. Users can modify the source code to add more image processing operations as needed.

Currently the DSP plugin provides properties as below. More properties can be added so that they can be passed from gst-launcher.
 * kerneltype: select the kernel type
 * filtersize: the size of the filter, choose from (5,9,25)
 * lum-only: true for applying the filter on luminance only, false for applying on all three planes.
 * arbkernel: provide a way to specify the name of the kernel invoked via OpenCL.

Details of a specific image processing kernel can also be modified, e.g., the coefficients for Conv5x5 kernel, which are defined in kernels/oclconv/conv.cl::kernel void Conv5x5 function.

Rebuilding and Installing the Plugin
After modifications/additions are made for the DSP plugin source code, the plugin needs to be rebuilt, and this can be done from the Yocto build.

First, please refer to Processor SDK Building The SDK to set up the build environment and bitbake the original recipe for gstreamer1.0-plugins-dsp66, i.e.,

After the bitbake command above is successfully done, ./build/arago-tmp-external-linaro-toolchain/work/cortexa15hf-vfp-neon-linux-gnueabi/gstreamer1.0-plugins-dsp66/git-r<*> will be created with the original source code under the git sub-folder. Copy the modified and/or the newly added files to the git sub-folder, and rebuild the plugin referring to Rebuild Recipe.

Last, install the rebuilt plugin on target filesystem referring to Install Package. After the installation, the following files will be updated and/or added. Gstreamer framework includes seamless detection and registration of the new plugin.
 * /usr/lib/gstreamer-1.0/libgstdsp66.so
 * /usr/lib/liboclconv.so
 * [optional] any additional shared library (as described in previous section), should be placed in /usr/lib