ARM Multimedia Users Guide

'''  Content is no longer maintained and is being kept for reference only! ''' Please bookmark the new links:
 * Linux, RT-Linux: http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/index.html
 * RTOS: http://software-dl.ti.com/processor-sdk-rtos/esd/docs/latest/rtos/index.html
 * Android: http://software-dl.ti.com/processor-sdk-android/esd/docs/latest/android/index.html

Return to the Sitara Linux Software Developer's Guide



Overview
'''  Content is no longer maintained and is being kept for reference only! '''

Multimedia codecs on ARM based platforms could be optimised for better performance using the tightly coupled Neon  co-processor. Neon architecture works with its own independent pipeline and register file. Neon technology is a 128 bit SIMD architecture extension for ARM Cortex-A series processors. It is designed to provide acceleration for multimedia applications.

Supported Platforms

 * AM37x
 * Beagleboard-xM
 * AM35x
 * AM335x EVM
 * AM437x GP EVM
 * AM57xx GP EVM

Multimedia on AM57xx Processor
On AM57xx processor, ARM offloads H.264, VC1, MPEG-4, MPEG-2 and MJPEG codecs processing to IVA-HD hardware accelerator. Please refer to AM57xx Multimedia Training guide to learn more on AM57xx multimedia capabilities, demos, software stack, gstreamer plugins and pipelines. Also refer to AM57xx Graphics Display Getting Started Guide to learn on AM57xx graphics software architecture, demos, tools and display applications.

Cortex-A8 Features and Benefits

 * Support ARM v7 with Advanced SIMD (NEON)
 * Support hierarchical cache memory
 * Up to 1 MB L2 cache
 * Up to 128-bit memory bandwidth
 * 13-stage pipeline and enhanced branch prediction engine
 * Dual-issue of instructions

Neon Features and Benefits

 * Independent HW block to support advanced SIMD instructions
 * Comprehensive instruction set with support of 8, 16 &amp; 32-bit signed &amp; unsigned data types
 * 256 byte register file (dual 32x64/16x128 view) with hybrid 32/64/128 bit modes
 * Large register files enables efficient data handling and minimizes access to memory, thus enhancing data throughput
 * Processor can sleep sooner which leads to an overall dynamic power saving
 * Independent 10-stage pipeline
 * Dual-issue of limited instruction pairs
 * Significant code size reduction

Neon support on opensource community
NEON is currently supported in the following Open Source projects.


 * ffmpeg/libav
 * LGPL media player used in many Linux distros
 * NEON Video: MPEG-4 ASP, H.264 (AVC), VC-1, VP3, Theora
 * NEON Audio: AAC, Vorbis, WMA
 * x264 –Google Summer Of Code 2009
 * GPL H.264 encoder –e.g. for video conferencing
 * Bluez –official Linux Bluetooth protocol stack
 * NEON sbc audio encoder
 * Pixman (part of cairo 2D graphics library)
 * Compositing/alpha blending
 * X.Org, Mozilla Firefox, fennec, &amp; Webkit browsers
 * e.g. fbCompositeSolidMask_nx8x0565neon 8xfaster using NEON
 * Ubuntu 09.04 &amp; 09.10 –fully supports NEON
 * NEON versions of critical shared-libraries
 * Android –NEON optimizations
 * Skia library, S32A_D565_Opaque 5xfaster using NEON
 * Available in Google Skia tree from 03-Aug-2009

For additional details, please refer the NEON - ARM website.

SDK Example Applications
This application can be executed by selecting the "Multimedia" icon at the top-level matrix.

NOTE

The very first GStreamer launch takes some time to initialize outputs or set up decoders.



Codec portfolio
Processor SDK includes ARM based multimedia using opensource GPLv2+ FFmpeg/Libav codecs, the codec portfolio includes MPEG-4, H.264 for video in VGA/WQVGA/480p resolution and AAC codec for audio. Codec portforlio for Processor SDK on AM57xx device is listed here

The script file to launch multimedia demo detects the display enabled and accordingly decodes VGA or 480p video. In AM37x platform VGA clip is decoded when LCD is enabled and 480p is decoded when DVI out is enabled. Scripts in "Settings" menu can be used to switch between these two displays.

MPEG4 + AAC Decode
MPEG-4 + AAC Dec example application demonstrates use of MPEG-4 video and AAC audio codec as mentioned in the description page below.



The multimedia pipeline is constructed using gst-launch, GStreamer elements such as qtdemux is used for demuxing AV content. Parsers are elements with single source pad and can be used to cut streams into buffers, they do not modify the data otherwise. gst-launch-0.10 filesrc location=$filename ! qtdemux name=demux demux.audio_00 ! queue ! ffdec_aac ! alsasink sync=false demux.video_00 ! queue ! ffdec_mpeg4 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0 "filename" is defined based on the selected display device which could be LCD of DVI.

MPEG4 Decode
MPEG-4 decode example application demonstrates use of MPEG-4 video codec as mentioned in the description page below. gst-launch-0.10 filesrc location=$filename ! mpeg4videoparse ! ffdec_mpeg4 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

H.264 Decode
H.264 decode example application demonstrates use of H.264 video codec as mentioned in the description page below. gst-launch-0.10 filesrc location=$filename ! h264parse ! ffdec_h264 ! ffmpegcolorspace ! fbdevsink device=/dev/fb0

AAC Decode
AAC decode example application demonstrates use of AAC video codec as mentioned in the description page below. gst-launch-0.10 filesrc location=$filename ! aacparse ! faad ! alsasink

Streaming
Audio/Video data can be streamed from a server using souphttpsrc. For example to stream audio content, if you set-up an apache server on your host machine you can stream the audio file HistoryOfTI.aac located in the files directory using the pipeline

Multimedia Peripheral Examples
Examples of how to use several different multimedia peripherals can be found on the ARM Multimedia Peripheral Examples page.

CPU and Memory load
The following results are obtained on AM37x EVM using pre-build SDK MLO, u-boot, kernel and file-system on a NFS set-up.

The following results are obtained on an AM335X EVM (DDR2) and an AM335X Starter Kit (DDR3) using pre-built SDK MLO, u-boot, kernel and file-system on an SD Card setup.

oProfile
OProfile is a common profiling tool used on Linux platforms, a kernel driver and a user daemon is used to collect samples of data. This tool can be used to profile a huge variety of system-level statistics. Please refer OProfile for additional details.

Listed below are the profiling results of Mpeg4 + AAC in VGA resolution at 300MHz and 1GHz via NFS server. Profiling results are obtained from the serial terminal.

In order to profile the multimedia application kernel has to be compiled with oprofile enabled. Followed by this, opcontrol has to be enabled for the first time root@am37x-evm:~# opcontrol --vmlinux=/boot/vmlinux-`uname -r` Setopp power/clocks application can be used to scale the clock settings, the example below shows the settings for opp2, similarly clock can be set to 1GHz using the setopp1.sh application. root@am37x-evm:~# setopp2.sh Set to operating point 2: CPU = 300 MHz - Processor      : ARMv7 Processor rev 2 (v7l) BogoMIPS       : 299.95 Features       : swp half thumb fastmult vfp edsp thumbee neon vfpv3 CPU implementer : 0x41 CPU architecture: 7 CPU variant    : 0x3 CPU part       : 0xc08 CPU revision   : 2

Hardware       : OMAP3 EVM Revision       : 0020 Serial         : 0000000000000000 profiling can be calibrated by executing the following set of applications in the given sequence root@am37x-evm:~# opcontrol --reset root@am37x-evm:~# opcontrol --start root@am37x-evm:~# runMpeg4AacDec.sh root@am37x-evm:~# opcontrol –-stop root@am37x-evm:~# opreport -l root@am37x-evm:~# opreport opcontrol is used to initiate the profiling and opreport to generate the profiling results

Power Benchmark
The power measurements below are analyzed on the AM3730 ES1.2 processor on a Rev.G. EVM. Here we indicate the power measurement after first boot, in suspend mode and with additional power features enabled.

The jumper settings for power measurements are:

Suspend Mode
The Suspend/Resume application in Power/Clocks SDK example application drives the system in suspend mode. The measurements below indicate total power consumption in suspend mode.

After Resume
After the system is resumed from sleep mode, when sleep_while_idle and enable_off_mode features are enabledr.

MPEG4 Decode
After the system resume with enhanced power save features enabled, the total power consumption for MPEG4 decode is as shown below.

MPEG4 Decode with Scaling Governor Enabled
These power measurements can be further optimized by setting the scaling governor feature to 'ondemand'. Here the power values are dynamically scaled at run time depending on the system level requirements. It can be enabled using sysfs, the resultant improvement in power values are as shown below. echo "ondemand" &gt; /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Further power optimization
The system level power consumption can be further optimized by disabling clocks of unused modules and enabling Smart Reflex. Additional details of power optimization can be obtained from OMAP power management guide and PSP user guide for 2.6.37 kernel

SDK Multimedia Framework
Multimedia framework for cortex-a8 SDK will leverage GStreamer multimedia stack with gst-ffmpeg plug-in's to support GPLv2+ FFmpeg/libav library code.



gst-launch is used to build and run basic multimedia pieplines to demonstrate audio/avideo decoding examples.



GStreamer

 * Multimedia processing library
 * Provides uniform framework across platforms
 * Includes parsing &amp; A/V sync support
 * Modular with flexibility to add new functionality via plugins
 * Easy bindings to other frameworks

Some of the build dependencies for GStreamer are shown here:



Open Source FFmpeg Codecs
FFmpeg is an open source project which provides a cross platform multimedia solution.


 * Free audio and video decoder/encoder code licensed under GPLv2+ (GPLv3 licensed codecs can be build separately)
 * A comprehensive suite of standard compliant multimedia codecs

- Audio - Video - Image - Speech


 * Codec software package
 * Codec libraries with standard C based API
 * Audio/Video parsers that support popular multimedia content
 * Use of SIMD/NEON instructions cortex-A8 neon architecture
 * Neon provides 1.6x-2.5x performance on complex video codecs

Multimedia Neon Benchmark
Test Parameters:


 * Sep 21 2009 snapshot of gst-ffmpeg.org
 * Real silicon measurements on Omap3 Beagleboard


 * Benchmarks released by ARM demonstrating an overall performance improvement of ~2x.



FFmpeg Codec Licensing
FFmpeg libraries include LGPL, GPLv2, GPLv3 and other license based codecs, enabling GPLv3 codecs subjects the entire framework to GPLv3 license. In the Sitara SDK GPLv2+ licensed codecs are enabled. Enabling Additional details of legal and license of these codecs can be found on the FFmpeg/libav webpage.

Third Party Solutions
Third parties like Ittiam and VisualON provide highly optimized ARM only codecs on Linux, WinCE and Android OS.

Software Components &amp; Dependencies
The following lists some of the software components and dependencies associated with the Sitara SDK.

Dependancies: Required packages to build Gstreamer on Ubuntu:

sudo apt-get install automake autoconf libtool docbook-xml docbook-xsl fop libxml2 gnome-doc-utils


 * build-essential
 * libtool
 * automake
 * autoconf
 * git-core
 * svn
 * liboil0.3-dev
 * libxml2-dev
 * libglib2.0-dev
 * gettext
 * corkscrew
 * socket
 * libfaad-dev
 * libfaac-dev

Software components for Sitara SDK Release:


 * glib
 * gstreamer
 * liboil
 * gst-plugins-good
 * gst-ffmpeg
 * gst-plugins-bad
 * gst-plugins-base

Re-enabling Mp3 and Mpeg2 decode in the Processor SDK
Starting with version 05.05.01.00, mp3 and mpeg2 codecs are no longer distributed as part of the SDK. These plugins can be re-enabled by the end user through rebuilding the gst-plugins-ugly package. The following instructions have been tested with gst-plugins-ugly-0.10.19 which can be found at gstreamer.freedesktop.org. Note that these instructions will work for any of the gstreamer plugin packages found in the sdk.


 * Source environment-setup at the terminal
 * Navigate into the example-applications path under the SDK install directory
 * Extract the GStreamer plug-in source archive
 * Navigate into the folder that was created
 * On the command line type
 * Notice that some components are not built because they have dependencies that are not part of our SDK
 * Run  to build the plugins.
 * Run

Other

 * For technical support please post your questions at http://e2e.ti.com or search forum post Database.

Archived
Sitara SDK 05.07