CMEM Overview

From Texas Instruments Embedded Processors Wiki

Introduction

CMEM is an API and library for managing one or more blocks of physically contiguous memory. It also provides address translation services (e.g. virtual to physical translation) and user-mode cache management APIs. This physically contiguous memory is useful as data buffers that will be shared with another processor (e.g. for the DSP on an OMAP3) or a hardware accelerator/DMA (e.g. used by codecs on a DM355/365)

Using its pool-based configuration, CMEM enables users to avoid memory fragmentation, and ensures large physically contiguous memory blocks are available even after a system has been running for very long periods of time.

It was originally developed for the DM644x, and has been ported to several Operating Systems (e.g. Linux, WinCE, Nucleus, Green Hills Integrity, and others). Although generally associated with Codec Engine, it has no dependency on Codec Engine and can be used on its own.

It's currently distributed as a component in the Linux Utils product, which is also included in all Linux and WinCE based DVSDK releases.

Overview

TBD. (Need to add some of the OS-independent features of CMEM - how memory is reserved, pools vs. heaps, etc)

Also, see the Linux Utils Roadmap for details about each release.

Configuration

TBD. (Need to add some of the OS-independent config options of CMEM - base addr and size, multiple blocks, pools vs. heaps, etc)

Linux Configuration

Linux configuration is done when installing the cmemk.ko driver, typically done using the insmod command. The cmemk.ko driver accepts command line parameters for configuring the physical memory to reserve and how to carve it up.

The following is an example of installing the cmem kernel module:

/sbin/insmod cmemk.ko pools=4x30000,2x500000 phys_start=0x0 phys_end=0x3000000 

• phys_start and phys_end must be specified in hexadecimal format
• pools must be specified using decimal format (for both number and size), since using hexadecimal format would visually clutter the specification due to the use of "x" as a token separator

This particular command creates 2 pools. The first pool is created with 4 buffers of size 30000 bytes and the second pool is created with 2 buffers of size 500000 bytes. The CMEM pool buffers start at 0x0 and end at 0x3000000 (max).

Pool buffers are aligned on a module-dependent boundary, and their sizes are rounded up to this same boundary. This applies to each buffer within a pool. The total space used by an individual pool will therefore be greater than (or equal to) the exact amount requested in the installation of the module.

The poolid used in the driver calls would be 0 for the first pool and 1 for the second pool.

Pool allocations can be requested explicitly by pool number, or more generally by just a size. For size-based allocations, the pool which best fits the requested size is automatically chosen.

WinCE Configuration

Configuration of CMEM in WinCE-based environments is typically done via the registry and/or statically built into the driver (for closed systems). Here is an example for a line to be added to the MEMORY section of 'config.bib' of your BSP:

   CMEM_DSP	 89000000    02800000    RESERVED ; 40 MB

That reserves 40MB of memory for CMEM, DSPLINK, DSP code as well as DSP heap usage starting at virtual address 0x89000000. There is no distinction here between the different modules memory usage. Obviously all of them need to be configured accordingly. Registry settings for CMEM use physical start and end addresses for any defined block of pools.

Here is an example CMEM configuration registry entry in platform.reg for TI EVM3530:

;-- CMEM --------------------------------------------------------------------
IF SYSGEN_CMEM
[HKEY_LOCAL_MACHINE\Drivers\BuiltIn\CMEMK]
    "Prefix"="CMK"
    "Dll"="cmemk.dll"
    "Index"=dword:1
    ; Make 7 pools available for allocation for block 0
    ; Make 1 pool available for allocation for block 1
    "NumPools0"=dword:7
    "NumPools1"=dword:0
 
    "Block0_NumBuffers_Pool0"=dword:20
    "Block0_PoolSize_Pool0"=dword:1000 ; size in bytes (hex)
    "Block0_NumBuffers_Pool1"=dword:8
    "Block0_PoolSize_Pool1"=dword:20000 ; size in bytes (hex)
    "Block0_NumBuffers_Pool2"=dword:5
    "Block0_PoolSize_Pool2"=dword:100000 ; size in bytes (hex)
 
    "Block0_NumBuffers_Pool3"=dword:1
    "Block0_PoolSize_Pool3"=dword:15cfc0 ; size in bytes (hex)
    "Block0_NumBuffers_Pool4"=dword:1
    "Block0_PoolSize_Pool4"=dword:3e800 ; size in bytes (hex)
    "Block0_NumBuffers_Pool5"=dword:1
    "Block0_PoolSize_Pool5"=dword:36ee80 ; size in bytes (hex)
 
    "Block0_NumBuffers_Pool6"=dword:3
    "Block0_PoolSize_Pool6"=dword:96000 ; size in bytes (hex)
 
    ;; "Block1_NumBuffers_Pool1"=dword:2
    ;; "Block1_PoolSize_Pool1"=dword:4000 ; size in bytes (hex)
 
 
    ; Physical start + physical end can be use to ask CMEM to map a specific
    ; range of physical addresses.
    ; This is a potential security risk.  If physical start == 0 then the code
    ; hits a special case.
    ; physical end - physical start == length of allocation.  In the special
    ; case, memory is allocated via a call to AllocPhysMem() (as shown in
    ; this example).  MmMapIoSpace() is used to map the normal case where
    ; physical start != 0.
    ;
    ; physical start and end for block 0
    "PhysicalStart0"=dword:85000000
    "PhysicalEnd0"=dword:86000000
    ; physical start and end for block 1
    "PhysicalStart1"=dword:0
    "PhysicalEnd1"=dword:0
ENDIF SYSGEN_CMEM
;------------------------------------------------------------------------------

The CMEM driver information must also be added to the platform.bib file (or some other .bib file that gets put into ce.bib). Here is an example of the CMEM driver entry in platform.bib:

;-- CMEM ----------------------------------------------------------------------
IF SYSGEN_CMEM
cmemk.dll  $(_FLATRELEASEDIR)\cmemk.dll               NK SHK
ENDIF BSP_CMEM
;------------------------------------------------------------------------------

Debugging Techniques

Linux users can execute "cat /proc/cmem" to get status on the buffers and pools managed by CMEM.

There is also a debug library provided that provides tracing diagnostics during execution. XDC Config users can link in this library by adding the following to their application's config script:

var CMEM = xdc.useModule('ti.sdo.linuxutils.cmem.CMEM');
CMEM.debug = true;

General Purpose Heaps

In CMEM 2.00, CMEM added support for a general purpose heap. Using the example above, in addition to the 2 pools, a general purpose heap block is created from which allocations of any size can be requested. Internally, allocation sizes are rounded up to a module-dependent boundary and allocation addresses are aligned either to this same boundary or to the requested alignment (whichever is greater).

The size of the heap block is the amount of CMEM memory remaining after all pool allocations. If more heap space is needed than is available after pool allocations, you must reduce the amount of CMEM memory granted to the pools.

The main disadvantage to using heap(s) over pools is fragmentation. After several sequences of codec creation/deletion, in different orders, with possibly different create() params, you may end up fragmenting your heap and being unable to acquire a requested memory block - possibly resulting in a codec creation failure.

Typically, during development, users will use CMEM with heap-based memory, as heap usage requires very little configuration, and users don't know how to configure pool memory(!). In a production system, however, it's strongly recommended that pool configuration be used to avoid memory fragmentation and confusing end user errors.

Application Cleanup

CMEM 2.23 introduced a facility to clean up unfreed buffers when an application exits, either prematurely or in a normal fashion. This facility is achieved by maintaining an "ownership" list for each allocated buffer that is inspected upon closing a device driver instance. During this inspection all allocated buffers are checked, and when it is determined that the closing process is on the ownership list of an allocated buffer, the process is removed from the list. If this causes the list to become empty the associated buffer is actually freed, otherwise it is maintained in the allocated state on behalf of other owners. A side-effect of this model is that only a buffer "owner" is allowed to free the buffer.

In order to facilitate multiple owners of an allocated buffer, a new set of APIs was introduced:

void *CMEM_registerAlloc(unsigned long physp);
int CMEM_unregister(void *ptr, CMEM_AllocParams *params);

CMEM_registerAlloc() takes a buffer physical address as input (achieved through CMEM_getPhys()) and returns a fresh virtual address that is mapped to that buffer, while also adding the calling process to the ownership list. CMEM_unregister() is equivalent to CMEM_free() and releases ownership of the buffer (as well as freeing it if all owners have released the buffer).

In CMEM 2.24, ownership is established on a per-process (and per-thread) basis. This detail becomes important when using CMEM in multiple threads of a given process - if one thread allocates a CMEM buffer and a separate thread of the same process is responsible for freeing that buffer, the "freeing" thread will not be allowed to free the buffer since it is not on the ownership list.

CMEM 2.24.01 changes the ownership policy to be based on the calling process' file descriptor instead of the calling process' process descriptor. This facilitates thread-based sharing of buffers, allowing any thread within a process to free a buffer that was allocated by a different thread within the same process, since threads within a process all use the same file descriptor.

CMEM FAQ

Q: Why am I'm getting this error when loading the CMEM (or other!) driver: "insmod: error inserting 'cmemk.ko': -1 Invalid module format"?

A: This error indicates the CMEM kernel module was built with a different Linux kernel version than the version running on the target. You need to rebuild CMEM against the kernel running on your target.

Q: Can CMEM_getPhys() be used to translate any virtual address to its physical address?

A: In theory, "yes". However, sometime after Linux version 2.6.10 the CMEM kernel module get_phys() function stopped working for kernel addresses. A new get_phys() was provided to work with newer kernels, but it was discovered that this new one didn't correctly translate non-direct-mapped kernel addresses, so code was added to CMEM to save the lower/upper bounds of the CMEM blocks' kernel addresses, and manually look for those in get_phys() before trying more general methods of translation.

So, in short, CMEM's get_phys() doesn't handle non-direct-mapped kernel addresses except the ones that correspond to CMEM's managed memory block(s).

Q: How does CMEM relate to DSPLink's POOL feature?

A: Though they provide overlapping features, they are independent, and each has unique features.

• CMEM
  • CMEM can be used on systems without a remote DSP slave (e.g. DM365 codecs require physically contiguous memory when using HW accelerators)
  • CMEM buffers can be cached
  • CMEM blocks support fixed size pools (no fragmentation) as well as heaps (easier to use)
  • CMEM configuration doesn't require a rebuild (they're provided as insmod params)
• POOL
  • POOL buffers can be allocated on one processor and freed on another

Licensing

In CMEM 2.00, the CMEM Linux release is LGPL v2 for the user mode lib and GPL v2 for the kernel mode driver.

In CMEM 2.21, the Linux user mode library licensing changed from LGPL to BSD. The Linux kernel mode driver continued to be GPL v2.