MKL Memory Leak?

We recently heard from an NMath user:

I am seeing a memory accumulation in my application (which uses NMath Core 2.5). From my memory profiler it looks like it could be an allocation in DotNetBlas.Product(), within the MKL dgemm() function.

I understand that MKL is designed such that memory is not released until the application closes. However, as this application runs in new worker threads all the time, I wondering if each new thread is holding onto it’s own memory for Product().

I’ve tried setting the system variable MKL_DISABLE_FAST_MM – but this seems to have made no difference – would I expect this to have an immediate effect (after re-starting the application)? Is there any other way within NMath to force MKL to release memory?

It’s true that for performance reasons, memory allocated by the Intel Math Kernel Library (MKL) is not released. This is by design and is a one-time occurrence for MKL routines that require workspace memory buffers. However, this workspace appears to be allocated on a per-thread basis, which can be a problem for applications that spawn large numbers of threads. As the MKL documentation delicately puts it, “the user should be aware that some tools might report this as a memory leak”.

There are two solutions for multithreaded applications to avoid continuous memory accumulation:

  1. Use a thread pool, so the number of new threads is bounded by the size of the pool.
  2. Use the MKL_FreeBuffers() function to free the memory allocated by the MKL memory manager.

The MKL_FreeBuffers() function is not currently exposed in NMath, but will be added in the next release. In the meantime, you can add this function to Kernel.cpp in NMath Core, and rebuild:

static void MklFreeBuffers() {
  BLAS_PREFIX(MKL_FreeBuffers());
}

Or, if you want some console output to confirm that memory is being released, try this:

static void MklFreeBuffers() {

  MKL_INT64 AllocatedBytes;
  int N_AllocatedBuffers;

  AllocatedBytes = MKL_MemStat(&N_ AllocatedBuffers);
  System::Console::WriteLine("BEFORE: " + (long)AllocatedBytes + " bytes in " + N_AllocatedBuffers + " buffers");

  BLAS_PREFIX(MKL_FreeBuffers()) ;

  AllocatedBytes = MKL_MemStat(&N_AllocatedBuffers);
  System::Console::WriteLine("AFTER: " + (long)AllocatedBytes + " bytes in " + N_AllocatedBuffers + " buffers");

}

Once you’ve rebuilt NMath Core, you’d use the new method like so:

using CenterSpace.NMath.Kernel;
...
DotNetBlas.MklFreeBuffers();

Note that some care should be taken when calling MklFreeBuffers(), since a drop in performance may occur for any subsequent MKL functions within the same thread, due to reallocation of buffers. Furthermore, given the cost of freeing the buffers themselves, rather than calling MklFreeBuffers() at the end of each thread, it might be more performant to do so after every n threads, or perhaps even based on the total memory usage of your program.

Ken

2 thoughts on “MKL Memory Leak?

  1. Am I missing something here? My understanding is that memory w.r.t. NMath is being managed by the .net __gc. Is this issue more to do with memory that is allocated within the scope of an MKL function call rather than that of the pointers being passed of the .net arrays used to represent NMath vectors/matrices?
    Could you elaborate a bit more on the ownership of the memory that needs to be released?

  2. Hi Sam,

    Yes, that’s right. The code above is required only to release the small amount of working memory allocated within MKL functions. This is unlikely to be an issue unless your application spawns large numbers of threads.

    All memory allocated by NMath and used by MKL is allocated from the managed heap, and will be garbage collected.

    Ken

Leave a Reply

Your email address will not be published. Required fields are marked *

Top