3

I have a C# application that does a few bits and pieces, but the main task it performs is done by a Delphi DLL which it calls.

This Delphi DLL is a total memory hog, which needs to cache a lot of DB-information locally for speed. I'm happy that it's not leaky, as FastMM4 isn't reporting any memory leaks when the code is run within Delphi.

I am starting to run into problems, however, when the control is returned to C#. The C# code attempts to do some calculations on the results of the Delphi app (all results marshalled via a DB). These calculations usually involve a million or so doubles so not extreme memory usage, however the code keeps returning me out of memory exceptions.

I assume that FastMM4 in the Delphi code still hasn't returned the freed memory to Windows (and hence available to the C# code), so the process is still using it's maximum 32-bit memory allocation and C# can't obtain more when it needs to.

So, how do I get the memory used (and freed) by Delphi usable again by the C# code? I thought we may want to do one of the following:

  • Force an unload of the Delphi DLL from the C# side (my colleague doesn't think this will work, as he thinks it'll just unload the code rather than the memory used on the heap) - probably LoadLibrary/FreeLibrary?
  • Make a call at the end of the Delphi DLL to release the memory back to Windows (I tried SetWorkingProcessSetSize before, but didn't seem to do anything, should I use a different call?)
  • Wrap the Delphi DLL in a C# DLL and call it in a different AppDomain (I don't like this from a style perspective as we're creating wrappers just to hold wrappers.
  • Anything else I've missed?
Matt Allwood
  • 1,448
  • 12
  • 25
  • How do you store the results in your C# code? Maybe it can't allocate the huge contiguous storage for your data due to memory fragmentation? – Ilya Polenov Aug 18 '15 at 09:15
  • Unloading the DLL will shut down the Delphi heap and return all the memory to the system. – David Heffernan Aug 18 '15 at 09:16
  • @IlyaPolenov The C# code just pulls down some data using standard DB and Linq routines, nothing special. – Matt Allwood Aug 18 '15 at 09:19
  • @DavidHeffernan We'll give that a go, that was my understanding, but my colleague thought that the heap memory would remain and just the code portion would be got rid of. Thanks – Matt Allwood Aug 18 '15 at 09:21
  • 3
    @MattAllwood No, the Delphi memory manager will tidy up and return all the memory to the system. – David Heffernan Aug 18 '15 at 09:26
  • What calculations are you trying to perform after delphi is done. And what are the value ranges within the doubles? – Nick Otten Aug 18 '15 at 10:43
  • @NickOtten Summing, averaging and sorting - nothing particularly mind-blowing. It's all financial values, so a few million-billion depending on the currency. It looks as though freeing the DLL via a WinAPI call is working (the memory usage shows a big drop), though it takes a few hours to re-run a test that was causing us problems. – Matt Allwood Aug 18 '15 at 11:03
  • @Matt Allwood: You could write a small class to write the doubles to a file(s). You could then load those back into your model from a stream reader so you can do system test without having to wait for the database. Anyway I hope it works for you now. If you still run into errors I would look at the sorting. You might overflow a list or recursive structure (depending on your implementation of it) – Nick Otten Aug 18 '15 at 11:05
  • @NickOtten it's not speed that's the issue (the Delphi process takes ~3-6h, the C# takes about 20s) the part that is causing us a problem is that when the C# tries to obtain memory to do the final calculations. Reading from file vs database wouldn't address this – Matt Allwood Aug 18 '15 at 11:09

2 Answers2

1

Force an unload of the Delphi DLL from the C# side (my colleague doesn't think this will work, as he thinks it'll just unload the code rather than the memory used on the heap) - probably LoadLibrary/FreeLibrary?

This will just work. When the DLL unloads, FastMM will finalize and return the memory that it reserved and committed.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
0

One thing I would do is make a call to GC.Collect before calling your library. .NET knows what to do when more managed memory is requested than can fit and calls the collector automatically, however it has no clue what you're doing in native code so there will be a lot of memory allocated needlessly.

I would also move from a 32 bit architecture. It's not that you ran out of memory, it's that you ran out of consecutive memory large enough to fit whatever you're trying to do in Delphi. A larger virtual address space will fix that issue for you, and there hasn't been a processor made in the past 6 years that didn't wasn't 64 bit capable. It's time to take those shy steps into the bright future ahead of us.

Blindy
  • 65,249
  • 10
  • 91
  • 131
  • As he appears to be Delphi XE (based on the tag), he can't build a 64bit version, as the 64bit Delphi compiler wasn't introduced until a version later in XE2 – Gerry Coll Aug 19 '15 at 11:19
  • Sure, you can't build 64bit executables with the Turbo C++ compiler either, but people have managed to somehow. Usually by using a newer compiler. – Blindy Aug 20 '15 at 01:18