[Mono-devel-list] Re: [Bug 71001][Nor] Changed - xsp.exe virtual size grows without bound -- large messages

[jrodman at mono-list.spamportal.net]

On Mon, Jan 31, 2005 at 02:34:56PM -0500, bugzilla-daemon at bugzilla.ximian.com wrote:
> -Status: REOPENED   
> -Resolution: 
> +Status: RESOLVED   
> +Resolution: NOTABUG
>  Severity: Unknown
>  Priority: Normal
>  Component: misc
>  AssignedTo: mono-bugs at ximian.com                            
>  ReportedBy: jrodman at ximian-bugzilla.spamportal.net               
>  QAContact: mono-bugs at ximian.com
> @@ -97,6 +97,22 @@
>  decent amount of uptime or scalability.
>  
>  Nevertheless, we appreciate this issue being investigated and look
>  forward to answers to the questions above.
>  
>  Thanks!
> +
> +------- Additional Comments From miguel at ximian.com  2005-01-31 14:34 -------
> +It is not solvable in the short term.
> +
> +Our GC engine is conservative non-moving, while the Microsoft one
> +is a moving GC which can compact the heap when pieces become unused.
> +
> +It is our plan to address this issue in the long term, but in the
> +short term (Mono 1.2) this will not be solved.
> +
> +This means that Mono has the same problems than an equivalent C or
> +C++ application would (as they do not have compatcing memory 
> +management units either), so you must take the same care that you
> +would with C or C++ when dealing with heap fragmentation.
> +
> +Miguel.

In an attempt to avoid contention over the bug status in bugzilla, while
still not really understanding the problem, I'm replying to the mono
development list.  I hope this is not a problem.

We are currently using the soap interface to provide remote access to a
network service.  Files are submitted, processed, and then reports are
made available.  A few other miscellaneous things can be done along the
way.  

Our goals with this interface are that it be well supported by existing
tools and lanauges, and that it be easy to use.  To this end we are
using SOAP and we are trying to make the steps of the process as simple
as possible.

To this end, we have implemented the submission of a file as a single
function call, rather than a series of calls with state management that
will be difficult to implement in the particular environment a customer
may happen to be using.  This is leading to significant heap growth.

Since complicating the API is undesirable due to the arbitrary
implementers who are unlikely to be seasoned developers, I want to
understand what is causing the fragmentation.  The pattern is: allocate
a large buffer.  Pass this buffer via soap.  Deallocate large buffer.

If I pass the same data with the exact same size repeatedly, I would not
expect to see fragmentation.  Even if the large (say 1MB) buffer is
partially occupied by other datastructures, I do not see why this should
lead to excessive growth, as the 1MB buffer should leave plenty of space
for arbitrary datastructures to live inside.  Further, if it were simply
a matter of fragmentation, I would expect to see the virtual size grow a
good deal, but not so much the resident set size, which also grows at an
alarming rate.  Why do the fragmented pages for the old data not grow
stale?  One would think that with 4k pages that most of the fragmented
space would be able to be swapped out.


Is this behavior pattern resulting solely from my naive allocation, and
subsequent freeing, of large buffers?  Or does it have something to do
with the behavior of the XML serialization?  This email from Gonzalo
suggested so. 

> On Fri, 2004-12-17 at 17:44 +0100, Lluis Sanchez wrote:
> > Hi,
> >
> > My guess is that this is a GC issue. Although the amount of memory
> > used is high, in my tests it looks steady. I'm pretty sure that MS
> > needs to allocate as much memory, but their GC may be performing
> > better.  Sending big chunks of byte arrays as parameters is not a
> > good idea, it has a lot of memory and processing overhead.
> 
> Yes it is. It has to do with code in system.xml that is fine when a
> text element is 'small' but does too many allocations/reallocations
> when the size is big. I already have a simple test case for this and
> will check it on windows and post the details in bugzilla.
> 
> -Gonzalo

If your response in the bug is the result of further investigation,
sorry for bringing this up, but it is not clear to me at this time where
this XML issue is resolved.  I will be making another attempt at finding
it in Subversion later today, but I am still learning how to effectively
locate patches in this tool and project and have not succeeded so far.
For example, I do not know if people always mean the main trunk  when
they say "fixed in subversion".


I'm sorry if this is starting to feel like a technical support request,
but the meat of it is that my program has no lasting allocated
datastructures, and thus I do not see how my code can thus be
responsible for any unbounded increasing fragmentation issue, let alone
memory which does not go stale over time.  Are we certain this is not a
library problem?