[Mono-devel-list] Re: [Bug 71001][Nor] Changed - xsp.exe virtual size grows without bound -- large messages

[Ben Maurer]

On Mon, 2005-01-31 at 16:14 -0800, jrodman at mono-list.spamportal.net
wrote:
> If I pass the same data with the exact same size repeatedly, I would not
> expect to see fragmentation.  Even if the large (say 1MB) buffer is
> partially occupied by other datastructures, I do not see why this should
> lead to excessive growth, as the 1MB buffer should leave plenty of space
> for arbitrary datastructures to live inside.  Further, if it were simply
> a matter of fragmentation, I would expect to see the virtual size grow a
> good deal, but not so much the resident set size, which also grows at an
> alarming rate.  Why do the fragmented pages for the old data not grow
> stale?  One would think that with 4k pages that most of the fragmented
> space would be able to be swapped out.

There are various factors causing this fragmentation. Basically, a
conservative GC is not going to do a great job on this.

A simpler test case is:

using System.Xml;

class X {
	static void Main ()
	{
		while (true) {
			byte [] b = new byte [1024*1024*50];
			XmlDocument d = new XmlDocument ();
			d.Load ("test.rdf");
		}
	}
}

The Xml document here is a huge XML document. It ensures that data of
various sizes gets put on the stack. When I run this test case, the heap
size grows without bound (it takes a bit longer than your test case),
but the number of live bytes says about constant. This means that there
is heap fragmentation here.

This will happen for different variations of the test cases

In terms of fragmentation, mmaping out large blocks, like malloc does,
might help. It'd be interesting to try mmap (and munmap) support and see
how it affects this.

The other thing that might help out this test case is the issue that
allocations in non-root domains are scanned 100% conservatively. I know
that Paolo submitted a patch for boehm to allow heap walking, which is
the first step in fixing this.

Doing these two things isnt a walk in the park, but they are a tad
easier than doing a new GC :-).

-- Ben