[Mono-dev] Issues with GC due to libgc
Miguel de Icaza
miguel at novell.com
Mon Sep 28 18:08:12 EDT 2009
Hello,
I tried your sample, in my machine the memory usage with Case1 does
go up very quickly, but then it tends to stay stable around 600 megs.
Later, I modified the program to not allocate 1 meg blocks, but
instead to allocation 1024 times 1k blocks (so that it allocates the
same amount of memory). The program stays stable at around 200 megs of
ram in this case.
I am sure your program is more complicated, but the difference
between these two patterns of memory usage in my opinion are caused by
memory fragmentation, not really conservative heap scanning.
My suggestion is to change the code in your server to use either
unmanaged buffers for large allocations, or to do buffering with smaller
blocks of memory instead of 10 megabyte blobs.
> Hi all,
>
> After several weeks working on a bunch of mem issues related to the
> libgc based garbage collector, we've identified the following issue and
> a possible solution (Dick already sent some workarounds to the list):
>
> The libgc garbage collector has a really hard time identifying pointers
> to objects since it "guesses" what is a pointer instead of actually
> "knowing" by using data passed by the mono runtime.
>
> It means something as simple as introducing a long on the stack (for
> instance something like array = new int[1000000]) will block (forever)
> the memory at address 1000000. Yes, as incredible as it sounds, it can
> cause important mem problems on long living apps (typically servers).
>
> (As a side note, this exact problem is present on sgen, since it also
> scans the stack "conservatively").
>
> A small improvement could be made in the current GC with little effort,
> and is supplying more class refmaps to libgc.
>
> Libgc is very hard to modify, it contains too many hacks and
> optimizations that have made the code a nightmare to understand and
> modify, so we don't find useful to make anything here beyond very small
> patches.
>
> That said, mono currently can provide reference bitmaps for objects,
> it's a matter of providing the right descriptor to the garbage collector.
>
> Libgc supports this kind of descriptors and mono already generates them
> for the sgen gc, so it's just a matter of joining those together (which
> should beeasy to do). This should improve a great number of scans in the
> arking process, leaving only stacks and several minor objects without
> precise marking. (Should become similar to the current sgen idea, where
> stacks and other roots are scanned conservatively, although not compacting).
>
> Attached is the sample code we use to reproduce the issue on 32 bit
> based Linux/Mono systems.
>
> Some notes about the test app below:
>
>
> =======================================
> the program accepts commands like gc, mem, exit, 2, or 1
>
> 2 n m creates n arrays of ints with m elements, and put them in an
> arraylist. After the call completes, they are no longer referenced.
> 1 n m same, but waiting for a key press after each new array
> gc n performs n gcs
> exit exits
>
> So, the case:
>
> mono test.exe
> > 2 2000000 70 creates 2 million int arrays of 70 elements
> each (virtual goes up to 777MB)
> > gc 10 should free everything, but around 33MB
> remain allocated acording to pmap:
>
> ...
> bf4b5000 32K 0K 0K ---p [anon]
> bfc9e000 88K 32K 28K rwxp [stack]
> ffffe000 4K 0K 0K r-xp [vdso]
> Total: 777820K 33852K 29336K
>
>
> > 2 20 25000000 creates 20 int arrays of 25 million elements
> each (2.7GB)
> > gc 10 now pmap shows everything is screwed up:
>
> ...
> b7f2b000 8K 8K 8K rwxp /lib/ld-2.6.1.so
> bf4b5000 32K 0K 0K ---p [anon]
> bfc9e000 88K 32K 28K rwxp [stack]
> ffffe000 4K 0K 0K r-xp [vdso]
> Total: 2764356K 1696132K 1691616K
>
>
> Trying with smaller sizes lets you see that segments are joined and
> split, but seems that there is some inability to free everything.
>
> ======================================
>
>
> Regards,
>
>
> pablo
>
>
>
> plain text document attachment (Program.cs)
> using System;
> using System.Collections;
>
> namespace test
> {
> class Program
> {
> static void Main(string[] args)
> {
> WaitForEnter();
> }
>
> private static void WaitForEnter()
> {
> Console.WriteLine("Command:");
>
> while (true)
> {
> Console.Write("> ");
> string line = Console.ReadLine();
>
> string[] args = line.Split(' ');
>
> if (args.Length <= 0) continue;
>
> switch (args[0].ToLower())
> {
> case "exit": return;
>
> case "1": Case1(args); break;
>
> case "2": Case2(args); break;
>
>
> case "mem":
> Console.WriteLine("Memory now: {0}", GC.GetTotalMemory(false));
> break;
> case "gc":
> Gcs(args);
> break;
> default:
> Console.WriteLine("Unknown command");
> break;
> }
> }
> }
>
> private static void Gcs(string[] args)
> {
> int loop = (args.Length == 2) ? Int32.Parse(args[1]) : 1;
>
> for (int i = 0; i < loop; ++i)
> {
> Console.WriteLine("Memory {1} now : {0}", GC.GetTotalMemory(false), i);
> Console.WriteLine("Memory {1} after GC: {0}", GC.GetTotalMemory(true), i);
> }
> }
>
> private const int OneMeg = 1024 * 1024;
>
> private static void Case1(string[] args)
> {
> int loop = (args.Length >= 2) ? Int32.Parse(args[1]) : 5;
> int size = (args.Length >= 3) ? Int32.Parse(args[2]) : 10 * OneMeg;
>
> ArrayList container = new ArrayList();
>
> for (int i = 0; i < loop; ++i)
> {
> int[] s1 = new int[size];
>
> for (int j = 0; j < size; ++j)
> {
> s1[j] = j;
> }
>
> container.Add(s1);
>
> Console.Write("Iteration {0}, press enter for next", i);
> Console.ReadLine();
> }
>
> // Explicit in case it helps
> container = null;
> }
>
> private static void Case2(string[] args)
> {
> int loop = (args.Length >= 2) ? Int32.Parse(args[1]) : 5;
> int size = (args.Length >= 3) ? Int32.Parse(args[2]) : 10 * OneMeg;
>
> ArrayList container = new ArrayList();
>
> for (int i = 0; i < loop; ++i)
> {
> int[] s1 = new int[size];
>
> for (int j = 0; j < size; ++j)
> {
> s1[j] = j;
> }
>
> container.Add(s1);
>
> Console.WriteLine("Iteration {0}", i);
> }
>
> // Explicit in case it helps
> container = null;
> }
>
>
> }
> }
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list
More information about the Mono-devel-list
mailing list