[Mono-list] Garbage collection and memory usage

[Mike Glenn]

Just to add some info to the discussion, I just put a webmail application in
production this week which is handling around 30,000+ logins a day using a
custom written pop3 library. Here's the details of the setup:

Application Code is written using the Castle Monorail web framework (think ruby
on rails for .Net): http://www.castleproject.org/index.php/Main_Page

Dell 2850 Dual 2.8GHz Xeon Processors (HyperThreading Disabled)

8 Gig of RAM

136 Gig of Raid 5 drive space

Gentoo 2006.0 sync'd and emerged to Sundays Releases and nptl enabled.

2.6.15 Gentoo-Source Kernel

Mono 1.1.13.4 (custom ebuild for latest version)

XSP and Mod_Mono 1.1.13 (custom ebuild for latest version)

Apache 2.0.55-r2 (MPM Worker)

My Pop3 Library, I plan to have the website for this back up sometime real soon
at www.ilude.com.

Apache mono config lines:

        MonoRunXSP True
        #MonoDebug  True
        MonoAutoApplication disabled

        <IfModule !mod_mono.c>
        	LoadModule mono_module modules/mod_mono.so
        </IfModule>

        <IfModule mod_mime.c>
            # there's a long story behind this file extention
            # but this is not the place to discuss it.
        	AddHandler mono .asp
        </IfModule>

        AddMonoApplications default "/:/web/webmail"

        <Location /webmail>
        	SetHandler mono
        </Location>

        <Location /admin>
        	SetHandler mono
        </Location>


On launch day I ran into some OutOfMemoryException's. I rewrote the attachment
processing of the pop3 library to write out the raw network streams to disk and
then read that temp file back in, using more reasonable chunks while performing
base64 or quoted-printable decoding and writing the results out to another temp
file used to cache the decoded attachment until the user requests it. Previously
I used a StringBuilder to store the network stream. When I started decoding the
data in the StringBuilder a second StringBuilder, which was used as an output
buffer while cleaning out non-base64 characters from the stream, was created.
This object creation would throw the OutOfMemoryException during the internal
character array allocation. (I'm sorry I didn't save the exact stacktrace). This
occurred both with a normal new StringBuilder() call as well as with the
StringBuilder(int size) call. These Exceptions would start when top reported
around 2.3 Gigs of Resident memory in use by the mod-mono-server.exe process.
This started occurring on three different occasions after restarting the
mod-mono-server.exe, each time top was reporting 2.3 Gig of resident memory use.
While these exceptions where being thrown for some users, others where still
able to log in. 

Now that the process uses disk caching, max resident memory gets up to around
1.7 Gig reported by top at peak times. 

Here's a current screen cap of top (load is currently dropping off for the day):

top - 22:48:45 up 4 days, 10:48,  8 users,  load average: 0.55, 0.37, 0.27
Tasks:  84 total,   2 running,  82 sleeping,   0 stopped,   0 zombie
Cpu(s): 24.6% us,  2.5% sy,  0.0% ni, 72.1% id,  0.0% wa,  0.2% hi,  0.7% si
Mem:   8312740k total,  2871220k used,  5441520k free,   180996k buffers
Swap:   506036k total,        0k used,   506036k free,   697836k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16006 apache    16   0 98244  11m 2516 S   19  0.1   0:12.59 apache2
 4447 apache    15   0 1248m 1.2g 9776 S   18 14.8  22:06.23 mono
12449 apache    18   0  104m  20m 2520 S    7  0.3   0:50.89 apache2
14955 apache    17   0 99.9m  15m 2524 S    7  0.2   0:23.47 apache2
13985 apache    18   0  100m  16m 2516 S    1  0.2   0:32.85 apache2


Now a few other interesting things I discovered. With hyperthreading enabled
mono was much less stable, crashing with a native stacktrace pointing to the
pthread.so glibc lib. I'm fairly certain this is related to a race condition
somewhere in the Novell Ldap library (I've reported an issue similar to this to
the Dev list and submitted a code fix from JD Conley which Konstantin Triger
refined and committed in Feb. That fix greatly improved stability with the
library but I believe there's still an issue related to heavy use on an SMP
machine that is further exaggerated when hyperthreading is enabled). With
hyperthreading disabled and some caching of ldap lookup results to prevent
excessive calls the Novell lib I've not had any serious issues with mono
crashing unexpectedly.

I apologize for not having more details, I'm hoping to get sometime soon to try
and put together some code that could reproduce some of thses  issues using more
generalized data sources. If there's any questions I can answer that may help
please let me know. Anything I can do to help mono scale to these and higher
levels... 

As a side note the webmail application on mono is already showing more stability
than a much less used app on MS 2003 and IIS, but I'll admit that it was my
first production .net app and some of the code and design decisions are less
than pretty :-)

Mike Glenn 
Senior Web Developer
The Armstrong Group of Companies