[Mono-aspnet-list] issues with intermittent apache hangups

[Marek Habersack]

On Tue, 5 Apr 2011 15:24:25 -0500 (CDT)
Dan <parnham at gmail.com> wrote:

Hey Dan,
> This appears to have fixed the problem but may have uncovered a more sinister
> bug. With the patch applied it seems our application now runs out of memory
> (so the previously seen exception was actually being triggered by a memory
> leak). 
[snip]
> At first I thought that something within our application was causing the
> memory leak, but now I'm not so sure. I first tried updating all of the
> third-party libraries we use to the latest versions, but that didn't make a
> difference. 
> 
> Next I hacked the xsp startup script on my development machine to enable the
> heapshot profiler and ran the application for about 45 mins. The results
> were as follows:
[snip]
> By the end of the test CacheItem was on top for memory usage and instance
> count (even managed to beat System.String). It seems that these cache items
> are never being garbage collected!
> 
> With traces enabled in mprof-report I gleaned this:
> 2947320      24561      120 System.Web.Caching.CacheItem (bytes: +123600,
> count: +1030)
> 		51 root references (0 pinning)
> 		24554 references from: System.Web.Caching.CacheItem[]
> 		34 references from: System.Collections.Generic.LinkedListNode
> 
> which doesn't help too much.
> 
> So my next step was to hack the System.Web.Caching.Cache so that it logged
> every time an item was added to the timedItems queue. The log was filled
> with these (always an InProcSessionItem):
> 
> Enqueued timed cache item @@@InProc at AE0F3F94D18773EB8CC2821E with value
> System.Web.SessionState.InProcSessionItem expiring at 05/04/2011 19:21:57
[snip]
> Hopefully somebody who understands how HttpApplication works can provide
> some insight into what is happening and whether it is a bug in mono or how
> we are using it?
The issue here was the way the in-proc session handler handled item timeout reset. It would first
remove the item from the cache (it uses an internal instance of the Cache class) and then insert
it back in the cache. With a big number of items, that would grow the timed items priority queue
heap much more than was necessary and generate a lot of duplicate internal cache items. The effects
would be different depending on the application's usage of session data.
I've just committed a possible fix for the issue to Mono's master branch (I haven't committed it to
the mono-2-10 branch since it is a potentially breaking change which needs to be tested in real
life first) in commit 0eff07d0fc4ef1ef3c584e71a8dd54ab708543e8 (tests have been committed in
c00c5d00fddf18969c9869492f7e1235d960076b if you're interested). The fix applies cleanly to the 2.10
branch, so please apply it and test with your application to see whether it fixed the issue. Please
let me know about the results, thanks :)

marek

> Dan wrote:
> > 
> > I've applied just the changes in that commit to our 2.10.1 installation
> > and have our test system back up and running again. It would usually crash
> > within 1-3 days, so hopefully I will be able to confirm whether or not
> > that seems to have done the trick by the end of the week.
> > 
> > Many thanks,
> > Dan
> > 
> > 
> > Marek Habersack wrote:
> >> 
> >> This should be fixed by the commits:
> >> 
> >> 2ec75014bdf04074faeb4126f26dd3f71bb90c9d (master)
> >> e57e16fee47961b8aaa2b0b22d70617b1a5b769f (mono-2-10)
> >> 
> >> Thanks to Rodrigo Kumpera for spotting the issue :)
> >> 
> >> marek
> >> 
> > 
> 
> 
> --
> View this message in context:
> http://mono.1490590.n4.nabble.com/issues-with-intermittent-apache-hangups-tp3264509p3429084.html
> Sent from the Mono - ASP.NET mailing list archive at Nabble.com.
> _______________________________________________ Mono-aspnet-list mailing list
> Mono-aspnet-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-aspnet-list