[Mono-aspnet-list] issues with intermittent apache hangups

Dan parnham at gmail.com
Tue Apr 5 16:24:25 EDT 2011 

This appears to have fixed the problem but may have uncovered a more sinister
bug. With the patch applied it seems our application now runs out of memory
(so the previously seen exception was actually being triggered by a memory
leak). 

Just to give a bit of background, the system we are developing consists of
an MVC web application with a kiosked browser continuously accessing it, the
client-side of the application polls (using ajax) the server-side every
half-second or so. Therefore this a bit of an unconventional setup since the
web session belonging to the client lasts for a long time.

This appears to have fixed the problem but may have uncovered a more
sinister bug. With the patch applied it seems our application now runs out
of memory (so the previously seen exception was actually being triggered by
a memory leak). 

Just to give a bit of background, the system we are developing consists of
an MVC web application with a kiosked browser continuously accessing it, the
client-side of the application polls the server-side using ajax every
half-second or so. Therefore this a bit of an unconventional setup since the
web session belonging to the client lasts for a very long time.

At first I thought that something within our application was causing the
memory leak, but now I'm not so sure. I first tried updating all of the
third-party libraries we use to the latest versions, but that didn't make a
difference. 

Next I hacked the xsp startup script on my development machine to enable the
heapshot profiler and ran the application for about 45 mins. The results
were as follows:

At 60 seconds in:
110760        923      120 System.Web.Caching.CacheItem (bytes: +106200,
count: +885)

At 1000 seconds in:
2191080      18259      120 System.Web.Caching.CacheItem (bytes: +107280,
count: +894)

At 2300 seconds in:
4971120      41426      120 System.Web.Caching.CacheItem (bytes: +119040,
count: +992)

By the end of the test CacheItem was on top for memory usage and instance
count (even managed to beat System.String). It seems that these cache items
are never being garbage collected!

With traces enabled in mprof-report I gleaned this:
2947320      24561      120 System.Web.Caching.CacheItem (bytes: +123600,
count: +1030)
		51 root references (0 pinning)
		24554 references from: System.Web.Caching.CacheItem[]
		34 references from: System.Collections.Generic.LinkedListNode

which doesn't help too much.

So my next step was to hack the System.Web.Caching.Cache so that it logged
every time an item was added to the timedItems queue. The log was filled
with these (always an InProcSessionItem):

Enqueued timed cache item @@@InProc at AE0F3F94D18773EB8CC2821E with value
System.Web.SessionState.InProcSessionItem expiring at 05/04/2011 19:21:57

I also added a stacktrace log at that point so that we could see what was
actually adding the InProc items:
   at System.Environment.get_StackTrace() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/corlib/System/Environment.cs:line
243
   at System.Web.Caching.Cache.EnqueueTimedItem(System.Web.Caching.CacheItem
item) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
373
   at System.Web.Caching.Cache.SetItemTimeout(System.Web.Caching.CacheItem
ci, DateTime absoluteExpiration, TimeSpan slidingExpiration,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback,
System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String
key, Boolean doLock) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
336
   at System.Web.Caching.Cache.Insert(System.String key, System.Object
value, System.Web.Caching.CacheDependency dependencies, DateTime
absoluteExpiration, TimeSpan slidingExpiration, CacheItemPriority priority,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback,
System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, Boolean doLock)
in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
287
   at System.Web.Caching.Cache.Insert(System.String key, System.Object
value, System.Web.Caching.CacheDependency dependencies, DateTime
absoluteExpiration, TimeSpan slidingExpiration, CacheItemPriority priority,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
261
   at
System.Web.SessionState.SessionInProcHandler.InsertSessionItem(System.Web.SessionState.InProcSessionItem
item, Int32 timeout, System.String id) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionInProcHandler.cs:line
111
   at
System.Web.SessionState.SessionInProcHandler.ResetItemTimeout(System.Web.HttpContext
context, System.String id) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionInProcHandler.cs:line
326
   at
System.Web.SessionState.SessionStateModule.OnReleaseRequestState(System.Object
o, System.EventArgs args) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionStateModule.cs:line
297
   at System.Web.HttpApplication+c__Iterator2.MoveNext() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1044
   at System.Web.HttpApplication+c__Iterator3.MoveNext() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1403
   at System.Web.HttpApplication.Tick() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
926
   at System.Web.HttpApplication.Start(System.Object x) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1562
   at
System.Web.HttpApplication.System.Web.IHttpHandler.ProcessRequest(System.Web.HttpContext
context) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1639
   at System.Web.HttpRuntime.Process(System.Web.HttpWorkerRequest req) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
531
   at System.Web.HttpRuntime.RealProcessRequest(System.Object o) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
456
   at System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest wr)
in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
562
   at Mono.WebServer.MonoWorkerRequest.ProcessRequest() in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer/MonoWorkerRequest.cs:line
400
   at
Mono.WebServer.BaseApplicationHost.ProcessRequest(Mono.WebServer.MonoWorkerRequest
mwr) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer/BaseApplicationHost.cs:line
121
   at Mono.WebServer.XSPApplicationHost.ProcessRequest(Int32 reqId,
System.Net.IPEndPoint localEP, System.Net.IPEndPoint remoteEP, System.String
verb, System.String path, System.String queryString, System.String protocol,
System.Byte[] inputBuffer, System.String redirect, IntPtr socket,
Mono.WebServer.SslInformation ssl) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer.XSP/XSPApplicationHost.cs:line
115
   at Mono.WebServer.XSPWorker.RunInternal(System.Object state) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer.XSP/XSPWorker.cs:line
193

Hopefully somebody who understands how HttpApplication works can provide
some insight into what is happening and whether it is a bug in mono or how
we are using it?

Regards,
Dan


Dan wrote:
> 
> I've applied just the changes in that commit to our 2.10.1 installation
> and have our test system back up and running again. It would usually crash
> within 1-3 days, so hopefully I will be able to confirm whether or not
> that seems to have done the trick by the end of the week.
> 
> Many thanks,
> Dan
> 
> 
> Marek Habersack wrote:
>> 
>> This should be fixed by the commits:
>> 
>> 2ec75014bdf04074faeb4126f26dd3f71bb90c9d (master)
>> e57e16fee47961b8aaa2b0b22d70617b1a5b769f (mono-2-10)
>> 
>> Thanks to Rodrigo Kumpera for spotting the issue :)
>> 
>> marek
>> 
> 


--
View this message in context: http://mono.1490590.n4.nabble.com/issues-with-intermittent-apache-hangups-tp3264509p3429084.html
Sent from the Mono - ASP.NET mailing list archive at Nabble.com.

More information about the Mono-aspnet-list mailing list