[Mono-aspnet-list] issues with intermittent apache hangups
Dan
parnham at gmail.com
Tue Apr 5 16:24:25 EDT 2011
This appears to have fixed the problem but may have uncovered a more sinister
bug. With the patch applied it seems our application now runs out of memory
(so the previously seen exception was actually being triggered by a memory
leak).
Just to give a bit of background, the system we are developing consists of
an MVC web application with a kiosked browser continuously accessing it, the
client-side of the application polls (using ajax) the server-side every
half-second or so. Therefore this a bit of an unconventional setup since the
web session belonging to the client lasts for a long time.
This appears to have fixed the problem but may have uncovered a more
sinister bug. With the patch applied it seems our application now runs out
of memory (so the previously seen exception was actually being triggered by
a memory leak).
Just to give a bit of background, the system we are developing consists of
an MVC web application with a kiosked browser continuously accessing it, the
client-side of the application polls the server-side using ajax every
half-second or so. Therefore this a bit of an unconventional setup since the
web session belonging to the client lasts for a very long time.
At first I thought that something within our application was causing the
memory leak, but now I'm not so sure. I first tried updating all of the
third-party libraries we use to the latest versions, but that didn't make a
difference.
Next I hacked the xsp startup script on my development machine to enable the
heapshot profiler and ran the application for about 45 mins. The results
were as follows:
At 60 seconds in:
110760 923 120 System.Web.Caching.CacheItem (bytes: +106200,
count: +885)
At 1000 seconds in:
2191080 18259 120 System.Web.Caching.CacheItem (bytes: +107280,
count: +894)
At 2300 seconds in:
4971120 41426 120 System.Web.Caching.CacheItem (bytes: +119040,
count: +992)
By the end of the test CacheItem was on top for memory usage and instance
count (even managed to beat System.String). It seems that these cache items
are never being garbage collected!
With traces enabled in mprof-report I gleaned this:
2947320 24561 120 System.Web.Caching.CacheItem (bytes: +123600,
count: +1030)
51 root references (0 pinning)
24554 references from: System.Web.Caching.CacheItem[]
34 references from: System.Collections.Generic.LinkedListNode
which doesn't help too much.
So my next step was to hack the System.Web.Caching.Cache so that it logged
every time an item was added to the timedItems queue. The log was filled
with these (always an InProcSessionItem):
Enqueued timed cache item @@@InProc at AE0F3F94D18773EB8CC2821E with value
System.Web.SessionState.InProcSessionItem expiring at 05/04/2011 19:21:57
I also added a stacktrace log at that point so that we could see what was
actually adding the InProc items:
at System.Environment.get_StackTrace() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/corlib/System/Environment.cs:line
243
at System.Web.Caching.Cache.EnqueueTimedItem(System.Web.Caching.CacheItem
item) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
373
at System.Web.Caching.Cache.SetItemTimeout(System.Web.Caching.CacheItem
ci, DateTime absoluteExpiration, TimeSpan slidingExpiration,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback,
System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, System.String
key, Boolean doLock) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
336
at System.Web.Caching.Cache.Insert(System.String key, System.Object
value, System.Web.Caching.CacheDependency dependencies, DateTime
absoluteExpiration, TimeSpan slidingExpiration, CacheItemPriority priority,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback,
System.Web.Caching.CacheItemUpdateCallback onUpdateCallback, Boolean doLock)
in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
287
at System.Web.Caching.Cache.Insert(System.String key, System.Object
value, System.Web.Caching.CacheDependency dependencies, DateTime
absoluteExpiration, TimeSpan slidingExpiration, CacheItemPriority priority,
System.Web.Caching.CacheItemRemovedCallback onRemoveCallback) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.Caching/Cache.cs:line
261
at
System.Web.SessionState.SessionInProcHandler.InsertSessionItem(System.Web.SessionState.InProcSessionItem
item, Int32 timeout, System.String id) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionInProcHandler.cs:line
111
at
System.Web.SessionState.SessionInProcHandler.ResetItemTimeout(System.Web.HttpContext
context, System.String id) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionInProcHandler.cs:line
326
at
System.Web.SessionState.SessionStateModule.OnReleaseRequestState(System.Object
o, System.EventArgs args) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web.SessionState_2.0/SessionStateModule.cs:line
297
at System.Web.HttpApplication+c__Iterator2.MoveNext() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1044
at System.Web.HttpApplication+c__Iterator3.MoveNext() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1403
at System.Web.HttpApplication.Tick() in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
926
at System.Web.HttpApplication.Start(System.Object x) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1562
at
System.Web.HttpApplication.System.Web.IHttpHandler.ProcessRequest(System.Web.HttpContext
context) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpApplication.cs:line
1639
at System.Web.HttpRuntime.Process(System.Web.HttpWorkerRequest req) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
531
at System.Web.HttpRuntime.RealProcessRequest(System.Object o) in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
456
at System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest wr)
in
/home/dan/Downloads/Mono/Base/mono-2.10.1/mcs/class/System.Web/System.Web/HttpRuntime.cs:line
562
at Mono.WebServer.MonoWorkerRequest.ProcessRequest() in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer/MonoWorkerRequest.cs:line
400
at
Mono.WebServer.BaseApplicationHost.ProcessRequest(Mono.WebServer.MonoWorkerRequest
mwr) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer/BaseApplicationHost.cs:line
121
at Mono.WebServer.XSPApplicationHost.ProcessRequest(Int32 reqId,
System.Net.IPEndPoint localEP, System.Net.IPEndPoint remoteEP, System.String
verb, System.String path, System.String queryString, System.String protocol,
System.Byte[] inputBuffer, System.String redirect, IntPtr socket,
Mono.WebServer.SslInformation ssl) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer.XSP/XSPApplicationHost.cs:line
115
at Mono.WebServer.XSPWorker.RunInternal(System.Object state) in
/home/dan/Downloads/Mono/XSP/xsp-2.10/src/Mono.WebServer.XSP/XSPWorker.cs:line
193
Hopefully somebody who understands how HttpApplication works can provide
some insight into what is happening and whether it is a bug in mono or how
we are using it?
Regards,
Dan
Dan wrote:
>
> I've applied just the changes in that commit to our 2.10.1 installation
> and have our test system back up and running again. It would usually crash
> within 1-3 days, so hopefully I will be able to confirm whether or not
> that seems to have done the trick by the end of the week.
>
> Many thanks,
> Dan
>
>
> Marek Habersack wrote:
>>
>> This should be fixed by the commits:
>>
>> 2ec75014bdf04074faeb4126f26dd3f71bb90c9d (master)
>> e57e16fee47961b8aaa2b0b22d70617b1a5b769f (mono-2-10)
>>
>> Thanks to Rodrigo Kumpera for spotting the issue :)
>>
>> marek
>>
>
--
View this message in context: http://mono.1490590.n4.nabble.com/issues-with-intermittent-apache-hangups-tp3264509p3429084.html
Sent from the Mono - ASP.NET mailing list archive at Nabble.com.
More information about the Mono-aspnet-list
mailing list