[Mono-aspnet-list] mod-mono-server2 hangs

Antonio Anzivino anzivino at studenti.unina.it
Sun Dec 6 19:23:18 EST 2009


Hello,
I'm building a website with ASP.NET but I found that mod_mono hangs after a
certain period of time, varying from 5 minutes to 1 hour, but it does
EVENTUALLY.

I'm running openSUSE 11.1 i586 on an OpenVZ VPS with 256/512MB of RAM (256
guaranteed, 512 burst), and I haven't currently ran out of memory.

My webserver configuration is:
Apache 2.2.14-2.1 configured for NameVirtualHost, with mod_mono on a socket
mod-mono-server2 2.4.2.0 process running on socket as a specific system user
with init script (sudo -u djechelon mod-mono-server2 etc.) for filesystem
permissions reasons

My application mainly uses Gaia Ajax WebWidgets, NHibernate and log4net.
I analyzed the project with MoMa and it was not obviously all green as for
the project dependencies. Most errors are due to Design non-implementation,
but I develop with Visual Studio, so I don't worry about it.

I have run the mod_mono server with output onto a log file.
After a very long period of "work" (best say "hang without restart") I found
this in the output

Handling exception type ExecutionEngineException
Message is Couldn't create thread
IsTerminating is set to True
System.ExecutionEngineException: Couldn't create thread
  at (wrapper managed-to-native)
object:__icall_wrapper_mono_delegate_begin_invoke (object,intptr)
  at (wrapper delegate-begin-invoke)
System.IO.FileSystemEventHandler:begin_invoke_IAsyncResult__this___object_Fi
leSystemEventArgs_AsyncCallback_object
(object,System.IO.FileSystemEventArgs,System.AsyncCallback,object)
  at System.IO.FileSystemWatcher.RaiseEvent (System.Delegate ev,
System.EventArgs arg, EventType evtype) [0x00000] 
  at System.IO.FileSystemWatcher.OnChanged (System.IO.FileSystemEventArgs e)
[0x00000] 
  at System.IO.FileSystemWatcher.DispatchEvents (FileAction act,
System.String filename, System.IO.RenamedEventArgs& renamed) [0x00000] 
  at (wrapper remoting-invoke-with-check)
System.IO.FileSystemWatcher:DispatchEvents
(System.IO.FileAction,string,System.IO.RenamedEventArgs&)
  at System.IO.InotifyWatcher.ProcessEvents (System.Byte[] buffer, Int32
length) [0x00000] 
  at System.IO.InotifyWatcher.Monitor () [0x00000]

and the server was crashed.

In this moment, the server is hanging but the output log report only
NHibernate's queries, as it's probably misconfigured to write them not only
to ASP.NET trace and a file but also to console.

Other information that could be useful: my application crashed even before
switching to the stand-alone server (when I didn't put anything into
Apache's VirtualHost configuration and let it do all the thing); pages are
now marked asynchronous, but the server crashes no matter that; I do use
multithreading from inside the application, as a couple of background
threads are spawned from Global.asax: I triple-checked that NO race
condition/deadlock is generated from MY code; I'm sure that I don't hit the
limit of 20 concurrent requests in mod-mono-server2 as I'm the only user
that can physically access the application from browser since it's not
public; I develop the code on Windows and upload it to an SVN repository,
then deploy the code by (1) turning off mod-mono, (2) updating SVN, (3)
deleting bin directories of all assemblies, (4) run xbuild on each Visual
Studio project in the order of dependencies without getting errors, (5)
start mod-mono.

I tried to re-create the same environment on my laptop running openSUSE 11.2
and the latest version of Mono, using a new virtual host that maps on
localhost and the same directory layout, but I haven't got enough time to
detect a failure.
I'm not giving the URL of the application as it is currently in development
and there are known security holes (Trace.axd is currently accessible from
"anywhere" and you will read my DB credentials from NHibernate, access to
some admin functions without login, hahahahaha :-), so I told Apache to
allow only my home IP).
By the way, I'm double sure that the fault is not in my code because even
Trace.axd becomes unaccessible when the server hangs, else I wouldn't
theorically be able to access only my code. Running mono-ctrl reports
pending requests.
The only thing to do at this point is to restart mod_mono server, but it has
to be done manually and OBVIOUSLY it takes tome time for the application to
re-compile.

I would like to debug the server, but I'm not an expert in that. I installed
(from zypper) the -debug and -debugsource packages for Mono so I hope the
next stack trace will provide me with more information. I tried to run mdb
/usr/lib/[.]/mod-mono-server2.exe passing all the arguments required to
properly start the ASP.NET application but I had no success, or at least I'm
good only in debugging by Visual Studio and Eclipse.

I now need to be sure if the problem is related to memory before buying new
expensive RAM. I'm not sure if the exception I reported occurred after a
kernel exception in memory allocation, but I found no "cannot allocate
memory" message in system logs and I forgot to see if
/proc/user_beancounters reported failure before rebooting the whole server
(now I will and let you know: no fault yet anyway).

I found a number of messages about mod_mono hanging, and some patch code,
but since they were dated 2008 I presume those patches were implemented into
the trunk and release versions, weren't them?

Thank you in advance for your help.
Antonio Anzivino
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4496 bytes
Desc: not available
Url : http://lists.ximian.com/pipermail/mono-aspnet-list/attachments/20091207/f8a7869e/attachment.bin 


More information about the Mono-aspnet-list mailing list