[Mono-list] mod_mono stability problems

[Oscar van Tol]

Hi there and thanks for the advice,

 > > For the past two months we've been hosting a webservice  
application on
 > > apache2 and mod_mono.
 > > Since this week we have some real traffic, and stability issues
 > > started. We now have about 30.000 requests a day, these make the  
mono
 > > process grow in memory usage to about 2gigs. and the response  
becomes
 > > really bad.
 > You might want to increase the thread pool size by setting the  
MONO_THREADS_PER_CPU
 > environment variable to a value higher than the default 5 (the  
formula for
 > calculating the number of threads in the pool is 20 +  
($MONO_THREADS_PER_CPU *
 > number_of_cpus)
The application feels a bit faster now, but with a lot of traffic it  
still sometimes does not come back after the autorestart
 > > The memory problem is probably cause by the none compacting garbage
 > > collector, I solved the performance issue by doing an autorestart
 > > every 400 requests. This works great for a while but after a few  
auto-
 > That sounds quite often - I think 1000 would be quite enough in  
your case.
With about 800 requests it uses 1.5gb of ram and the performance  
decreases..


 > > restarts it lookes live apache failes to spawn a new mod-mono- 
server.
 > > Apache respondes with a 503, when I kill the (old still running)   
mono
 > > process and restart apache it's back.
 > >
 > > Suggestions?
 > It's hard to determine what it is just from the description - it  
might be a
 > deadlock, might be a memory issue with shared memory. Please  
compile mod_mono with
 > - --enable-debug and post apache logs when the 503 happens - that  
should make it
 > easier to diagnose what's going on.
I recompiled on debug, it looks like a deadlock:
[Thu May 15 23:53:40 2008] [warn] No backend found, will start a new  
copy.
[Thu May 15 23:53:42 2008] [warn] Socket file name /tmp/ 
mod_mono_server_global
[Thu May 15 23:53:42 2008] [warn] try_connect: -1
[Thu May 15 23:53:42 2008] [warn] After setup_socket
[Thu May 15 23:53:42 2008] [warn] No backend found, will start a new  
copy.
[Thu May 15 23:53:42 2008] [warn] Acquiring the /tmp/ 
mod_mono_dashboard_XXGLOBAL_1.lock lock for bac
[Thu May 15 23:53:42 2008] [warn] Socket file name /tmp/ 
mod_mono_server_global
[Thu May 15 23:53:42 2008] [warn] try_connect: -1
[Thu May 15 23:53:42 2008] [warn] forking XXGLOBAL
[Thu May 15 23:53:42 2008] [warn] Applications: (null)
[Thu May 15 23:53:42 2008] [warn] Config file: (null)
[Thu May 15 23:53:42 2008] [warn] Config dir.: /etc/mono-server2
[Thu May 15 23:53:42 2008] [warn] Listen port: (null)
[Thu May 15 23:53:42 2008] [warn] Listen address: (null)
[Thu May 15 23:53:42 2008] [warn] child started
[Thu May 15 23:53:42 2008] [warn] PATH: /usr/local/bin:/usr/bin:/bin
[Thu May 15 23:53:42 2008] [warn] serverdir: /opt/mono-1.9/bin
[Thu May 15 23:53:42 2008] [warn] PATH after: /opt/mono-1.9/bin:/usr/ 
local/bin:/usr/bin:/bin
[Thu May 15 23:53:42 2008] [warn] Started new backend, sleeping 2s to  
let it configure
[Thu May 15 23:53:44 2008] [warn] Socket file name /tmp/ 
mod_mono_server_global
[Thu May 15 23:53:44 2008] [warn] try_connect: -1
[Thu May 15 23:53:44 2008] [warn] After setup_socket
[Thu May 15 23:53:44 2008] [warn] No backend found, will start a new  
copy.