[Mono-list] How to debug: my ASP.NET website hangs with multiple simultaneous requests

[Marvin]

Hi everyone,

I've been struggling with a problem for 4 days and I'm still unable to
solve it. First I will describe the situation.

I've built a website with Asp.NET:
.NET 4
MVC 3; No razor
S#arp Architecture 1.9.6.0
MySQL

I compile my project on Windows with Visual Studio 2010. I then
publish it to a Ubuntu 10.04 x86 virtual machine with Mono 2.10.2
compiled from source. After making some changes to some 3rd party libs
I got everything working. Then I started to load test with JMeter and
I found out that Mono hangs after a few hundred requests when I'm
using more than 1 thread in JMeter. The Mono process will hang with
~100% CPU load and after a minute or 2 it will return a 500 with
Request timed out. The webserver won't process any more requests while
it hangs. I tested this with Apache and xsp4, same results.

I tried to reproduce this with a clean MVC 3 project but it didn't
hang. It really seems to be something in S#arp Architecture.

To ensure it wasn't because of that specific virtual machine I tried
it with Mono 2.10.5 on another virtual machine. Same result.

So I tried to find out why and where in the process it was hanging. I
followed the steps from this article:
http://www.mono-project.com/Debugging#Debugging_Managed_Lock_Deadlocks
because I thought it was because of a deadlock. I attached gdb to the
hanging process and tried some of the commands listed on the page.

Executing "call mono_locks_dump (0)" usually returns this:
Total locks (in 6 array(s)): 1008, used: 391, on freelist: 2, to recycle: 614
or this:
Total locks (in 6 array(s)): 1008, used: 583, on freelist: 93, to recycle: 332

Executing "thread apply all bt" will return a very long list of about
50 threads, some waiting for a lock but most of them are similar to:

Thread 1 (Thread 0xb7725b40 (LWP 2832)):
#0  0xb78cf430 in __kernel_vsyscall ()
#1  0xb788cf5b in read () from /lib/tls/i686/cmov/libpthread.so.0
#2  0x081f9183 in read (handle=0x0, buffer=0x1ef010, numbytes=1024,
bytesread=0xbfbf656c, overlapped=0x0)
    at /usr/include/bits/unistd.h:45
#3  console_read (handle=0x0, buffer=0x1ef010, numbytes=1024,
bytesread=0xbfbf656c, overlapped=0x0) at io.c:1109
#4  0x081fa86d in ReadFile (handle=0xfffffe00, buffer=0x1ef010,
numbytes=1024, bytesread=0xbfbf656c, overlapped=0x0)
    at io.c:2314
#5  0x0823ab9e in ves_icall_System_IO_MonoIO_Read (handle=0x0,
dest=0x1ef000, dest_offset=0, count=1024, error=
    0xbfbf6600) at file-io.c:838
#6  0xb665832a in ?? ()
#7  0xb6658270 in ?? ()
#8  0xb6657cb4 in ?? ()
#9  0xb6657ab2 in ?? ()
#10 0xb66577f3 in ?? ()
#11 0xb66599a4 in ?? ()
#12 0xb6560197 in ?? ()
#13 0xb655e56a in ?? ()
#14 0xb655e48a in ?? ()
#15 0xb655e436 in ?? ()
#16 0xb65533bc in ?? ()
#17 0xb6551e0c in ?? ()
#18 0xb6551c80 in ?? ()
#19 0xb65dbf84 in ?? ()
#20 0xb65dbdfc in ?? ()
#21 0xb714d7ae in ?? ()
#22 0xb714be60 in ?? ()
#23 0xb714bf94 in ?? ()
#24 0x08062c78 in mono_jit_runtime_invoke
(method="Mono.WebServer.XSP.Server:Main ()", obj=0x0,
params=0xbfbf6f1c, exc=
    0x0) at mini.c:5751
#25 0x08171a20 in mono_runtime_invoke
(method="Mono.WebServer.XSP.Server:Main ()", obj=0x0,
params=0xbfbf6f1c, exc=0x0)
    at object.c:2754
#26 0x08175773 in mono_runtime_exec_main
(method="Mono.WebServer.XSP.Server:Main ()", args=0x3be00, exc=0x0)
    at object.c:3929
#27 0x080bb0a5 in main_thread_handler (argc=3, argv=0xbfbf7104) at driver.c:1003
#28 mono_main (argc=3, argv=0xbfbf7104) at driver.c:1855
#29 0x0805944a in mono_main_with_options (argc=3, argv=0xbfbf7104) at main.c:66
#30 main (argc=3, argv=0xbfbf7104) at main.c:97

I'm actually very interested what's going on at the steps 6-23. I
guess these are the calls in my application. Seems gdb is unable to
use the pdb files I supply from Visual Studio. Running pdb2mdb on my
DLL's gives a BadImageFormatException.

I fiddled around a bit with the settings from
http://www.mono-project.com/ThreadPool_DeadLocks but no luck.

Back to Windows...

I installed the latest Mono 2.10.6 on my Windows machine and ran my
website with xsp4. With 1 thread in JMeter it doesn't hang, with 2 or
more it does after some requests. 2 threads will make it hang at 50%,
3 thread 75% and so on, I have a quad core machine. Because of all the
threads hanging at the same time I'm very eager to say it's a deadlock
or something.

I tried to find out what was going wrong by attaching Visual Studio to
the hanging process but it only gives disassembled code.

Process explorer did give me a lot of insight. If I make Mono hang
with 8 threads in JMeter and inspect the process I get these results:
18 threads with 8 of them having >1% cpu usage. These 8 are stuck here:
mono-2.0.dll!mono_thread_interruption_requested+0x4e
mono-2.0.dll!mono_get_special_static_data+0x1a
mono-2.0.dll!mono_error_raise_exception+0x12559
mono-2.0.dll!mono_get_special_static_data+0x1a
mono-2.0.dll!mono_get_special_static_data+0x1a
mono-2.0.dll!mono_get_special_static_data+0x1a
mono-2.0.dll!mono_get_special_static_data+0x1a
mono-2.0.dll!mono_get_special_static_data+0x1a

9 threads are at
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0

1 thread is at
ntdll.dll!NtRequestWaitReplyPort+0xa

But I don't know what to do with this.

The Development Server distributed with Visual Studio actually never
hangs. I haven't tested Windows Server yet.

So my problem is a hanging webserver and I don't know why it's
happening. Could you help me with troubleshooting and solving this?

I hope I've given enough information. I could provide a full report of
"thread apply all bt" if you'd like.

Thank you for your time!