[Mono-dev] NancyFX self hosting (HttpListener) locking up on linux

Sun Aug 4 14:43:06 UTC 2013

Hi Esben.

Certainly a good suggestion. I've tried increasing it:

nitoback at mulder:~$ ulimit -Sn
200000

nitoback at mulder:~$ ulimit -Hn
200000

Doesn't really have any effect. Here's what happens when I start hammering it - regarding connections - The self host is running on port 9591:

root at mulder:~# lsof -n | grep -i 9591 | wc -l
93
root at mulder:~# lsof -n | grep -i 9591 | wc -l
642
root at mulder:~# lsof -n | grep -i 9591 | wc -l
573
root at mulder:~# lsof -n | grep -i 9591 | wc -l
677
root at mulder:~# lsof -n | grep -i 9591 | wc -l
1270
root at mulder:~# lsof -n | grep -i 9591 | wc -l
1270

The connection count goes up to 1270 and at that point it hangs, this is slightly different each time I run this.

After having stopped the tests and 10 minutes after it:

root at mulder:~# lsof -n | grep -i 9591 | wc -l
1170

Here is an example of the connections that are in the CLOSE_WAIT state:

mono      2788 2840   nitoback   71u     IPv4              65081      0t0        TCP 127.0.0.1:9591->127.0.0.1:40372 (CLOSE_WAIT)
mono      2788 2840   nitoback   72u     IPv4              65082      0t0        TCP 127.0.0.1:9591->127.0.0.1:40373 (CLOSE_WAIT)

They seem to be stuck in CLOSE_WAIT which I guess means that there is unread data on the socket. When it gets into this situation its not recoverable, just hangs and keeps the connections in CLOSE_WAIT state.
-----Original message-----
From: hyber at hyber.dk <hyber at hyber.dk>
Sent: Sunday 4th August 2013 14:39
To: Alfred Hall <ahall at ahall.org>
Cc: mono-devel-list at lists.ximian.com
Subject: SV: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux

Hi Alfred

Are you sure its not a ulimit issue? I have had this issue before with the "nofile" limit as each tcp session is a "file" under /proc and when you have 1024 open (or close wait) sessions the OS does not allow you to open more files and the code seems to hang for a bit until a session is completely gone. Eg. Timed out and you can create files again...

--
Esben
(on mobile)

----- Reply message -----
Fra: "Alfred Hall" <ahall at ahall.org>
Til: "Andrés G. Aragoneses" <knocte at gmail.com>, <mono-devel-list at lists.ximian.com>, <mono-devel-list at lists.ximian.com>
Emne: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
Dato: søn., aug. 4, 2013 13:17

I meant to say I tried master too:

root at mulder:~# /opt/ahall-mono/bin/mono -V
Mono JIT compiler version 3.3.0 (master/2354865 Sun Aug  4 00:42:51 BST 2013)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS:           __thread
SIGSEGV:       altstack
Notifications: epoll
Architecture:  amd64
Disabled:      none
Misc:          softdebug
LLVM:          supported, not enabled.
GC:            sgen

That should have the fix in from https://github.com/ysw, but setting MONO_DISABLE_AIO should have worked around that anyway, as its meant to bypass the epoll backend.

My Nancy service is literally just returning a very simple JSON:

    public class HelloWorldModule : NancyModule
    {
        public HelloWorldModule()
        {
            Get["/"] = parameters => {
                return Response.AsJson(new HomeResponse { Message = "Test" });
            };
        }
    }

In JMeter I'm using 100 threads and loop count of 100 and it locks up after like 15 seconds even over the network. Very odd.

-----Original message-----
> From:"Andrés G. Aragoneses" <knocte at gmail.com <mailto:knocte at gmail.com> >
> Sent: Sunday 4th August 2013 10:03
> To: mono-devel-list at lists.ximian.com <mailto:mono-devel-list at lists.ximian.com> 
> Subject: Re: [Mono-dev] NancyFX self hosting (HttpListener) locking up on linux
> 
> On 04/08/13 03:07, Alfred Hall wrote:
> > Hi there.
> >
> > I'm running the NancyFX web framework in self hosting mode which is
> > using HttpListener which basically means I deploy an executable and run
> > it and it will start a webserver on localhost on a TCP port of choice. I
> > then use nginx to proxy to it.
> >
> > I first ran my executable on my macbook pro and bombarded it with jmeter
> > and it coped fine and no issues.
> > I then deployed on my debian wheezy 64 bit linux box running on top of
> > KVM using mono 3.2.0 and performed the same jmeter experiment. It locks
> > up fairly quickly and wont take any new requests. I've tried using both
> > sgen and boehm but they behave similarly although it seems to lock up
> > faster when using sgen. I also tried enabling MONO_DISABLE_AIO but it
> > doesn't make any difference.
> >
> > Anyone had similar issues?
> >
> > I tried using self hosted ServiceStack which also uses HttpListener and
> > had similar issues so I'm finding it unlikely that the bug is in NancyFX
> > itself.
> >
> > I tried installing mono 2.10.8 to check if this is a regression, but
> > getting exactly the same results.
> >
> > Any idea how I can debug this further or what could be going on.
> 
> Hey Alfred.
> 
> Could you try mono master (3.3) instead of 3.2? I mention this because 
> this pull request [1] has been merged recently, which could help in this 
> situation (and wouldn't make much difference in Mac since the problem in 
> this platform is worked-around by avoiding kqueue [2]).
> 
> [1] https://github.com/mono/mono/pull/703
> 
> [2] https://github.com/mono/mono/blob/master/configure.in#L1823
> 
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com <mailto:Mono-devel-list at lists.ximian.com> 
> http://lists.ximian.com/mailman/listinfo/mono-devel-list <http://lists.ximian.com/mailman/listinfo/mono-devel-list> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ximian.com/pipermail/mono-devel-list/attachments/20130804/d356877d/attachment-0001.html>