[Mono-dev] HttpListener stops accepting requests

[Greg Young]

We have found a situation where HttpListener on mono (reproduced in
linux/OSX) will permanently stop accepting requests.

The issue involves ulimit. Basically if you get a connection flood
greater than ulimit the server will stop accepting all requests until
it is restarted.

After the connection flood you can see the connections were all
properly killed (I won't put all the ones that were there during the
flood!):

greg at clown:~$ lsof -p 16651
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
COMMAND     PID USER   FD   TYPE DEVICE  SIZE/OFF     NODE NAME
eventstor 16651 greg  cwd    DIR    8,2      4096  2105316
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0
eventstor 16651 greg  rtd    DIR    8,2      4096        2 /
eventstor 16651 greg  txt    REG    8,2  37443029  2105361
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/eventstored
eventstor 16651 greg  mem    REG    8,2     92504 12849525
/lib/x86_64-linux-gnu/libgcc_s.so.1
eventstor 16651 greg  mem    REG    8,2     51736 12845148
/lib/x86_64-linux-gnu/libnss_files-2.21.so
eventstor 16651 greg  mem    REG    8,2     47616 12845159
/lib/x86_64-linux-gnu/libnss_nis-2.21.so
eventstor 16651 greg  mem    REG    8,2     97224 12845153
/lib/x86_64-linux-gnu/libnsl-2.21.so
eventstor 16651 greg  mem    REG    8,2     35688 12845149
/lib/x86_64-linux-gnu/libnss_compat-2.21.so
eventstor 16651 greg  mem    REG    8,2   8464208  9444549
/usr/lib/locale/locale-archive
eventstor 16651 greg  mem    REG    8,2   1869392 12845162
/lib/x86_64-linux-gnu/libc-2.21.so
eventstor 16651 greg  mem    REG    8,2    142080 12845083
/lib/x86_64-linux-gnu/libpthread-2.21.so
eventstor 16651 greg  mem    REG    8,2     14592 12845158
/lib/x86_64-linux-gnu/libdl-2.21.so
eventstor 16651 greg  mem    REG    8,2     31680 12845075
/lib/x86_64-linux-gnu/librt-2.21.so
eventstor 16651 greg  mem    REG    8,2   1084840 12845161
/lib/x86_64-linux-gnu/libm-2.21.so
eventstor 16651 greg  mem    REG    8,2    154376 12845156
/lib/x86_64-linux-gnu/ld-2.21.so
eventstor 16651 greg  mem    REG   0,21      4096       88 /dev/shm/mono.16651
eventstor 16651 greg    0u   CHR  136,1       0t0        4 /dev/pts/1
eventstor 16651 greg    1u   CHR  136,1       0t0        4 /dev/pts/1
eventstor 16651 greg    2u   CHR  136,1       0t0        4 /dev/pts/1
eventstor 16651 greg    3r  0000   0,11         0     7966 anon_inode
eventstor 16651 greg    4u   REG    8,2         8  2624038
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/writer.chk
eventstor 16651 greg    5u   REG    8,2         8  2624039
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chaser.chk
eventstor 16651 greg    6u   REG    8,2         8  2624040
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/epoch.chk
eventstor 16651 greg    7u   REG    8,2         8  2624041
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/truncate.chk
eventstor 16651 greg    8r   CHR    1,9       0t0       11 /dev/urandom
eventstor 16651 greg    9u   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   10r   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   11r   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   12r   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   13r   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   14r   REG    8,2 268435712  2624042
/home/greg/Downloads/EventStore-OSS-Ubuntu-14.04-v3.5.0/data/chunk-000000.000000
eventstor 16651 greg   15u  IPv4 279150       0t0      TCP
localhost:1113 (LISTEN)
eventstor 16651 greg   16u  0000   0,11         0     7966 anon_inode
eventstor 16651 greg   17u  IPv4 278173       0t0      TCP
localhost:2113 (LISTEN)


My guess is that the too many files kills the asynchronous loop around
the accept in the httplistener code and will take a look there.
Perhaps someone else has some ideas.

Cheers,

Greg

-- 
Studying for the Turing test