[Mono-dev] TcpServerChannel loosing connections under heavy load

[pablosantosluac at terra.es]

Hi,

I'm having issues with Mono remoting under high performance networking
scenarios. Some clients are rejected since the server is not able to
handle connections.

Look at the following code inside TcpServerChannel
(http://anonsvn.mono-project.com/viewvc/trunk/mcs/class/System.Runtime.Remoting/System.Runtime.Remoting.Channels.Tcp/TcpServerChannel.cs?revision=110042&view=markup)

I've removed the #ifdef code for clarity.

    void WaitForConnections ()
    {
      try
      {
        while(!stopped)
        {
          Socket socket = listener.AcceptSocket ();
          ClientConnection reader = new ClientConnection (this, socket,
sink);
          try {
            if (!threadPool.RunThread (new ThreadStart
(reader.ProcessMessages)))
              socket.Close ();
          } catch (Exception e)
          {
          }
        }
      }
      catch (Exception e)
      {
      }
    }

Well, each time an "accept" is done, a new thread is launched (from the
pool).

When a high number of threads is managed (about 130 concurrent clients),
it makes the channel reject remote requests.

Inside the "threadPool" the following code:

 threadDone.WaitOne(PoolGrowDelay, false);

Makes thread creation wait 500ms (I guess just forcing the CPU to switch
to another one) each time a new thread is created (when the pool is
growing).

If you remove it, connection problems go away, but I guess it's not the
right way to implement it.

So, what I'll do is to "enqueue" the newly created ClientConnection in
the "accept" code instead of spawning the thread just that, so a
different one (or even the pool itself, I'll check) will consume
elements and spawn threads.

This way we avoid the "accept" thread from getting "distracted" from its
main job: accepting client connections.

It actually makes me wonder if it would be better to use asynchronous
socket calls instead of synch ones + threading, as they strongly
recommend here:

http://msdn.microsoft.com/es-es/magazine/cc300760(en-us).aspx

.NET TCP Channel is based on asynch socket calls.

My question is: are asynchronous socket calls better on Linux as they're
on Windows?

Also, it would be great (and very easy to implement) if new
configuration properties would be added to the TCP channel so things
like socket options, socket buffer sizes and so on could be set to
improve performance without touching the code (I'll work on that too).

Some very interesting guidelines that could be easily applied then are
found here: Boost socket performance on Linux:

http://www.ibm.com/developerworks/linux/library/l-hisock.html

And I see some have been already applied:

http://anonsvn.mono-project.com/viewvc/trunk/mcs/class/System.Runtime.Remoting/System.Runtime.Remoting.Channels.Tcp/TcpConnectionPool.cs?r1=113395&r2=113394&pathrev=113395

Since NoDelay was set by Robert Jordan to the TcpConnectionPool.
Question: I see it applies to the client but, wouldn't it be useful for
the server too?


Thanks,

pablo


www.plasticscm.com