[Mono-dev] Remoting and thread pool limits

Tue Dec 18 07:15:59 EST 2007

Hi Robert,

I'm trying this together with a SSL remoting channel.

I'm observing huge variations: I don't see anymore the "80" connections 
problems: I can run the test launching 200 threads and doing 200 connections 
in less than 2 seconds. (mono/Linux)

When I use the SSL channel on Mono/Linux I can get about 7 seconds to 
complete the test.

But the "bad" thing is that from time to time the same test takes about 40 
secs for tcp and about 180 for ssl!!!

I still didn't find out why there's such a difference!!

Do you think it could be related to the tcp stack? Maybe it is needed to 
create sockets with different params or something like that?

Thanks,

pablo

----- Original Message ----- 
From: "pablosantosluac" <pablosantosluac at terra.es>
To: <mono-devel-list at lists.ximian.com>; "Robert Jordan" <robertj at gmx.net>
Sent: Monday, December 17, 2007 11:59 PM
Subject: Re: [Mono-dev] Remoting and thread pool limits

> Ok, so this are the ones to inspect then: ProcessMessages,
> ReceiveMessageStatus and StreamRead. Well, looks ok that they are the ones
> eating CPU (the system is processing requests), but... why they go worse
> when dealing with more than 80 threads?
>
> Looking at the code I can't see anything at these methods... :-( I'll try 
> to
> look into it again in more detail...
>
> Thanks,
>
> pablo
>
> ----- Original Message ----- 
> From: "Robert Jordan" <robertj at gmx.net>
> To: <mono-devel-list at lists.ximian.com>
> Sent: Monday, December 17, 2007 11:45 PM
> Subject: Re: [Mono-dev] Remoting and thread pool limits
>
>
>> Hi Pablo,
>>
>> pablosantosluac wrote:
>>> Hi Robert,
>>>
>>> Do you think it is related to the thread creation?
>>
>> I don't think so. Most of the time is spent in TcpChannel:
>>
>> System.Runtime.Remoting.Channels.Tcp.ClientConnection::ProcessMessages()
>> System.Runtime.Remoting.Channels.Tcp.TcpMessageIO::ReceiveMessageStatus(Stream,byte[])
>> System.Runtime.Remoting.Channels.Tcp.TcpMessageIO::StreamRead(Stream,byte[],int)
>>
>> When I stop the test at 40 threads, these methods are not at the top
>> anymore.
>>
>> Robert
>>
>>>
>>> I'll run some tests with the mono profiler on Linux and try to figure 
>>> out
>>> where the time is being lost...
>>>
>>>
>>> pablo
>>>
>>>
>>>
>>> ----- Original Message ----- 
>>> From: "Robert Jordan" <robertj at gmx.net>
>>> To: <mono-devel-list at lists.ximian.com>
>>> Sent: Saturday, December 15, 2007 9:25 PM
>>> Subject: Re: [Mono-dev] Remoting and thread pool limits
>>>
>>>
>>>> pablosantosluac wrote:
>>>>> Well, it is not a *bug* but a feature. I wonder if it should be
>>>>> changed.
>>>>> If
>>>>> you look into RemotingThreadPool.cs there is a line like:
>>>>>
>>>>>
>>>>> threadDone.WaitOne(PoolGrowDelay, false);
>>>>>
>>>>>
>>>>> This one is actually the one making the process too slow. I'm afraid 
>>>>> it
>>>>> must
>>>>> be something similar on the .NET code too!
>>>>>
>>>>> Of course removing this line the problem gets solved, but I guess 
>>>>> there
>>>>> is a
>>>>> reason in the channel to do that.
>>>> I noticed that too, but this doesn't solve the real problem: the 200
>>>> thread test is still too slow on my machine. The degradation seems to
>>>> start after 60-80 threads on my pretty weak SMP machine.
>>>>
>>>> Just to be sure,  I've replaced RemotingThreadPool with an own, simple
>>>> version based on the standard BCL ThreadPool => same issue, although
>>>> I've raised the env var MONO_THREADS_PER_CPU to an insane value.
>>>>
>>>> Robert
>>>>
>>>>>
>>>>> pablo
>>>>>
>>>>>
>>>>> ----- Original Message ----- 
>>>>> From: "Robert Jordan" <robertj at gmx.net>
>>>>> To: <mono-devel-list at lists.ximian.com>
>>>>> Sent: Saturday, December 15, 2007 4:12 PM
>>>>> Subject: Re: [Mono-dev] Remoting and thread pool limits
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> pablosantosluac wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've found the following difference working with .NET and Mono
>>>>>>> remoting.
>>>>>>> In
>>>>>>> .NET when the remoting ThreadPool reaches the pool's maximum (25
>>>>>>> threads
>>>>>>> per
>>>>>>> process), it is able to continue creating new threads. In fact, you
>>>>>>> can
>>>>>>> end
>>>>>>> up having a remoting server with hundreds of threads.
>>>>>>>
>>>>>>> In mono the behaviour is different. Once the limit is reached it
>>>>>>> starts
>>>>>>> refusing connections.
>>>>>> This is unrelated to mono's remoting thread pool. You have probably
>>>>>> ran your tests on Windows, where mono indeed fails with a GC failure
>>>>>> when too many threads are created because the GC has a hard coded
>>>>>> max thread count limit.
>>>>>>
>>>>>> On Linux (x86_64, Mono 1.2.6) I can finish the tests:
>>>>>>
>>>>>> Linux Mono client -> Linux Mono server (same machine)
>>>>>>
>>>>>> poseidon [~/foo] $ mono client/bin/Debug/client.exe
>>>>>> tcp://localhost:8084/remote
>>>>>> 1 - Time 1206 ms
>>>>>> 5 - Time 2045 ms
>>>>>> 10 - Time 3575 ms
>>>>>> 20 - Time 8174 ms
>>>>>> 40 - Time 12055 ms
>>>>>> 50 - Time 8185 ms
>>>>>> 200 - Time 46150 ms
>>>>>>
>>>>>>
>>>>>> MS.NET client -> Linux Mono server (different machines)
>>>>>>
>>>>>> troll:/cygdrive/u/foo/client/bin/Debug $ ./client.exe
>>>>>> tcp://poseidon:8084/remote
>>>>>> 1 - Time 1297 ms
>>>>>> 5 - Time 2000 ms
>>>>>> 10 - Time 3515 ms
>>>>>> 20 - Time 6016 ms
>>>>>> 40 - Time 11031 ms
>>>>>> 50 - Time 6016 ms
>>>>>> 200 - Time 94375 ms
>>>>>>
>>>>>>
>>>>>> The numbers don't look very well, though. I promise to look at
>>>>>> this when you file a bug report so that it doesn't get
>>>>>> overlooked.
>>>>>>
>>>>>> Robert
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mono-devel-list mailing list
>>>>>> Mono-devel-list at lists.ximian.com
>>>>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>>> _______________________________________________
>>>> Mono-devel-list mailing list
>>>> Mono-devel-list at lists.ximian.com
>>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>
>> _______________________________________________
>> Mono-devel-list mailing list
>> Mono-devel-list at lists.ximian.com
>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list 