[Mono-dev] Remoting and thread pool limits

[pablosantosluac]

Ok, so this are the ones to inspect then: ProcessMessages, 
ReceiveMessageStatus and StreamRead. Well, looks ok that they are the ones 
eating CPU (the system is processing requests), but... why they go worse 
when dealing with more than 80 threads?

Looking at the code I can't see anything at these methods... :-( I'll try to 
look into it again in more detail...

Thanks,

pablo

----- Original Message ----- 
From: "Robert Jordan" <robertj at gmx.net>
To: <mono-devel-list at lists.ximian.com>
Sent: Monday, December 17, 2007 11:45 PM
Subject: Re: [Mono-dev] Remoting and thread pool limits


> Hi Pablo,
>
> pablosantosluac wrote:
>> Hi Robert,
>>
>> Do you think it is related to the thread creation?
>
> I don't think so. Most of the time is spent in TcpChannel:
>
> System.Runtime.Remoting.Channels.Tcp.ClientConnection::ProcessMessages()
> System.Runtime.Remoting.Channels.Tcp.TcpMessageIO::ReceiveMessageStatus(Stream,byte[])
> System.Runtime.Remoting.Channels.Tcp.TcpMessageIO::StreamRead(Stream,byte[],int)
>
> When I stop the test at 40 threads, these methods are not at the top
> anymore.
>
> Robert
>
>>
>> I'll run some tests with the mono profiler on Linux and try to figure out
>> where the time is being lost...
>>
>>
>> pablo
>>
>>
>>
>> ----- Original Message ----- 
>> From: "Robert Jordan" <robertj at gmx.net>
>> To: <mono-devel-list at lists.ximian.com>
>> Sent: Saturday, December 15, 2007 9:25 PM
>> Subject: Re: [Mono-dev] Remoting and thread pool limits
>>
>>
>>> pablosantosluac wrote:
>>>> Well, it is not a *bug* but a feature. I wonder if it should be 
>>>> changed.
>>>> If
>>>> you look into RemotingThreadPool.cs there is a line like:
>>>>
>>>>
>>>> threadDone.WaitOne(PoolGrowDelay, false);
>>>>
>>>>
>>>> This one is actually the one making the process too slow. I'm afraid it
>>>> must
>>>> be something similar on the .NET code too!
>>>>
>>>> Of course removing this line the problem gets solved, but I guess there
>>>> is a
>>>> reason in the channel to do that.
>>> I noticed that too, but this doesn't solve the real problem: the 200
>>> thread test is still too slow on my machine. The degradation seems to
>>> start after 60-80 threads on my pretty weak SMP machine.
>>>
>>> Just to be sure,  I've replaced RemotingThreadPool with an own, simple
>>> version based on the standard BCL ThreadPool => same issue, although
>>> I've raised the env var MONO_THREADS_PER_CPU to an insane value.
>>>
>>> Robert
>>>
>>>>
>>>> pablo
>>>>
>>>>
>>>> ----- Original Message ----- 
>>>> From: "Robert Jordan" <robertj at gmx.net>
>>>> To: <mono-devel-list at lists.ximian.com>
>>>> Sent: Saturday, December 15, 2007 4:12 PM
>>>> Subject: Re: [Mono-dev] Remoting and thread pool limits
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> pablosantosluac wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've found the following difference working with .NET and Mono
>>>>>> remoting.
>>>>>> In
>>>>>> .NET when the remoting ThreadPool reaches the pool's maximum (25
>>>>>> threads
>>>>>> per
>>>>>> process), it is able to continue creating new threads. In fact, you 
>>>>>> can
>>>>>> end
>>>>>> up having a remoting server with hundreds of threads.
>>>>>>
>>>>>> In mono the behaviour is different. Once the limit is reached it 
>>>>>> starts
>>>>>> refusing connections.
>>>>> This is unrelated to mono's remoting thread pool. You have probably
>>>>> ran your tests on Windows, where mono indeed fails with a GC failure
>>>>> when too many threads are created because the GC has a hard coded
>>>>> max thread count limit.
>>>>>
>>>>> On Linux (x86_64, Mono 1.2.6) I can finish the tests:
>>>>>
>>>>> Linux Mono client -> Linux Mono server (same machine)
>>>>>
>>>>> poseidon [~/foo] $ mono client/bin/Debug/client.exe
>>>>> tcp://localhost:8084/remote
>>>>> 1 - Time 1206 ms
>>>>> 5 - Time 2045 ms
>>>>> 10 - Time 3575 ms
>>>>> 20 - Time 8174 ms
>>>>> 40 - Time 12055 ms
>>>>> 50 - Time 8185 ms
>>>>> 200 - Time 46150 ms
>>>>>
>>>>>
>>>>> MS.NET client -> Linux Mono server (different machines)
>>>>>
>>>>> troll:/cygdrive/u/foo/client/bin/Debug $ ./client.exe
>>>>> tcp://poseidon:8084/remote
>>>>> 1 - Time 1297 ms
>>>>> 5 - Time 2000 ms
>>>>> 10 - Time 3515 ms
>>>>> 20 - Time 6016 ms
>>>>> 40 - Time 11031 ms
>>>>> 50 - Time 6016 ms
>>>>> 200 - Time 94375 ms
>>>>>
>>>>>
>>>>> The numbers don't look very well, though. I promise to look at
>>>>> this when you file a bug report so that it doesn't get
>>>>> overlooked.
>>>>>
>>>>> Robert
>>>>>
>>>>> _______________________________________________
>>>>> Mono-devel-list mailing list
>>>>> Mono-devel-list at lists.ximian.com
>>>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>>> _______________________________________________
>>> Mono-devel-list mailing list
>>> Mono-devel-list at lists.ximian.com
>>> http://lists.ximian.com/mailman/listinfo/mono-devel-list
>
> _______________________________________________
> Mono-devel-list mailing list
> Mono-devel-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-devel-list