[Mono-list] High concurrency http requests with WebClient

[Robert Jordan]

ledjon wrote:
> I'm not sure that's the problem. The threads themselves fire off just fine
> (and work everything independently), but they all end up synchronizing on
> WebClient requests even though the threads have nothing to do with one
> another.

It's not about your app's threads. WebClient is bound to the thread
pool regardless of how multithreaded your app is.

> I can see by looking at the WebClient.cs source it specifically sets various
> "is_buys" variables to prevent concurrent requests.

Irrelevant as it's an instance member. You're not supposed to
share WebClients between threads anyways, because WebClient
is not thread safe.

Try to raise the thread pool and file a bug if you believe
it is one, especially if the code is working on MS.NET but
not on mono.

Robert

> 
> - Jon
> 
> 
> Robert Jordan wrote:
>> ledjon wrote:
>>> I've noticed something with mono (well, .net as well): If you write an
>>> app
>>> that create, say, 100 threads, and each thread fires off WebClient
>>> requests,
>>> they end up getting serialized and done in order instead of in parallel.
>>>
>>> This seriously hurts the scalability of any application that might want
>>> to
>>> do a lot of web requests such as a web indexing service.  This service
>>> needs
>>> to be able to do a lot of work as one process (i.e., threads) but if
>>> WebClient synchronizes across all the threads, then it is all for not.
>>>
>>> Any thoughts or comments on how to get around this problem?
>> Try raising the number of threads in the thread pool.
>>
>> See mono(1), environment var MONO_THREADS_PER_CPU.
>>
>> Robert
>>
>> _______________________________________________
>> Mono-list maillist  -  Mono-list at lists.ximian.com
>> http://lists.ximian.com/mailman/listinfo/mono-list
>>
>>
>