[Mono-list] High concurrency http requests with WebClient

[ledjon]

I'm not sure that's the problem. The threads themselves fire off just fine
(and work everything independently), but they all end up synchronizing on
WebClient requests even though the threads have nothing to do with one
another.

I can see by looking at the WebClient.cs source it specifically sets various
"is_buys" variables to prevent concurrent requests.  I'm just wondering why
it is so simplistic, and are there other "more advanced" classes that are
more "thread independent"?

- Jon


Robert Jordan wrote:
> 
> ledjon wrote:
>> I've noticed something with mono (well, .net as well): If you write an
>> app
>> that create, say, 100 threads, and each thread fires off WebClient
>> requests,
>> they end up getting serialized and done in order instead of in parallel.
>> 
>> This seriously hurts the scalability of any application that might want
>> to
>> do a lot of web requests such as a web indexing service.  This service
>> needs
>> to be able to do a lot of work as one process (i.e., threads) but if
>> WebClient synchronizes across all the threads, then it is all for not.
>> 
>> Any thoughts or comments on how to get around this problem?
> 
> Try raising the number of threads in the thread pool.
> 
> See mono(1), environment var MONO_THREADS_PER_CPU.
> 
> Robert
> 
> _______________________________________________
> Mono-list maillist  -  Mono-list at lists.ximian.com
> http://lists.ximian.com/mailman/listinfo/mono-list
> 
> 

-- 
View this message in context: http://www.nabble.com/High-concurrency-http-requests-with-WebClient-tp23862474p23891979.html
Sent from the Mono - General mailing list archive at Nabble.com.