[Mono-list] High concurrency http requests with WebClient

[Robert Jordan]

ledjon wrote:
> I've noticed something with mono (well, .net as well): If you write an app
> that create, say, 100 threads, and each thread fires off WebClient requests,
> they end up getting serialized and done in order instead of in parallel.
> 
> This seriously hurts the scalability of any application that might want to
> do a lot of web requests such as a web indexing service.  This service needs
> to be able to do a lot of work as one process (i.e., threads) but if
> WebClient synchronizes across all the threads, then it is all for not.
> 
> Any thoughts or comments on how to get around this problem?

Try raising the number of threads in the thread pool.

See mono(1), environment var MONO_THREADS_PER_CPU.

Robert