[Mono-dev] Memory usage on Mono remoting

[Gonzalo Paniagua Javier]

On Tue, 2009-07-14 at 11:12 +0200, pablosantosluac at terra.es wrote:
> Hi Gonzalo,
> 
> >> Testing PlasticSCM under really heavy load (hundreds of clients against
> >> a single server delivering hundreds of Gb over the network).
> >>     
> >
> > So no profiling...
> >
> >   
> Comparing the same code, the same hardware and the same test under Linux
> and Windows. On Windows we run under .NET, and the test passes successfully.

Oh, when I said profiling I meant the mono --profile=stat or similar
that can tell you what and where is being allocated.
[...]
> > It's HttpResponseStream.cs all the *Bucket classes that then use an
> > IntPtrStream.
> >
> > I still don't think that allocating MemoryStream (256 bytes by default)
> > is hurting that bad. 
> Considering Boehm GC seems to have really hard times releasing memory
> and we're delivering GBs of data... it could be.

Delivering GBs of data and having hundreds of connections should not be
a problem. Years ago, when testing iFolder under those conditions
everything worked just fine. But it was mod-mono-server/apache.

> I'm not 100% sure, but it seems reusing buffers could be a very good idea.

Xsp does it too and it's much better than allocating 32kB for every
request every time.

> > A wild guess is that the BufferedStream wrapping
> > the NetworkStream is allocating much more memory (4kB by default). But
> > if the code is rewritten following what xsp does, this should not be a
> > problem any more.
> >   
> Ok, I'm not familiar with xsp, I'll take a look.
> 
> I've just noticed that you use "send" from libc instead of the socket
> functions... I guess it is due to performance reasons, right?

That's because I wanted to use the TCP_CORK option to avoid sending
headers and the beginning of the content in separate packets.

> Also, you mentioned in a previous email that the TcpChannel should be
> changed so it uses Asynch sockets. I've seen you use AsyncCallBack on XSP.
> 
> My question is: I guess AsynchCallback will use a thread underneath,
> won't it? If so: what's the advantage over launching threads to accept
> calls?

Your guess is wrong. Those asynchronous calls from Socket are treated as
if they were a WorkItem for a ThreadPool, only that when they are made,
the socket is added to an epoll fd (if you're on linux with support for
epoll). And when there's an event in the socket, there's a dedicated IO
threadpool to take care of reading/writing data and invoking the
callbacks. The advantages: if you have 10k connections, you don't need
10k threads, threads are reused (no creation overhead), ...

> You mentioned it is better to use the default ThreadPool instead of the
> internal one in the TcpChannel, why is it going to be better?

Coupled with asynchronous I/O, it will make better use of the resources
available. There's no need to create 100 threads for 100 client or
having 1 threadpool thread blocking on a socket asynchronous
operation,... Also, if you're thinking of reusing buffers, this helps
too, as the number of buffers will be bound to the maximum number of
threads in the threadpool, ...

-Gonzalo