[Mono-dev] Memory usage on Mono remoting

[pablosantosluac at terra.es]

Hi Gonzalo,

>> Well, I'll try to use the ansync sockets then, but I guess to get the
>> best out of them I'll need not only to use them during accept, but also
>> read data asynchronously, right?
>>     
>
> Correct. If possible, Write should also be asynchronous, but as long as
> the OS buffers everything, there should be no problem.
> [...]
>   

Ok, well, in fact I think I'm experiencing this right now: I've 112
clients against the same server, each of them will download about 300Mb.

I see how the thread pool gets full (quad core) and then new requests
are accepted but not "scheduled".

CPU is almost doing nothing, so I guess all threads are waiting for the
synchronous socket write to complete, so performance could be much higher.

>>> Coupled with asynchronous I/O, it will make better use of the resources
>>> available. There's no need to create 100 threads for 100 client or
>>> having 1 threadpool thread blocking on a socket asynchronous
>>> operation,... Also, if you're thinking of reusing buffers, this helps
>>> too, as the number of buffers will be bound to the maximum number of
>>> threads in the threadpool, ...
>>>   
>>>       
>> Good, so, whenever I wait for a read or a write using async, the thread
>> should be able to work on another request?
>>     
>
> Correct. In fact, you don't 'wait' for an asynchronous read or write.
> For instance, when you call BeginRead, the socket is added to an epoll
> fd and you BeginRead call returns immediately. The callback you
> provided, if any, will be called from a different thread as soon as new
> data is available. Just don't spoil it by doing something like
> socket.EndRead (socket.BeginRead (...)))   ;-)
>   
ok ! :-P

Well, I see it will mean a good number of changes in the Channel.


Besides, as I also told Dick and Dave on a separate thread, it seems the
latest MySql provider has a HUGE memory leak (ok, or it never frees
mem), which is causing a good number of problems to my test.

And, it seems I also have to do something with my code since all data
requests are reading byte[] in blocks of 4Mb, but not reusing the
buffers at all, so when a lot of threads are doing its job, huge amounts
of mem are being allocated and giving extra work to the GC which should
be completely avoided.

I'm a little bit concerned about the following, though:

If you've a method like byte[] GetData() which is going to be invoked
through remoting, even if you do custom serialization (maybe creating a
DataArray class) two copies of the data will be created: one the
original you created, and a second one in the serialization buffer prior
to be sent to the network... :-(

pablo