[Mono-dev] Fundamental performance problems with Mono

[zvikag]

Hi,

Writing my server in a managed language, be it Mono or Java, is a basic
choice of effectiveness over cost. I prefer good performance in no time than
super-great-performance that takes ages to develop and will almost surely
rain upon me bugs from hell. And if you write your managed code correctly
(mainly aim towards zero GC, and this includes the framework...) then your
performance should be equivalent to a native implementation. If you can
point out a managed async socket framework that works I'd be happy to try
it. I don't have time to get down and dirty. I got business logic on my
hands... 

Now, from what I understand, the implementation in mono follows the
well-known recipe for high-performance socket servers in Linux: event
polling (epoll) with I/O Thread pool - you call send()/receive() when the
socket is ready. so, yes, eventually you call the simple socket API, but
this is how you write high-performance servers in Linux. but more
importantly, any socket framework that wants to wrap this model will need
primitives like BeginReceive\ReceiveAsync + OnDataReceived callback and
BeginSend\SendAsync + OnDataSent calback. can't escape it. so it doesn't
matter if the operating system doesn't have native support for async sockets
- you end up with more or less the same managed async socket API on Windows
and Linux.

Therefore, I do blame Mono for the poor performance. If the epoll+I/O Thread
pool model is implemented correctly and if the managed async socket API is
implemented properly (and there is a lot to benefit from the XXXAsync
methods) then there is no reason why a Mono server should not be equal in
performance to a .NET on Windows server. And again, this is good enough
performance for me.


damageboy wrote:
> 
> Hi Zvika,
> I'll start by saying that I've been there, I've also seen abysmal
> performance with the mono async socket implementation.
> If you'll dig down in the code (I did so last time around Mono 2.2) you
> should also see that there is no such thing in Mono/Linux at any rate...
> By this, I mean that a fundamental difference you'll find in the Linux
> world from the Windows world is that there is no async socket API for
> Linux. This is a "limitation" (if you want to call it like that) of the
> Linux kernel, and in no way related to mono.
> 
> While calling BeginSend/Receive in Windows + MS.NET is implemented by
> means of true async sockets on Windows, which ultimately are a Winsock /
> Windows NT Kernel feature, calling BeginSend on Mono simply queues a work
> item into the thread pool that will call the normal socket apis.
> This is a fundamental difference in how Mono/MS.NET work.
> Feel free to gaze at the code on
> "mcs/class/System/System.Net.Sockets/Socket.cs" and see this for
> yourself...
> While the Mono people could write two implementations for BeginXXX (one
> for Windows + async sockets, one for Linux) I don't really blame them for
> implementing the BeingXXX APIs the way they did.
> 
> In a way, using a BeginXXX APIs for sockets on Mono generally degrades
> performance (in terms of overhead and latency for packet send/receive)
> under heavy load than using the regular non-async apis.
> This should pretty much leave you asking yourself why would you ever want
> to use the so-called more advanced "XXXAsync Socket API" (which was your
> original intent, as far as I can tell).
> I personally see very little benefit even if were implemented in Mono.
> 
> This definitely does not mean that all is lost. On the contrary, you can
> achieve much higher throughput / lower latency with Mono + Linux, but
> achieving this with the Microsoft centric APIs / paradigms (as
> System.Net.Sockets is) is highly unlikely IMO (again, I would like to
> stress that this is really not Mono's fault).
> 
> I suggest you read up on the C10K problem either on Wikipedia or Dan
> Kegel's site:
> http://www.kegel.com/c10k.html
> 
> There are many possible solutions, including some that are not mentioned
> in the C10K page, such as using
> P/Invoke to call vmsplice/splice for sending/receiving data with Zero Copy
> networking or, as I've done in the past,
> wrapping up Evgeniy Polyakov's netchannels and userspace network stack:
> http://www.ioremap.net/projects/unetstack
> http://www.ioremap.net/projects/netchannels
> 
> Although this means getting down and dirty, often using unsafe code and
> pointers and 
> what not, let me assure you, that you will be able to make a very modest
> server/desktop 
> machine blow away anything you've ever sen with Windows before.
> 
> In short, I think you're looking at the wrong problem.
> 
> Hope this helps.
> 
> 
> 
> zvikag wrote:
>> 
>> Hello all,
>> The bottom line of this message is that I don't see how can one write a
>> high-performance socket server in Mono...
>> Here is the story:
>> I am writing a proxy server using .NET Socket API. This proxy does almost
>> entirely I/O work - copying buffers from one socket to another. Now, Mono
>> doesn't implement the newer 
>> http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx
>> XXXAsync Socket API  that was introduced in .NET 2.0 SP1 (or more
>> accurately, implements it 
>> http://www.mail-archive.com/mono-list@lists.ximian.com/msg28621.html
>> perfunctorily ). So I was left to use the APM Socket API which produces
>> tons of garbage objects under heavy load.
>> When testing the server on Linux under load we saw very frequent CPU
>> bursts that crippled the throughput of the server. After profiling with
>> the mono built-in profiler I confirmed that the reason for the high CPU
>> usage was the GC collections that got more and more frequent and took
>> more and more time. I then read a little bit and realized that the Mono
>> GC is non-generational which might explain the long GC cycles (if it was
>> generational it could have collected the garbage objects that were
>> created during async socket operations in generation 0 and probably stop
>> there, but it has to traverse the entire managed heap).
>> So the combination of the non-generational GC and the unimplemented
>> XXXAsync Socket API result in very poor performance of the Mono server.
>> The maximum throughput of the server with Mono on Linux is about half of
>> that on Windows using .NET.
>> 
>> I attached the GC stats and profiling results of a 15 minute run.
>>  http://old.nabble.com/file/p27026906/profile_alloc.log profile_alloc.log 
>>  http://old.nabble.com/file/p27026906/gc_stats.log gc_stats.log 
>> Can you help me out here?
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Fundamental-performance-problems-with-Mono-tp27026906p27074993.html
Sent from the Mono - Dev mailing list archive at Nabble.com.