[Mono-list] Performance tests/benchmarks for mono

[Edward Ned Harvey (mono)]

> From: mono-devel-list-bounces at lists.ximian.com [mailto:mono-devel-list-
> bounces at lists.ximian.com] On Behalf Of Liwei Peng
> 
> I am evaluating mono for Linux (Ubuntu). One of my plans is to evaluate
> mono CLR perf compared with Windows .NET CLR perf.

Before you get into it, you should know a few things - 

Everything, literally everything, is variable.  Meaning, you'll write some code that uses some class method, and you run it on Windows, with optimizations, and then run again without optimizations, and then run on some version of mono with and without optimizations, and then run on a different version of mono with and without optimizations, and then run on a different platform (windows vs mac vs linux vs etc) with and without optimizations, and then run on a different class of hardware including or not including some hardware optimizations, and in every one of those situations, the performance varies.

Sometimes windows is faster for some things.  Sometimes mono is faster for other things.  Sometimes the .NET api provides different implementations to the same interface or abstract class - and sometimes mono ignores that and implements them the same behind the scenes just because that's what's available.  For example, some RSACryptoServiceProvider, and RSAManaged, and RSAWhatever ...  All functionally equivalent with different performance characteristics on windows relative to each other, but then in mono, they all map to the same RSAManaged.  (I'm speaking vaguely for example, maybe it's RSA or AES or other classes, I forget, I assume there are a bunch of things like this.)

The behavior is not always exactly the same - for example, in .NET you create an instance of RSA with RSA.Create(), and it not only gives you an object, it actually generates keys for you.  But in mono, they don't bother creating keys for you until you do some operation that requires the existence of keys within the RSA object.  So if you just measured the time to create an instance of RSA, you would be inaccurate in measuring the actual work time.

And guess what, .NET recently changed its behavior to be like mono.  Lazy RSA key generation.  So those performance measurements, once again, have changed.  Moving target.  And I have reason to believe that windows pre-generates a cache of keys to be available upon user request, so it might perform really well for a number of keys and then suddenly drop to normal mono-like performance for sustained repeated key generation.  And the windows keys apparently aren't made to the same quality specification.

Etc etc etc etc.

You write some really simple code just to test the performance of calling methods recursively or something like that.  Compile it on windows, whose compiler will have some set of compiler optimizations, then run it on windows and on mono, where the JIT compiler will apply a second set of optimizations.  Now compile it on mono, whose compiler will have a different set of compiler optimizations.  Again, run it on mono and windows for another set of JIT optimizations.  Yet another variable in the performance characteristics.

Perform an AOT compilation.  Yet another variable.

There is only one possible conclusion you can reach (I've done a lot of this myself):  Some things are faster in windows, others are faster in mono.  If you pick one specific thing and dig into it you can then explain the difference for that specific thing.  But you cannot draw any generalization.  You can only create a meaningful summary of performance comparison for a specific application.  And it will change over time, because of changes in both Windows and in Mono.