[Mono-list] lies and microbenmarks (Was: Mono 0.23 windows installer)

[David Jeske]

On Fri, Mar 14, 2003 at 12:55:18PM +1100, Fergus Henderson wrote:
> On 13-Mar-2003, David Jeske <jeske@chat.net> wrote:
> > I can see how the anti-benchmarking clause was important to the MS
> > lawyers, because if an article was published showing a brand new .NET
> > runtime slower than the 5 year old JVM technology, readers would not
> > respect the newness of .NET. In reality, the legal people should have
> > talked to the technical people and realized that nearly all MS.NET
> > vs. Java benchmarks ALREADY come out with .NET on top
> 
> All the benchmarks you've seen, you mean?

I mean the few benchmarks I've written and plenty of the benchmarks in
The Win32 version of Doug's Language Shootout.

My paragraph was tongue-and-cheek. I'm joking about how MS should have
required people to publish benchmark numbers because they actually did
a good job. Laugh Laugh. Ha Ha. I'm expecting a cease and desist in
the mail any day now.

> Don't you think it is possible that the benchmark results that
> you have seen are skewed, because of the .NET license condition?

Anything is possible, but no, I don't think that is the case. Check
out the tests, the source is available. My micro-benchmarks agreed
with the kind of skew seen here:

  http://dada.perl.it/shootout/

Java was faster at 3 of the tests, csharp was faster at the other 18.

> Also, Java is not the only competition.  In performance comparisons with
> native code, .NET is going to come off worse.  The question is how much
> worse.  Furthermore, the answer may be different for different
> programming styles or different programming languages.

This is a good point, and in the shootout above, you can certainly see
C# getting waxed by C/C++ and ML to name a few. I'm anxious to see how
much C# can close that gap with CIL v2.0 and Generics. They/we should
be able to get close to C++, and when that happens, I'll be thrilled.

My original post served two purposes: (a) to show that 1-2 year old C#
implementations are doing well relative to much older Java
implementations already. (b) Highlight the areas where Mono's
performance was orders of magnitude worse than MS so someone would
have the chance to recognize a design flaw or implemenation limitation
early enough should there be one.

My goal wasn't to poke fights about performance. I really could care
less. Today I spend most of my time programming in Python which is
certainly no speed daemon.

> One of .NET's big selling points is that it supports multiple
> programming languages.  However, if you get a 1000-times slow down
> for non-C#-like languages, that claim is a bit of a sham, isn't it?
> It's all very well to say that you can use other languages, but if
> the performance of .NET for those other languages is prohibitively
> bad, people will prefer to compile those other languages to native
> code.

I'm not sure how this multi-language discussion got in here, but I'm
happy to respond:

IMO, the multi-language thing itself is mostly hype. JVM still
supports more languages than .NET. Neither of them support "different"
languages very well.

However, from what I've seen, MSIL simply has better performance
characteristics. Today it is because of things like structs and
references (i.e. safe pointers). Tommorow it will be because of CIL
support for Generics. Which means those "workarounds" for non-C#/Java
languages will probably run faster. Here are some references:

  http://www.dcs.ed.ac.uk/home/stg/MRG/comparison/slides.pdf
  http://www.citi.qut.edu.au/research/plas/projects/cp_files/virtual_machines.pdf

That first paper talks about an approach where they extended their VM
to read JVM bytecodes instead of trying to force ML/SML into
JVM/CIL. In some ways this seems more sensible and just as useful.

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + jeske@chat.net