[Mono-list] lies and microbenmarks (Was: Mono 0.23 windows installer)

[Stefan Matthias Aust]

David Jeske wrote:

> I mean the few benchmarks I've written and plenty of the benchmarks in
> The Win32 version of Doug's Language Shootout.

It greately depends on the benchmarks I guess.  I benchmarks two more 
complex applications - a Smalltalk-Interpreter I once wrote for fun and 
a Scheme interpreter (actually an older version of Peter Norvigs SILK 
aka Jscheme) which I tried to port 1:1 from Java to C#.  Both 
applications run much faster in Java than in C#.  .NET was nearly as 
fast as the Java client VM, but the Server VM was much faster.

Both benchmarks make heavy (heaviest) use of garbage collection, object 
allocation and method calling and do nothing of the "usual" benchmarks - 
looping over arrays, doing small arithmetics etc.  I consider such 
application benchmarks more useful as they better mirror the typical 
work of a application server or complex OO application.

Suns Hotspot VM is definitely not the best (there's an Intel resarch VM 
which is - according to Intel - 20% better than the client VM for 
example) and Microsofts VM is pretty fast.  However, it's more tailored 
towards faster one-shot execution (which is what most benchmarks 
measure) and doesn't do (AFAIK) agressive long term, profile based 
optimations like the hotspot architecture.

One example are precompiled libraries.  Linking and therefore starting 
time is improved, but as you (probably, I don't know for sure, perhaps 
the precompiled libraries still contain the IL code) try to gain speed 
by inlining and devirtualising method calls, precompiled code is an 
obstacle.

> My original post served two purposes: (a) to show that 1-2 year old C#
> implementations are doing well relative to much older Java
> implementations already.

Actually, C# doesn't matter here.  You compare the performance of the 
Microsoft VM (aka VES) with the JVM and Microsoft is working on that VM 
since the early days of Java.  I'm pretty sure that the .NET VM shares 
the same technology (and perhaps even the code) of Microsofts Java VM 
which also wasn't new technology back than but based on existing rearch 
work.

> IMO, the multi-language thing itself is mostly hype. JVM still
> supports more languages than .NET. Neither of them support "different"
> languages very well.

Actually, the JVM doesn't support multiple languages.  Bytecode is 
nearly a 1:1 representation of the Java semantics. The one exception are 
gotos which would be possible in JVM bytecode and which are not part of 
Java.  Therefore, all other languages which different semantics (which 
aren't just a syntactic skin as C# and VB.NET are) must emulate their 
semantics in Java.

This meants, stuff that isn't fast in Java can't be cast in these other 
languages and those languages are always second level citizen.  So it's 
no real wonder for me that other languages never cought up.

For .NET things a slightly better.  The CIL is still architectured 
toward one (boring) kind of language but at least, there're some ways 
out.  The tail-end-recursion-call in the CIL might be the most prominent 
example.  Efficient tail-end-recursion is a requirement for most 
functional languages.  Values types, automatic boxing and unmanaged code 
can probably also help to implement other languages for .NET.

Still you won't get efficient dynamic languages, no Lisp, no Smalltalk, 
no Ruby or Python.  Fortunatley, computers are probably fast enough to 
sacrify one order of magnitute of performance for better languages with 
better development performance (using python for rapid prototyping 
instead of C for example).

> However, from what I've seen, MSIL simply has better performance
> characteristics. Today it is because of things like structs and
> references (i.e. safe pointers). Tommorow it will be because of CIL
> support for Generics. Which means those "workarounds" for non-C#/Java
> languages will probably run faster. Here are some references:
> 
>   http://www.dcs.ed.ac.uk/home/stg/MRG/comparison/slides.pdf

This paper emphazises the fact mentioned above, the CIL is slightly 
better for supporting other languages - specially stically typed 
functional languages - probably because that's what most Microsoft 
research guys love and use.

Unfortunately, the (only IMHO of couse) practically more important group 
of dynamic languages isn't supported well on neither platform.


>   http://www.citi.qut.edu.au/research/plas/projects/cp_files/virtual_machines.pdf

The example they pickec, a Pascal dialekt, is IMHO nearly meanless for 
all "interesting" modern languages with high level abstraction, object 
orientation and so on.  For fast execution of OO lanuages, it doesn't 
matter how fast you can add two numbers but how fast GC and method 
calling works in very dynamic environments where you will reload 
classes, deal with large amounts of differently sized memory chunks, 
where you need to run applications for days without failure and so on.


bye
-- 
Stefan Matthias Aust   //
www.3plus4software.de // Inter Deum Et Diabolum Semper Musica Est