[Mono-list] JVM performance: JVM as a basis for CLR

[Rhys Weatherley]

Tom wrote:

> This particular comparison is doubly important
> because Miguel, I believe, claimed that even an
> unoptimized CLR implementation beats Java.

Maybe it's time I posted the following:

Rhys's Laws of VM Performance:

1. It's never as fast as you think it is.
2. Benchmarks are essentially useless because
    they can be made to measure the wrong thing.
3. Comparing two different implementation
    techniques will always result in bogus figures.
4. It's never as fast as you think it is.

I've written a lot of VM's over the years, and I've
fooled myself many times into thinking I have a faster
solution than someone else.  I've usually been wrong,
but it took a while to discover that.

The reason why Miguel is seeing the Mono beat
the pants off the JVM is very simple: VM size.

The Mono VM is so small right now that it loads
very quickly and doesn't overly tax the OS's
virtual memory system.  The JVM plus class
libraries is huge in comparison.  So, I suspect
that the Fib numbers quoted earlier were
measuring load time and OS paging performance,
and not the benchmark itself.

Eventually Mono will grow in size to the point
where the size advantage will disappear.  At that
point, no amount of optimization will make the
Mono interpreter faster than the JVM.  That's
because the polymorphic instruction set requires
more clock cycles per instruction in the interpreter
than the Java instruction set, but roughly the same
number of instructions.

Changing to a JIT will improve runtime performance,
but will actually impede load time performance, due
to the overhead of re-compiling the code.  This means
that compute-bound benchmarks will run faster,
but I/O-bound benchmarks will run slower.

VM performance is a many headed beast.  Cut one
off and two more rise up in its place.  There is no
way to understand it fully, and comparisons between
different technologies are essentially meaningless.

The real question is this: is it possible to build a
VM that gives acceptable performance on the most
common applications that a user is likely to use in
practice.  Since users don't use Fib, there's really
no point optimising for it.

Cheers,

Rhys.