[Mono-list] JVM performance: JVM as a basis for CLR

[Jay Freeman (saurik)]

Tom:

Oh, I totally agree.  See, I'm also a Math major, however, and I have to
grit my teeth when people use inappropriate proofs to show things,
regardless of whether or not they are true (albeit in our biased opinions,
hehe).

There _is_ only one data point.  Let's say we were looking at the first car.
The very first car was built off of technology that was easily improved to
become a rather speedy little thing (even if not capable of approaching
_near_ the speeds of our current engine designs which work off a slightly
different principal).  Now, if you compared that car to a survey of all the
avaiable breeds of horse, and found that only a few (such as the Clydsdale)
were slower, have you shown that we should be examining horses better would
have resulted in a better automobile?  No.  You would only have proven what
you directly showed, which is that the only available implementation of car
was, in general, slower than the available implementations of horse.

Ximian (in what I believe to be arrogance) believes they can construct a
better implementation of .NET in a few months to at most a year by
coat-tailing the design decisions Microsoft spent years figuring out (as
I've shown before, they've been working on this for at least 3 years with
the understanding that it was going to be some runtime working off of COM
concepts if not COM itself), and tacking on a bunch of haphazardly gained
knowledge from academia (which may very well be patented, and they don't
seem to care).  Due to this, any argument that Microsoft's implementation is
slower than a JVM are inapplicable.

The question of optimization of Beta products doesn't have to do with
whether or not their technology is up to snuff.  Not only did Microsoft have
a _mature_ JVM, they also had the _FASTEST_ JVM (note the "had").  The
question is whether or not they compiled their code to be heavily optimized
or not.  Let's say you have a program, and want to distribute it to end
users, and you are't sure if the logic in the code is even correct.  If you
sent it out heavily optimized, and someone ran across a bug, it is going to
be damned difficult to figure out where the bug in the original code was.
This is why debugging symbols don't make sense when you compile with any
optimizations: the code gets scrambled.  Considering their existing compiler
seems trustworthy to not destroy the code during optimizations, and that
bugs introduced in that way are either going to be so difficult to test that
no one will notice them until it is time for SP1 or so blaringly obvious
that it isn't going to get out the door, I know _I_ wouldn't have shipped an
optimized beta.

The code may be perfectly capable of operating faster, it simply might not
be set up for it right now.  Now, sure, if the code that the JIT outputted
wasn't optimized that would be a bad usage of beta test (as that could be
affected by logic), but think about the garbage collector.  The garbage
collector has to run a lot in memory intensive operations (such as the
in-memory test of sharphsql).  Is there any reason to have shipped an
optimized garbage collector?  No.  To me this is a major consideration when
examining those performance tests.

Let's attack their argument at the achilles heel before we try to use
benchmarks and claim that they have a long academic and practical history...
most of the information out there in both computer classes and trade
magazines is that benchmarks _aren't_ useful tests of practical performance
(rather, just like your argument above, they only test what they test, the
ability of a system to run benchmarks), if you want me to give you
references I'd be happy to.  Even worse, implementations are often tweaked
to better run specific kinds of common benchmarks (sometimes even at the
EXPENSE of regular code) for purposes of making a better showing to those of
us who believe in such things (once again, you just tested that the system
is good at running benchmarks).  Once again, I'd be happy to provide
references.

The standard basis of the argument for the CLR's fundmental speed
improvements over Java is... well, now isn't that interesting.  There isn't
one; at least one that I know of (so if anyone knows one, speak up).  To me
a lack of argument is a pretty good indication of something being seriously
wrong with the assumption.  Some might claim that "tail." is an important
feature for speed of functional languages, but that isn't an advantage in
the slightest.  It is important to note that many JVM's do tailcall
optimization, detecting functions that would benefit and _automatically_
emitting the right code, even if the original byte-code didn't explicitely
specify it (and AFAIK C# never does anyway).  ORP, obviously my favorite
JIT, happens to be one of the ones that does this.  If you have such a JIT
you don't _need_ that instruction.  Now, yes, when Java is being executed
low-level on hardware _without_ a JIT, it wouldn't be able to take advantage
of this.  However, that is an unfair comparison as .NET is terribly unsuited
for this due to both its untyped operations and its arbitrary sized stack
slots and ends up requiring at least some sort of simple JIT to maintain its
level of performance.

I'd guess that a big advantage should be the file format.  I don't see how
Java classloaders expect to operate that quickly (especially after having
examined how Java's class reference pools work).  However, A) I'm probably
ill-founded on that point, B) this would only affect loading and not
execution, and C) the Java specifications actually state that the on-disk
storage of classes is left up to the implementation, and is not defined by
spec.  Current implementations use folder hierarchies that are either mapped
on-disk or compressed into zip/jar files.


For performance evaluation of ORP and friends, I point you to:
http://www.shudo.net/jit/perf/ .

Seeing as you like benchmarks, I'll use an argument appropriate to you.

You should be interested to see that ORP's O3 JIT kicks _ass_ at Linpack as
well as Eratosthenes Sieve.  Really, it only occassionally falls down when
compared against IBM's JVM (which is a pretty damned powerful runtime).  It
also holds one of the top spots on SciMark 2.0, although not by such wide
margins.  To me the most interesting, however, is SPEC JVM, which runs
semi-real-life applications against a JVM and sees how well they do.  Here
ORP did well at a few of the application types, but not others.  I'd be
interested in see a more recent test with the latest version of ORP, but I
doubt things would change _that_ much.

To me the biggest reason to look into ORP is because it is only a JVM
"currently", and was designed to be expanded to support different JITs and
different instruction sets.  Intel has even made the statement that they
might be look into adding CLI support at some point.  To me OpenJIT doesn't
fall into this much as it would require a Java implementation to exist to
bootstrap it anyway.  Also, as much as I think "free software" pushes the
limit, OpenJIT's license isn't GPL compatible.  While I strongly feel a
BSD-type license is "a good thing", I feel at least being GPL-compatible is
as well.  Verified by RMS, through Miguel (who has no reason to lie about
this), ORP is GPL-compatible.  On a technical implementation side, while it
can likely be done, I would think it difficult to get a high-level JIT such
as OpenJIT (which is actually written in Java) to allow for seamless
integration of unmanaged code with managed (something that I consider, as an
MC++ developer, to be one of the most important aspects of the runtime),
which is also something that turns me off from it.  This is also something
that might be non-trivial with any JVM implementation, however.

As for the status of my project, at the time I was spending my time working
on a metadata parsing engine (specifically a reimplementation of Microsoft's
CorMetaDataDispenser & friends for compatibility sake).  That was the
original goal of the project.  A few people were interested, Miguel, Rhys,
and a few people from the DOTNET mailing list.  None of them actually wanted
my code, they wanted what I had figured out about the file format (this is
before the ECMA documents for metadata were released, Rhys figured out how
the tables were stored, but not what was in most of them... I had that bit),
they wanted the fact that it was even possible, but _none_ of them wanted
the code to actually use it.  Rhys wanted a C version that he owned all the
copyright for, and actually wanted to see mine just so he could do a one-man
clean room reimplementation of it (frankly, I would have been happy to
expose a C interface to everything, as I was looking at COM compatibility
anyway and COM is primarily a C API structure).

This isn't the kind of motivation you want on a project.  One of the people
from the DOTNET mailing list, one of the big reasons I was interested in the
project in the first place, recently asked if he could see it so he could
see some of the concepts involved for writing his own C# version.  Frankly,
I'm fed up.  At this point it is a more efficient usage of my time to just
write a document and hand it to people than write any code, but then you
might as well use Microsoft's specifications.  I've been taking my
high-level API (Metallurgy) and converting it over to MC++ over the last few
days.

I still have a lot of my Chordata code (which can to go into a Beta 1 binary
and find all of the classes and methods, enough for initial testing), and we
could even use Rhys's metadata parsing code (which I believe is also
currently setup for Beta 1, but either could be moved without too much
difficulty, although it would be annoying to move all of ORP over to GPL
just for this one piece... Rhys might be flexible in licensing, but I'm not
so sure).  Although, after having given this much thought over the last few
months, to me the way to go about this is to start even smaller, first step
being to rip apart a Java class file and stick MSIL in it.  There is
actually an applicable abstraction that can be made for "tokens" between the
two languages.  The largest annoyance in this form of casting are local
variables, which I don't think is going to be too terrible.

Sincerely,
Jay Freeman (saurik)
saurik@saurik.com

----- Original Message -----
From: "Tom" <tom7ca@yahoo.com>
To: "Jay Freeman (saurik)" <saurik@saurik.com>; "mono-list"
<mono-list@ximian.com>
Sent: Sunday, July 22, 2001 5:10 AM
Subject: Re: [Mono-list] JVM performance: JVM as a basis for CLR


> > Just because a specific implementation
> > of the CLR is worse off
> > than a specific implementation of a JVM doesn't mean
> > that Mono couldn't be a better implementation.
>
> I provided one data point, not a comprehensive
> and exhaustive study.  I hope other people will
> provide other data points and technical arguments.
>
> This particular comparison is doubly important
> because Miguel, I believe, claimed that even an
> unoptimized CLR implementation beats Java.
>
> > Secondly, the comparison also doesn't really mean
> > anything. I don't see what you are trying to show
> > by this... that Java byte-codes are a better
> > intedmediate representation than Microsoft IL for
> > writing JITs?
>
> No.  I view the discussion of using the JVM as
> irrelevant to this list.  Mono is a project involving
> C# and CIL, and it's clear Mono isn't going to use
> Java/JVM at this point no matter what.  The question
> is whether a JVM is a good basis for implementing
> a CLR.  Miguel and Microsoft have talked about
> supposed limitations of the JVM instruction set that
> they think make this a bad choice.  And there I think
> is still time for taking another path.
>
> > You are comparing Beta 2 of a
> > product that has _specifically_ been mentioned in
> > the past by Microsoft to not have been optimized
> > for speed for purposes of the beta tests against a
> > standing JVM.
>
> Microsoft had a mature JIT as part of their JVM,
> and it pretty clearly should have taken only a small
> effort to turn that into the CLR.  I think this claim
> is just spin, and I think the CLR is likely to be
> as good as it gets.  In fact, Microsoft's claim
> that CLR is close to C++ performance contradicts
> their claim that they will still be able to do lots
> of optimization on it.  In any case, when it comes
> to CLR, we have to make do with the choices we have.
> Promises of future enhancements are just promises,
> often unfulfilled if past experience with language
> vendors is any guide.
>
> > Worse yet, you are using a rather abstract test case
> > that you don't actually
> > run into much in the real world (even in low-level
> > stuff like 3D engines).
>
> I don't see anything "worse" about it.  Those happen
> to be two characteristics that matter to me.
> Microbenchmarks are important for understanding
> the characteristics of language implementations and
> they have have a long academic and practical history.
>
> If they exist, I would very much be interested in
> seeing other small test cases like this where the CLR
> outperforms Sun's JVM/JIT.
>
> > If you want a better comparison, check out
> > http://www.codeproject.com/useritems/sharphsql.asp .
>
> Yes, I'm aware of that, and I was assuming others
> were as well.  As I indicated, my results point
> in the same direction as those other results.
>
> > At this point I'd normally mention the project I
> > started a few months ago to work off of ORP
> > (the one I couldn't get any interest from anyone
> in),
>
> Well, great!  I think people should have listened
> to you more and participated.  (Maybe you should
> have challenged people more with actual benchmarks
> and discussions :-)
>
> > but I'm quite soured on it at this point and _I'm_
> > not even interested in it anymore :-).
>
> Do you have actual performance results on the ORP?
> How does it compare to Sun's JVM and the OpenJIT?
> Have you actually made useful modifications to
> the ORP yet and are those available?
>
> Let me state it clearly again: I am really interested
> in seeing and understanding test cases where a CLR
> (theoretically or practically) outperforms a JVM.
> I'm asking out of technical interest, to understand
> both systems better, not to win an argument.
> But the existence or non-existence of
> such cases also should be an important data point
> for the choices Mono makes wrt. a CLR implementation.
>
> Tom.