[Mono-list] Mono C# versus Java 6 -server performance

[Jonathan Pryor]

On Thu, 2007-11-29 at 14:55 +0000, Robert Hulme wrote:
> > Out of 17 programs, Java was 9 x 1.xx times better, 2 x 2.xx times
> > better and 2 times significantly better. 

The short version is that it's JIT dependent.  The processor used is
very important -- you're looking at the "gp4" results, which is for a
Pentium IV.  The AMD Sempron has different results:

	http://shootout.alioth.debian.org/debian/csharp.php


        Test:              P4      Sempron  Notes:
        ----               -----   -------  -----
        binary-trees       1.3 J   1.6 J    Java faster on Sempron
        fannkuch           1.6 J   1.6 J    
        fasta              1.2 J   1.2 J
        k-nucleotide       1.3 J   1.2 J    Java not as much faster
        mandelbrot         2.2 J   1.4 M    Mono upset
        n-body             1.8 J   1.2 J
        nsieve             1.0 M   1.0 M
        nsieve-bits        1.1 J   1.5 J    Java faster on Sempron
        partial-sums       1.7 M   1.5 M    Mono not as much faster
        pidigits           6.1 J   1.5 M    Huge Mono upset.
        recursive          1.1 J   1.6 J
        regex-dna          7.9 J   7.2 J    Java not as much faster
        reverse-complement 1.2 M   1.4 M    Mono faster
        spectral-norm      1.9 J   1.4 J    Java not as much faster
        startup            1.9 M   1.3 J    Java upset
        sum-file           2.5 J   2.3 J    Java not as much faster

So one of the P4 Java blowouts -- pidigits -- is *faster* on
Mono/Sempron.  Mono's Mandelbrot is also faster than Java on the
Sempron.  Startup got worse on the Sempron (which really doesn't make
sense, as `mcs --help' is *always* _much_ faster than `javac -help' for
me).

Alas, they don't have AMD64 platforms up.  I ran mandelbrot on my Intel
Core2 Duo processor, and got (times averaged over 5 runs):

        N     120    300    1000   10000
        -     ---    ---    ----   -----
        Java  0.180s 0.225s 0.421s 19.051s
        Mono  0.099s 0.131s 0.425s 33.894s
        
The shootout only does N=120 and N=300, so the shootout would list Mono
as being 1.7x - 1.8x faster.  It's when N gets much bigger that things
go pear-shaped.

So below 1000, Mono is faster, they're roughly equal at 1000 iterations,
and things get worse afterward.  As for the performance difference, it
seems to be algorithmic in nature.  "Porting" the Java code to C# yields
a performance increase over the shootout's C# code, getting 29.075s for
10000 iterations -- still worse than Java.  I have no idea why Mono does
worse on the larger iterations; I imagine it has to do with the assembly
Mono generates not being as optimal as what Java generates.

Note that this is done with the Mono 1.2.5 as provided with openSUSE
10.3.  We have a 1.2.6 release due out shortly, which will change things
(GC improvements, code gen improvements, etc.).

> OK, let's say they're special cases. That's fine. It still doesn't 
> explain why Java is 20%+ faster in most of the benchmarks though. For my 
> money it should be the same, or (if claims about CIL versus byte code 
> are to be believed) Mono should be faster.

IL vs. bytecode is only part of the issue.  The other part is quality of
the JIT, and Java's JIT is much older than Mono's.  Frankly, I'm
frequently surprised that Mono's is as good as it is, considering how
much more work has gone into Sun's JIT.

 - Jon