[Mono-list] First mono experiences / some benchmarks

[Stefan Matthias Aust]

Hi!

Some time ago, I manually ported two Java programs to C# to have some
non-trivial examples to compare the runtime behavior of Java and C# (and to
learn C#).  Today I installed mono4windows-0.11 and tried to run these
programs.  In case you're interested in the result, I'll post them here.
I'm no regular reader of this group, just scanning over the list archive
once or twice a month, so please contact me directly if you have questions.

I hope, my informations are helpful or at least encouraging for the project.

Both applications are command line applications, a tiny smalltalk
interpreter (derived from Tim Budd's Little Smalltalk System, originally
written in C, rewritten by myself in Java and now ported to C#) and JScheme,
a freely available scheme interpreter written by Peter Norvig in Java (and
ported by myself to C#).  I found Java and C# very similar.  So I did the
port by applying some obvious search&replacements, compiling the stuff and
doing more replacements until all errors vanished.  I'm no C# hacker yet,
but I think I managed to find the corresponding methods.

Both applications run with Mono!

The smalltalk system mostly exercises the GC.  It generates an enormous
amount of garbage (short lived objects) while it first compiles the entered
string into its own bytecode representation (which is then interpreted by
Java/C#) and then running it against its own library.  I used printing "100
factorial" as benchmark. This is a large number and the Smalltalk system has
its own LargeInteger implementation which is written in Smalltalk.
Evaluating that single statement results in more than a million of methods
calls. The interpreter uses arrays of objects to represent the Smaltalk
stack and a lot of tiny methods, so there's plenty of room for
optimizations.

The scheme system is a very straight forward, traditional implementation of
Lisp (or the Scheme dialect) using cons-lists for environments and so
generating a lot of garbage, too.  It's however not as heavy weight as the
Smalltalk system and other aspects like interning strings, converting data
types and walking along linked lists are more prominent. I used the (famous)
Gabriel-benchmarks to benchmark Jscheme and (as I called it) Nscheme.

                                littlest
                        start/stop factorial internal
Sun java 1.3              0.4s     1.3s        0.9s
IBM java 1.3              1.0s     2.4s        1.5s
Sun J1.4/client mode      0.6s     1.5s        0.9s
Sun J1.4/server mode      1.4s     ----        0.8s
JET 2.5                   0.7s     1.7s        1.0s
MS .NET                   0.4s     1.6s        1.2s
Mint                      2.0s    64.2s        ----
Mono                      0.4s     2.8s        ----

All times are measured with "time", the cygwin command, on an 1GHz Athlon
system running XP.  I always ran the benchmark once to warm up the system
and then took the average of three runs.  The JDK 1.4 server hotspot has to
warm up while running so I measured three consecutive compulations while the
interpreter was running.  "Internal" is the interpreter's own timing.  With
mint and mono, this gave very strange results so I guess the method I used
isn't implemented yet.  JET is a Java native code compiler whose latest
version, 2.5, got really close to the latest VM.  Prior version were much
slower.

I also tried "300 factorial" to get some better timings, but I got bored to
wait for Mint and mono causes my system swap to death.  The Java server mode
VM can enlarge its jutty with this benchmark, however.

Stil, I think, Mono is quite close.  IBM's JVM, which is known to be quite
fast, isn't much faster in this benchmark.  So even with this GC-heavy
benchmark, mono is within 50% of the best available VMs.  I like that.

I can't really benchmark nscheme, as, for what reason ever, the interpreter
doesn't terminate on System.Environment.Exit(0).  The Smalltalk interpreter,
however, does. Update: I found a workaround: "</dev/null" will work - for
some reason it seems to read from stdin when it shouldn't.

                     start/stop     TAK  DERIV PUZZLE
Sun J1.4/client mode      0.4s     4.8s   7.4s   1.7s
Sun J1.4/server mode      0.4s     3.6s   4.8s   2.9s
JET 2.5                   0.6s     3.9s   5.5s   1.3s

MS .NET                   0.4s    17.3s  27.0s   3.9s
Mint                      0.3s   587.3s  -----   -----
Mono                      0.3s    11.3s  17.0s   2.8s

With this benchmark, Mono is even significantly faster as .NET (although not
as fast as Java)!  I got bored to try Mint on the other benchmarks after
waiting more than 9 minutes for the first one.

Then, to get an impression of the MCS compiler, I tried to recompile both
projects. Unfortunately, this didn't work.  With nscheme, the compiler
throws an exception (and reports a lot of warnings of the MS compiler as
errors).  With littlest, it detects some compiler errors.

Here's the exception:

.\Primitive.cs(346) error CS0103: The name `d' could not be found in
`jscheme.Primitive'
.\Primitive.cs(347) error CS0103: The name `d' could not be found in
`jscheme.Primitive'
(process:2144): ** WARNING **: unhandled exception
System.NullReferenceException
: "A null value was found where an object instance was required"
in Mono.CSharp.Conditional:DoResolve ()
in Mono.CSharp.Expression:Resolve ()
in Mono.CSharp.Argument:Resolve ()
in Mono.CSharp.Invocation:DoResolve ()
in Mono.CSharp.Expression:Resolve ()
in Mono.CSharp.Return:Emit ()
in Mono.CSharp.Block:Emit ()
in Mono.CSharp.Switch:TableSwitchEmit ()
in Mono.CSharp.Switch:Emit ()
in Mono.CSharp.Block:Emit ()
in Mono.CSharp.Block:Emit ()
in Mono.CSharp.Block:Emit ()
in Mono.CSharp.EmitContext:EmitTopBlock ()
in Mono.CSharp.Method:Emit ()
in Mono.CSharp.TypeContainer:Emit ()
in Mono.CSharp.RootContext:EmitCode ()
in Mono.CSharp.Driver:MainDriver ()
in Mono.CSharp.Driver:Main ()

Here's line 346-347 of Primitive.cs

    case TRUNCATE:      d = num(x);
                        return num((d < 0.0) ? System.Math.Ceiling(d) :
System.Math.Floor(d));

and the Variable "d" is declared a few lines above, in another case.  I
know, bad style but both the Java compiler and MS C# compiler accept this.
Actually they don't allow to redeclare the variable.

Actually, Primitve.cs is one big switch defining all built-in Scheme
functions.  Perhaps that switch is too complex for MCS?

The error in littlest is probably related to the timing-problem.  It's an
error because it seems that some library function is missing:

$ mcs.bat *.cs
.\Context.cs(328) error CS1502: The best overloaded match for method
'System.TimeSpan op_Addition (System.TimeSpan, System.TimeSpan)' has some
invalid arguments
.\Context.cs(328) error CS1503: Argument 1: Cannot convert from 'string' to
'System.TimeSpan'
.\Context.cs(328) error CS0019: Operator + cannot be applied to operands of
type
 `string' and `System.TimeSpan'
Error: Compilation failed
RESULT: 1

Here's line 328:

 Console.WriteLine("[" + (DateTime.Now - time) + "]");

Time is declared as

 static DateTime time;

and initialized somewhere else as

 time = DateTime.Now;


So, while I'm a little bit disappointed that I couldn't recompile my code
yet, it's pretty good to see, that the mono project evolves and that
there'll be hopefully eventually a working Linux version and an alternative
implementation on Windows. While I'll stay with Java for now, I start to
like C# and - being a language freak - I especially like the .NET idea of
having different languages running on the same platform - which shouldn't be
always Windows.


bye
--
Stefan Matthias Aust   //
www.3plus4software.de // Inter Deum Et Diabolum Semper Musica Est