[Mono-devel-list] JIT profiling/benchmarking

[Ben Maurer]

On Mon, 2003-12-29 at 23:36, ppham at mit.edu wrote:
> Hi all,
> 
> I am interested in profiling/benchmarking the JIT compiler in the Mono runtime
> to see how long JITting takes as a function of method length (in IL 
> instructions). My objective is to find (roughly) the break-even point where
> pure interpretation and JIT compilation are equally fast.

IMHO, you are taking this from the wrong angle.

Most applications do not involve multiplying decimals in tight loops.
Thus, this test is not representative of the performance difference
between and interpeter and a jit on any practical application.

As well, the frequent throwing of exceptions is not a measure of a
correctly written application. Exceptions are designed to be thrown when
an application has an *error*. 99.99% of the time, the user should see
the exception, and it should result in either an aborting of the program
or a reentry of data.

Also, note that the Decimal code is just a set of function calls. If you
want to test the performance of the JIT, you should be using things that
are emitted as ASM instructions.

Also, make sure you have run with -O=all for mono.

> It was my assumption that the first time a method is run, it is JIT compiled and 
> cached, and for every call after that the cached/compiled version is run.
> However, that seems not to be the case as I cannot measure any significant
> difference between the first time a method is run and every subsequent time.

That may have something to do witht the delegates.

> Also, mint, the Mono interpreter, appears to run these methods faster than
> mono, which has JIT by default. I am further puzzled by mint's profiling
> output which includes statistics such as "Time spent in compilation" and
> "Slowest method to compile", which shouldn't be in an interpreted CLR at all.
Grep for mono_profiler_method_jit in iterp.c. That is where the times are coming from.

> If someone could explain how to better benchmark JIT times in Mono, or even
> in .NET on Windows for comparison, I would greatly appreciate it.
> Snippets of my benchmarking program are below.
You probably need to write your own profiler module here, take a look at
profiler.c in mono/mono/metadata. Also, look at the sample profiler in
samples.

I would also remind you that the only true measure of how good one of
these methods is would be to run real applications with it. Eg, MCS, or
monodoc.

Another interesting area to look at would be the performance of our AOT
compiler.

-- Ben