[Mono-dev] IronPython Performance

[Paolo Molaro]

[I continued your large cross-post, even if I'm not subscribed to
python-dev: hopefully the moderators will approve the post if needed.]

On 04/21/06 Brent Fulgham wrote:
> A while ago (nearly two years) an interesting paper was published by Jim Hugunin
> (http://www.python.org/pycon/dc2004/papers/9/) crowing about the significant
> speed advantage observed in this interpreter running on top of Microsoft's .NET
> VM stack.  I remember being surprised by these results, as Python has always
> seemed fairly fast for an interpreted language.
> 
> I've been working on the Programming Language Shootout for a few years now, and
> after growing tired of the repeated requests to include scores for IronPython I
> finally added the implementation this week.
> 
> Comparison of IronPython (1.0 Beta 5) to Python 2.4.3 [1] and IronPython (1.0 Beta 1)
> to Python 2.4.2 [2] do not match the results reported in the 2004 paper.  In fact,
> IronPython is consistenly 3 to 4 times slower (in some cases worse than that), and
> seemed to suffer from recursion/stack limitations.

You're comparing different benchmarks, so it's not a surprise that you
get different numbers.
I only tried two benchmarks from the paper, pystone and the globals one
(attached) and the results are largely equivalent to the ones from the
paper:
time mono IronPython-1.0-Beta6/IronPythonConsole.exe -O python-globals-bench2.py 

real 0m0.861s
user 0m0.815s
sys  0m0.043s
(Note the above includes the startup time that I guess was not
considered in the paper and is 0.620 seconds running test(1): just
increasing the number of iterations will make the difference converge at
about 10x faster mono vs python2.4).
time python2.4 -O python-globals-bench2.py 

real 0m2.239s
user 0m2.202s
sys  0m0.025s

python2.4 -O /usr/lib/python2.4/test/pystone.py
Pystone(1.1) time for 50000 passes = 1.4
This machine benchmarks at 35714.3 pystones/second

mono IronPython-1.0-Beta6/IronPythonConsole.exe -O /usr/lib/python2.4/test/pystone.py
Pystone(1.1) time for 50000 passes = 0.989288
This machine benchmarks at 50541.4 pystones/second

So IronPython is 40+% faster than CPython, like in the paper.

The mono version I used is roughtly 1.1.14.

As for the recursion limitations: what are they? I took the recursive
benchmark from the shootout page and both CPython and IronPython need
the call to setrecursionlimit (actually cpython needed it earlier for
me, anyway). This might be a difference between IronPython beta 5 and
the beta6 I'm using, though. It is also interesting because on my system
running the attached python-ack.py with 8 as argument, mono/IronPython
is 10% faster than CPython, while the shootout page reports CPython 2.4
50% faster. Feel free to investigate if this is because of improvements
in mono or IronPython:)

> I am aware of the following issues that might affect the results:
> 1)  Mono vs. Microsoft's CLR.

This can certainly be a factor: some things in the MS runtime run faster and
some slower than in the Mono runtime, so it depends which features are
exercised the most in a particular benchmark. It is very likely that in
most benchmarks the MS runtime will score better.

> 2)  Python 2.1 vs. Python 2.4

The python folks will have more details, but python2.4 seems to be
faster than 2.1 and 2.3 in several areas.

> 3)  Changes in IronPython over the last two years.

For some time the IronPython folks have focused on features and
correctness issues, so I guess they'll start caring about performance in
the future. It is also to note that IronPython now uses generics which
have yet to see optimizations work in the mono runtime and libraries
(they are a 2.0 feature).

> I thought these findings might be interesting to some of the Python, IronPython,
> and Mono developers.  Let the flames begin!  ;-)

Why flames? Hard numbers are just that: useful information. It's just
your conclusions that might be incorrect;-)

I think the summary to take away is:
*) Sometimes IronPython on mono is faster than CPython
*) Often CPython is faster than IronPython on mono
*) IronPython and mono are improving quickly.

Thanks for the numbers: it would be nice if someone with time in his
hands could rerun the benchmarks with ironpython beta6, mono from svn
(or at least mono 1.1.14/1.1.15) and also with the MS runtime on windows.
Better if this includes running pystone to relate to improvements in the
published paper and the piethon benchmarks (on the MS runtime, in mono
we need to implement some things that newer ironpythons use). In the
meantime here are the piethon results running with IronPython 0.6 with a
current mono.

Mono results:
b0 = 2.49623870849609 -- [2.59130096435547, 2.40117645263672]
b1 = 0.473392486572266 -- [0.496170043945312, 0.450614929199219]
b2 = 0.453083038330078 -- [0.543968200683594, 0.362197875976562]
b3 = 0.967914581298828 -- [0.995307922363281, 0.940521240234375]
b4 = 1.11585998535156 -- [1.23159027099609, 1.00012969970703]
b5 = 3.56014633178711 -- [3.70928192138672, 3.4110107421875]
b6 = 1.43364334106445 -- [1.42024230957031, 1.44704437255859]
all done in 21.01 sec

Python 2.4 results:
b0 = 2.185000 -- [2.1800000000000002, 2.1900000000000004]
b1 = 0.915000 -- [0.93999999999999995, 0.88999999999999879]
b2 = 0.395000 -- [0.39999999999999991, 0.39000000000000057]
b3 = 1.015000 -- [1.02, 1.0099999999999998]
b4 = 0.585000 -- [0.58999999999999986, 0.58000000000000007]
b5 = 0.995000 -- [0.99000000000000021, 1.0]
b6 = 1.410000 -- [1.4199999999999999, 1.4000000000000004]
all done in 15.00 sec

Python 2.3 results:
b0 = 2.270000 -- [2.2800000000000002, 2.2600000000000016]
b1 = 0.955000 -- [0.98999999999999977, 0.91999999999999993]
b2 = 0.395000 -- [0.39000000000000012, 0.39999999999999858]
b3 = 1.230000 -- [1.2300000000000004, 1.2300000000000004]
b4 = 0.805000 -- [0.80999999999999961, 0.80000000000000071]
b5 = 1.145000 -- [1.1500000000000004, 1.1399999999999988]
b6 = 1.500000 -- [1.5099999999999989, 1.4900000000000002]
all done in 16.60 sec

Since Guido designed it to excercise the tricky corners of the python
implementation this could be a better general evaluation of the speed
of a python runtime. Note that in the oscon paper, mono was reported to
be 23 time slower than CPython (but 2 times slower excluding b5 where we
had a really unoptimized codepath). Current mono is 27% slower than
CPython 2.3 and 40% slower than CPython 2.4. So while the CPython
performance increased, mono's rate of improvement is significantly
faster.

While having mono/IronPython be faster than CPython in more benchmarks
would be nice (and I guess we could run a few profile runs if there is
interest to see if some quick specific improvement could be done),
it is not really necessary.
Having mono run python code 1-2 times slower than CPython is pretty
good. People interested in raw performance could just write the code in
C#, gluelessly use it from python code and get the 1-2 orders of
magnitude improvements visible also in the shootout comparison of C# vs
python.

Thanks.

lupus

-- 
-----------------------------------------------------------------
lupus at debian.org                                     debian/rules
lupus at ximian.com                             Monkeys do it better