Rubinius vs. the benchmark from hell
This Mandelbrot benchmark really highlights Ruby’s weakest bit. Read the chart. Breathe. Fully internalize that the newest, hottest Ruby interpreter takes an hour and twenty minutes to do a simple render that C can do in 23 seconds.
Yup. Ruby is 200x slower than C here. The benchmark page doesn’t show Ruby 1.8.7 any more, but if you saw it, it would be more than 500x slower than C. On Stack Overflow, according to someone who sounds like they know their stuff, the big boost between 1.8 and 1.9 is that Ruby 1.9 can inline some basic math. I’ll take his word for it; I still can’t read the YARV internals worth beans.
Even with that boost, though, the performance is still bad; everything’s a method invocation, and it really, really hurts to watch this benchmark kick Ruby in its soft and tender bits.
I’ve been very interested lately by the different performance profile of Rubinius. I feel it has a lot philosophically in common with JRuby, with 100% less Oracle dependency. Especially once its hydra branch kills the GIL and introduces true multiprocessing, Rubinius could be something I get real use out of at work.
The best way for me to “benchmark” Rubinius is for me to just run my big Ruby apps on it. On balance for my real-world applications, Rubinius 1.1 seems to perform about the same as 1.8.7, occasionally reaching into the range of 1.9.2. (@headius will be happy to know that under heavy load, JRuby still beats the snot out of all of ‘em. Different story.)
This is a big deal for me; I could actually make a lateral move from 1.8.7 or ree to rbx and nobody would even know! Neat.
Except in one case. I have an old branch of a Rails app, created in-house by another developer, which does a lot of computationally intense operations in Ruby code. It’s a geo app, and those slow computations made this branch utterly unusable. In the shipped version, I delegated all that to GeoServer, where it’s quite peppy. But for giggles, I thought I’d run the original branch under Rubinius.
Okay, I admit, it still sucked out loud. But it was doing its work in much less time than on 1.8.7. Which made me wonder, and made me get out a copy of this depressing Mandelbrot bench.
Rubinius, shockingly, beats 1.9.2 by a noticeable margin on this benchmark. At 1000 iterations, with the JIT very aggressively set, despite the horrifying spaghetti being dumped out by the compiler for this simple bench, 1.9.2 has the win.
rob@hurricane:~$ time ruby mandelbrot.rb 1000 > mandelbrot-yarv.out real 0m19.387s user 0m19.380s sys 0m0.000s
rob@hurricane:~$ time rbx -Xjit.sync=true -Xjit.call_til_compile=1 mandelbrot.rb 1000 > mandelbrot-rbx.out real 0m20.802s user 0m19.360s sys 0m1.120s
But when you take out the compilation time and run for longer at a steady state, Rubinius starts to win! Rubinius takes the lead, on my machine, somewhere between 2000 and 2500 iterations. On the full run (a 16000×16000 render), Rubinius crosses the finish line a whole minute ahead of 1.9.2. Whoa.
[Edit] Here are the final standings amongst the legitimate contenders (leaving 1.8.7 out, ’cause who has that kind of time?)
JRuby: 85m49.466s 1.9.2: 85m10.357s Rubinius: 83m49.293s
[Further Edit] Read the comments below. Rubinius’s Brian Ford links to a gist which explains what’s slow, and optimizes the test script to cut the execution time substantially — almost in half. Haven’t tried it yet, but wow!
What’s potentially very neat is this: as near as I can read the JIT’s output as it flies by, Rubinius’s math *isn’t* being inlined. Looks like all ordinary object operations to me. And based on another quick looping 5-line bench I wrote, floating point divides in Rubinius continue to perform tragically, whereas they’re almost free in 1.9.2.
So I’m guessing — and hoping — that Ruby 1.9.2 and Rubinius have each optimized a completely different aspect of the problem. This bears further study! If the approaches can be combined, it would be great to get Ruby up into the range of merely crummy performance (like Python) on this benchmark, and out of the embarrassing basement.
If the improvements are indeed multiplicative, this would bode well for the ability to do some compute-intensive operations directly in Ruby from time to time when expedient, without always having to drop to a language with primitives and less OO noise. In practice, I see people getting away with little compute-heavy Python scripts all the time. I see nobody daring it in Ruby 1.8.7, for obvious reasons.
Three cheers for Rubinius and Evan Phoenix, anyway, for being the non-JVM Ruby to win this horrid bench for now.