Quite an interesting yet informative rant. I agree with you that benchmarks do depend on a lot of factors, but statistically speaking, if you run it enough times, you should be able to have conclusive evidence that one is faster than another.

Of course, in real life, it doesn’t matter the number of milliseconds of improvements. One can always scale up or scale out, just depends on what you want to do. However, on the other hand, processor intensive applications (graphics, simulation, cad, etc…) do need the fastest languages (and easiest of those.) But I do agree with you, for the majority of webapps, the language you choose should be a personal one and not one based solely on benchmarks.

I saw that the original tester there updated his test. Did he skew the numbers not to look wrong and prove himself wrong? Who knows. I’ll take +/- half a second or so as just normal variation. If you keep running the tests you’ll get different rules. Was there any other processes running on the computer? Did something else spike because it had to perform some kinda check on something in the system? I mean it’s so hard to ever test. However the jump in those test results where all of a sudden a newer version of Ruby is faster than Python…..?? Uh. caubullshcaught bullshcaugh …excuse me. Yea, well it makes perfect sense Python is faster and then you have C++. But I mean he might as well put the numbers as Ruby being faster than C++.

However. Half a second running a sort operation or even 4 seconds running it is of no consequence. What the original tester did point out (which is great that he did) was that who knows when it comes to the application of the language. Ruby on Rails is a great framework but if it slows things down…It slows things down. Now take PHP and it’s frameworks. Say CakePHP which aims to mirror RoR (mostly). Which is faster? At what? Doing what? Real world…Say a blog? Well it depends. Do you know that you can code a “blog” to do the same thing all of a couple thousand different ways using both frameworks??

So testing is a little silly because in the real world you have so many other factors. Plus none of these tests were under any HTTP load. So who knows how they interface and work with Apache. I mean that right there could make the difference. On top of user code.

The one thing I want to point out is the false assumption that none of these languages scale. NO. They ALL scale. Scaling is not defined by how fast a language can run through a loop and sort something. Scaling is done by means of being able to expand your infrastructure to handle the load (in this case web user traffic). If you can code a RoR or CakePHP app to use multiple databases and live in a load balanced Apache environment (and you can, both frameworks are designed for this…and with ease) then you CAN scale. It doesn’t really matter if one language can execute a sort faster or not. Your web app isn’t sorting and if fractions of a second all equate to you needing 50 or 51 servers to handle the traffic….When traffic itself fluctuates and even grows so you’d only need more servers one day anyway….Honestly, who cares? At the point where you’re talking about needing an additional server or two or fifty when you have dozens of them already…You better sure as hell have some kinda revenue paying for all those servers at that point. I don’t think using one language over another is ever going to be proved to equate to thousands of dollars in savings. If so. Where are those benchmarks?

Of course this is even more so the case almost a year later, as JRuby has made even more leaps since then…

Thank you for the test. Thank you big time!