April 11th, 2008
Dromaeo is the name that I've given to the JavaScript performance test suite that I've been working on over the past couple months.
I was hoping to hold off on this release for another week or two, while I finished up some final details, but since it's been discovered, and about to hit the Digg front page, there isn't a whole lot that I can do to stop it.
There's a ton of details concerning how it works, and how to use it, on the Dromaeo wiki page. I won't go through too much of it here, but it should clarify most question there.
Probably the most pressing question that'll be encountered (outside of what is answered on the wiki page) is "What is the relation of Dromaeo to SunSpider?" (SunSpider being the WebKit team's JavaScript testing suite).
Right now I'm working very closely with all the browser vendors to make sure that we have a common-ground test suite that is both highly usable and statistically sound (not to mention providing results that are universally interesting). There are a number of outstanding concerns that've been raised by users of the suite, along with a number of concerns that've already been rectified - again, all of this is clarified on the Dromaeo wiki page. It's of the utmost concern that this suite be as applicable as possible. It's very likely that the core suite will be moving to a common working ground where all browser vendors can work on it.
I especially want to thank Allan Branch of LessEverything who provided the awesome design for the site. It's like he tapped into my brain and produced exactly what I wanted - without knowing even it. I highly recommend them, if you have design work that needs to be done.
Tags: testing, performance, javascript, browsers, mozilla
22 Comments on 'Dromaeo: JavaScript Performance Testing'
December 18th, 2007
A big push of Firefox 3 has been to improve its overall performance (memory, speed, ui responsiveness, JavaScript, etc.). One thing that I wanted to see get a little love was the performance of timers (setTimeout and setInterval). However, in order to make my case, I had to do some analysis.
I built a super-simple test case: Moving a div, one pixel at a time, 100 pixels. I did this using setTimeout, setInterval, and using Mozilla's internal nsITimer interface. I wanted to try and get an accurate reading on a couple things:
- How long of a delay is there between timer calls?
- How much fluctuation is there on timer calls?
- How well do multiple simultaneous timers scale?
- What is the minimum delay achievable with a timer?
Test Case:
All charts were rendered using the jQuery chart plugin, Flot.
The Results (OSX):
Key: The horizontal axis is the recursion depth of the timer call, vertical axis is the actual delay between the timer calls - in milliseconds, and the line numbers correspond to the number of simultaneous timers being called.
The graphs are organized as such: (From top left, going clockwise) Firefox 2, Safari 3, Opera 9, Firefox 3.
setTimeout: 0ms delay, 10ms delay, 20ms delay
setInterval: 0ms delay, 10ms delay, 20ms delay
nsITimer: 0ms delay, 10ms delay, 20ms delay
Modified Firefox 3: Now, after having run all of those tests, I went through and commented out the timer filtering code in Firefox to see what the results would be. View full results (from left-to-right, nsITimer, setInterval, setTimeout, top-to-bottom 0ms, 10ms, 20ms delays).
Conclusions (OSX-only)
- WebKit's timer code is the thing of dreams. Their results are as smooth as a baby's bottom. Simply incredible.
- Opera is really messy - all over the place in its tests.
- Firefox 2 is generally pretty stabilized, but has nasty spikes in delay (from where the garbage collector kicks in).
- Firefox 3 has less nasty, long, delays - but has weird dips down to 0-2ms (this should probably be looked into).
- Firefox 2, Opera, and Safari all have a bottom window of 10ms for delays - Firefox 3 is now 15ms, for some reason (this should probably be looked into).
- Once you start moving into the range of 64-128 simultaneous timers, you're pretty much out of luck in most browsers.
- nsITimer can't be beat (for speed) with 1-2 simultaneous timers.
- There's something really wrong with nsITimer in Firefox 3.
- Removing the timer filtering code (in Firefox 3) reveals that there's a significant amount of timer logic located in the DOM code - as opposed to the pure timer code.
As noted above, these tests were run on OSX only. If anyone is interested in producing some results from Internet Explorer, Firefox 2, Firefox 3, and Opera, you're welcome to it. In the meantime, Vlad produced some basic results from Firefox 3 on Windows XP (setTimeout, setInterval, and nsITimer). Note how different they look from the OSX results. It may be a safe bet that some of the above results, and conclusions, may be localized to just the OSX platform.
I'm hoping to spawn some discussion from this, to better figure out which of these issues are actually bugs and need to be resolved (in particular, in Firefox 3). Don't worry, Webkit team, you guys are completely off-the-hook.
Tags: javascript, browsers
11 Comments on 'Analyzing Timer Performance'
