Dromaeo: JavaScript Performance Testing

April 11th, 2008

Dromaeo is the name that I've given to the JavaScript performance test suite that I've been working on over the past couple months.

I was hoping to hold off on this release for another week or two, while I finished up some final details, but since it's been discovered, and about to hit the Digg front page, there isn't a whole lot that I can do to stop it.

There's a ton of details concerning how it works, and how to use it, on the Dromaeo wiki page. I won't go through too much of it here, but it should clarify most question there.

Probably the most pressing question that'll be encountered (outside of what is answered on the wiki page) is "What is the relation of Dromaeo to SunSpider?" (SunSpider being the WebKit team's JavaScript testing suite).

Right now I'm working very closely with all the browser vendors to make sure that we have a common-ground test suite that is both highly usable and statistically sound (not to mention providing results that are universally interesting). There are a number of outstanding concerns that've been raised by users of the suite, along with a number of concerns that've already been rectified - again, all of this is clarified on the Dromaeo wiki page. It's of the utmost concern that this suite be as applicable as possible. It's very likely that the core suite will be moving to a common working ground where all browser vendors can work on it.

I especially want to thank Allan Branch of LessEverything who provided the awesome design for the site. It's like he tapped into my brain and produced exactly what I wanted - without knowing even it. I highly recommend them, if you have design work that needs to be done.

Tags: testing, performance, javascript, browsers, mozilla

23 Comments on 'Dromaeo: JavaScript Performance Testing'

Firefox 3 Memory Use

March 13th, 2008

Mozilla developer 'Pavlov' wrote up some extensive details on memory use in Firefox 3. I highly recommend that you check it out.

I borrowed some of his data and created another view of the results. For example, here's the results from Windows Vista of a number of browsers:

Note that both Safari 3 and IE 8 crashed during the test (which was a page runner which automatically opened and closed groups of web pages) so accurate numbers weren't able to be achieved for them. Some preliminary numbers show Safari 3 on a similar path to IE 7 (Stuart mentioned that IE 8 showed a similar path, as well).

It's great to see the massively-improved memory use of Firefox 3. It far excels anything that we offered, previously, and seems to best all other browsers on the platform.

Now, obviously, Windows Vista isn't the only platform available. I, personally, use OS X and am interested to see the memory numbers there as well. One portion of Stuart's blog post that I found to be particularly interesting was the discussion of measuring cross-platform browser memory use - and how difficult it can be. Here's how he explains it:

The short summary is Windows Vista (Commit Size) and Linux (RSS) provide pretty accurate memory measurement numbers while Windows XP and MacOS X do not.

...

On Mac, If you look at Activity Monitor it will look like we’re using more memory than we actually are. Mac OS X has a similar, but different, problem to Windows XP. After extensive testing and confirmation from Apple employees we realized that there was no way for an allocator to give unused pages of memory back while keeping the address range reserved.. (You can unmap them and remap them, but that causes some race conditions and isn’t as performant.) There are APIs that claim to do it (both madvise() and msync()) but they don’t actually do anything. It does appear that pages mapped in that haven’t been written to won’t be accounted for in memory stats, but you’ve written to them they’re going to show as taking up space until you unmap them. Since allocators will reuse space, you generally won’t have that many pages mapped in that haven’t been written to. Our application can and will reuse the free pages, so you should see Firefox hit a peak number and generally not grow a lot higher than that.

I think it's great to see these numbers come in. In many ways Firefox 3 is going to be a very different browser from its previous instantiations. I've, personally, been using it as my primary browser for a while now and have enjoyed the increased performance. It really does feel - and really even look - like a whole new browser.

If you'd like to try Firefox 3, especially without disturbing your current setup, it's really easy - and I've even written up instructions to help you out.

Tags: firefox, performance, browsers, memory

27 Comments on 'Firefox 3 Memory Use'

JavaScript Performance Stack

February 28th, 2008

Something that's frequently befuddled is the differentiation between where JavaScript is executing and where performance hits are taking place. The difficulty is related to the fact that many aspects of a browser engine are reliant upon many others causing their performance issues to be constantly intertwined. To attempt to explain this particular inter-relationship I've created a simplified diagram:

To break it down, there's a couple key areas:

• JavaScript - This represents the core JavaScript engine. This contains only the most basic primitives (functions, objects, array, regular expression, etc.) for performing operations. Unto itself it isn't terribly useful. Speed improvements here have the ability to affect all the various object models.
• Object Models - Collectively these are the objects introduced into the JavaScript runtime which give the user something to work with. These objects are generally implemented in C++ and are imported into the JavaScript environment (for example XPCOM is frequently used by Mozilla to achieve this). There are numerous security checks in place to prevent malicious script from accessing these objects in unintended ways (which produces an unfortunate performance hit). Speed improvements generally come in the way of improving the connecting layer or from removing the connecting layer altogether.
  • XMLHttpRequest and Timers - These are implemented in C++ and introduced into the JavaScript engine at runtime. The performance of these elements only indirectly affect rendering performance.
  • Browser - This represents objects like 'window', 'window.location', and the like. Improvements here also indirectly affect rendering performance.
  • DOM and CSS - These are the object representations of the site's HTML and CSS. When creating a web application everything will have to pass through these representations. Improving the performance of the DOM will affect how quickly rendering changes can propagate.
• Parsing - This is the process of reading, analyzing, and converting HTML, CSS, XML, etc. into their native object models. Improvements in speed can affect the load time of a page (with the initial creation of the page's contents).
• Rendering - The final painting of the page (or any subsequent updates). This is the final bottleneck for the performance of interactive applications.

What's interesting about all of this is that a lot of attention is being paid to the performance of a single layer within the browser: JavaScript. The reality is that the situation is much more complicated. For starters, improving the performance of JavaScript has the potential to drastically improve the overall performance of a site. However there still remain bottlenecks at the DOM, CSS, and rendering layers. Having a slow DOM representation will do little to show off the improved JavaScript performance. For this reason when optimization is done it's frequently handled throughout the entire browser stack.

Now what's also interesting is that the analysis of JavaScript performance can, also, be affected by any of these layers. Here are some interesting issues that arise:

• JavaScript performance outside of a browser (in a shell) is drastically faster than inside of it. The overhead of the object models and their associated security checks is enough to make a noticeable difference.
• An improperly coded JavaScript performance test could be affected by a change to the rendering engine. If the test were analyzing the total runtime of a script a degree of accidental rendering overhead could be introduced as well. Care needs to be taken to factor this out.

So while the improvement of JavaScript performance is certainly a critical step for browser vendors to take (as much of the rest of the browser depends upon it) it is only the beginning. Improving the speed of the full browser stack is inevitable.

Tags: javascript, browsers, performance

11 Comments on 'JavaScript Performance Stack'

A Typical HTML Page

February 20th, 2008

When doing DOM-based performance testing you frequently need to pick a sample HTML document to work against. This raises the question: What is a good, representative, HTML document?

For many people a good document seems to file into one of two categories:

1. A large web page with a lot of content. When we did our initial performance testing with jQuery we used Shakespeare's As You Like It (lots of elements, but a very flat structure) - Mootools uses an old draft of the W3C CSS3 Selectors recommendation. This has a lot of content, as well - thousands of elements with a medium depth structure.
2. A popular web page. Common recommendations include 'yahoo.com' and 'microsoft.com'.

What's troubling is that there doesn't really seem to be any way to determine what a representative web page actually is. There's a couple things that I'd like to propose as being good indicators:

• Standards-based semantic markup (including strong use of attributes: id, class, etc.).
• Non-trivial file size and element count (testing the scalability of the performance).
• Some use of tables and form elements (frequent inclusions in most web pages).
• Strong use of CSS (frequently implies a deep element structure, in order to accommodate complex layouts).
• Pervasive use of JavaScript. If JavaScript performance analysis is being done it's probably good to start with a page that already has a desire to use it.

I think, out of all of these aspects, one page stands out: CNN.com. Here's why:

• It uses semantic HTML 4 markup with a lot of classnames and ids.
• It's about 92kb in size and has 1648 elements in it.
• It has some tables (seemingly for legacy material) and some forms (search forms, drop-downs).
• Lots of CSS and JavaScript. Makes good use of Prototype which, at least, should show an desire in having a page with performant JavaScript.
• It's, also, imperfect. I would consider this to be desirable. Rarely are pages completely without fault - and the heavy use of embedded JavaScript, ads, and non-semantic tables helps to add a stark dash of reality.

Of course, analysis could always be done against multiple pages and be viewed in aggregate but we need a place to start. So, what do you think; is CNN a good, representative, page for doing performance analysis against?

Tags: javascript, css, html, performance

12 Comments on 'A Typical HTML Page'

Selectors that People Actually Use

February 12th, 2008

This post has been a long time coming. It's a combination of my distrust for JavaScript CSS selector performance analysis and my disdain for the CSS 3 Selector specification.

To start, I want to give a little bit of history regarding jQuery's selector engine. When I first started working on its implementation it was mid-2005. It was mostly done as a personal challenge to myself - implementing a specification for kicks. You can see some of my early thoughts in a post that I wrote on Selectors in JavaScript. I completed my implementation on the same day as the first, other, JavaScript CSS selector engine: cssQuery. I then held of and merged it with some of my other efforts, which eventually resulted in jQuery. When I first implemented the engine, I went for full CSS 3 compliance (mashing in XPath-capable queries, as well).

Fast-forward 7 months and jQuery is starting to get a capable community. In preparation for jQuery 1.0 I decide to analyze the features of the engine to see what people are actually using. I ran a poll (which, unfortunately, has been lost) in which I asked the users which selectors they used. As it turned out there were a great number that no-one had any use for, whatsoever (and, in fact, this remains true to this day). At this point I removed them from the engine, breaking compliance with the CSS 3 Selector spec. Here's some of the selectors that were removed:

• E:root - Rarely used in HTML. You already know what the root node is - it's named 'html'.
• E:empty - This might be useful if it could include empty whitespace text nodes, but it doesn't. This will only match elements like <img/> and <hr/>, for whatever use that is.
• E:lang(fr) - This could be achieved in so many other ways - but in the end, how many multi-language-on-the-same-page sites are there?
• E:nth-of-type(n) - I'm not sure what the motivation was for creating all the -of-type methods, I'm sure it sounded great on paper, but in the world of HTML it's not very useful.
• E:nth-last-child(n) - Another "great on paper" method. Don't think I've ever seen it used.
• E:nth-last-of-type(n)
• E:first-of-type
• E:last-of-type
• E:only-of-type
• E:only-child - When does this occur? and why would you need to select it?
• E ~ F - Only selects adjacent elements, in one direction. Why a ~? Why only one direction?
• E + F - Only the next element - rarely useful.
• E[foo~="bar"] - Only matches values in a space-separated list. This is only useful for classes (which is taken care of with .class) and the ref attribute. Why not just use *=?
• E[hreflang|="en"] - Another selector that is really only useful for a single attribute - and not a popular one, at that.

What's fascinating is that no one has ever, ever, requested that these features be added back in. They have virtually zero real-world use and applicability. In fact, with the exception of "E + F" all of these selectors were added, exclusively, in the CSS 3 specification. I'm not completely sure what the thought process was in selecting them, but it's pretty obvious that it wasn't grounded in application, but in theory (which isn't really the spec-writers fault, considering that there were very few CSS 2-compatible implementations at the time).

Only later, after performance test suites started to arrive, did people start to care about the existence of - and the performance of - these selectors (and hence why selectors like +, ~, and [foo~=bar] now exist in jQuery).

To compensate for the shoddy offering of current CSS selectors, JavaScript libraries have had to write whole supersets of selector functionality to compensate for missing features. For example, jQuery includes both new selectors (such as ":hidden" and ":has()") and new selector methods (like ".parent()" and ".prev()") - all of which provide the user which phenomenally more functionality and clarity than the what is in CSS 3.

Now, I'm sure I'll probably get lots of feedback saying "but 'E + F' can be useful, look at this example" or "of course ~= is useful, you can use it on rel attributes" - that's not the point. The fact is that they are woefully un-used. To the point that they are a burden upon the implementors of the specification. What's the point of implementing the above features - or more importantly: optimizing the above features for speed - if no one is using them.

Which leads me to my next bone to pick:

Performance isn't Compliance

Everyone and their brother seems to use the SlickSpeed selector speed test suite. That's fine, as far as implementation goes it's a pretty good take on the matter. It runs quickly, spits out pretty results - users love it. However, it's doing two things - and that's one thing too many: It's testing for both performance AND compliance of the selector engines. For example, if a user were to run the tests and see poor performance for, oh say, :nth-child(2n+1), they would be shocked, nay, appalled at the overall performance of that selector engine. But here's the rub: That's from a selector that is virtually un-used. (:nth-child is occasionally useful, in and of itself, but the An+B syntax is virtually worthless). But this is a point on which SlickSpeed does not care - since it's also testing for compliance, in addition to performance, all tests are treated equally and "without bias."

However, that's precisely what isn't needed: Selectors require bias. I've often argued that the speed of an ID selector is far more important than the speed of an attribute selector (for example) because of how commonly it's used. However, up until this point I've never had data to back up this claim. I have resolved that.

I present to you the most commonly used CSS selectors used by jQuery from 59 of the most popular jQuery-using sites (which was borrowed from the featured sites list). (Fun fact: The use of $(DOMElement) was more popular than all other selectors combined.) Here's a small selection of the selectors found:

View the rest of the selectors with a full explanation...

I spidered the JavaScript of all the sites, parsed through them, and found the appropriate selectors to compile the list. Here's some things that I learned from the data:

There doesn't seem to be a correlation between performance and selector use. For example, ".class" is far more popular than "tag.class" even though the second one is much more performant. What's especially important about this is that the degree of performance hit isn't that much of an issue. For example, the difference between 4ms and 30ms is virtually imperceptible. Instead there is an overwhelming trend towards simpler selectors. Obviously, user education could help, but it's unclear as to how much that will change things in the end.

A couple of jQuery's custom selectors are immensely popular: :visible, :hidden, and :selected. However it is unclear as to how useful they would be outside of a JavaScript-based CSS selector engine (there's no real point in styling a hidden element).

A bunch of jQuery's convenience selectors: :checkbox, :radio, :input, etc. would be quite useful, within a CSS selector spec - and it's good to see them in wide use here.

There's a bunch of unexpected queries that are used: "*", ".class .class", "[name=value]", and "#id #id". These types of queries are grossly under-represented in current performance test suites.

...there's one thing that needs to be taken from all of this data, though: Speed test suites need to test reality rather than specification.

I'm perfectly ok with having two completely separate suites, one focusing on speed and one focusing on compliance, however mixing them does no one any favors: Users get a confused perception and suite authors (and browser vendors!) waste time dealing with optimizing things that don't matter.

My proposal: A standardized performance suite (based on SlickSpeed, is fine) but populating the tests with comparable selectors to the ones shown above and weighted based upon their relevance. Thus, the speed of the #id selector should, actually, consume 51.29% of the total final score. This means that being 3x faster at this test would actually be 102x more important than becoming 3x faster at "tag tag". This is absolutely not represented in any, current, test suites and needs to be rectified. Of course, I'll be happy to seed my selector list with results from other popular sites of other selector libraries.

I'll be fine constructing this suite, as well - I just want to make sure that there's enough interest. I think this proposal has a lot of merit and should be strongly considered - the result will be a selector performance suite which will benefit everyone.

Tags: jquery, javascript, performance, css

70 Comments on 'Selectors that People Actually Use'

XPath Overnight

February 10th, 2008

A fascinating thing has happened in the world of JavaScript DOM traversal: Over the course of a couple months in 2007 three of the major JavaScript libraries (Prototype, Dojo, and Mootools) all switched their CSS selector engines to using the browser's native XPath functionality, rather than doing traditional DOM traversal. What's interesting about this is that the burden of functionality and performance has, literally, flipped overnight on to the browser's XPath engine, from its pure DOM implementation.

There's some really interesting things about this switch:

• Native XPath is blazing fast. For a majority of CSS selectors it completely trumps using native DOM methods (like getElementsByTagName, for example). Sometimes it pays to special-case your code for selectors like #id, but overwhelmingly XPath is the direction in which JavaScript libraries are heading.
• Since a large percentage of JavaScript users use JavaScript libraries (and, thus, use the behind-the-scenes XPath, as well) this means that browsers are now spending significantly more time processing XPath queries than they ever were before. This means that the performance field is now, effectively, split between two areas: Traditional DOM querying and XPath.
• No one is analyzing the performance of browser XPath queries. Or, if they are, it's certainly not public. I'm working on some new XPath performance tests, in order to bring them some more visibility, and hope to have them released this week.
• XPath, while incredibly useful, is a black box. The developer has no control over how fast the results come back - or if they are even correct. Contrast this with traditional DOM scripting (where you can fine-tune your queries to perfection). Browsers will always be bound to have some bugs in their implementations. For example, Safari 3 isn't capable of doing "-of-type" or ":empty" style CSS selectors, nor is any browser able to access the 'checked' property, or namespaced attributes (all noted in Prototype's implementation) which means that they have to fall back to a traditional DOM scripting model.
• Internet Explorer is a dead-end. Since most users want a CSS selector implementation that will work against HTML documents - and IE is unable to provide one - all CSS Selector implementations must provide two (2) side-by-side selector engines in order to handle these cases (not to mention the aforementioned cases where browsers provide unexpected behavior).

A couple things to take away from all of this:

1. XPath (and new methods like querySelector) are the way of the future for a lot of JavaScript libraries - and the next frontier for browser optimization.
2. These implementations are black boxes that are unable to be modified by the developer (leaving them vulnerable to browser bugs).
3. A dual DOM-only CSS selector engine must be provided well into the foreseeable future, by libraries, in order to account for browser mis-implementations.

I should, also, probably answer the inevitable question: "Why doesn't jQuery have an XPath CSS Selector implementation?" For now, my answer is: I don't want two selector implementations - it makes the code base significantly harder to maintain, increases the number of possible cross-browser bugs, and drastically increases the filesize of the resulting download. That being said, I'm strongly evaluating XPath for some troublesome selectors that could, potentially, provide some big performance wins to the end user. In the meantime, we've focused on optimizing the actual selectors that most people use (which are poorly represented in speed tests like SlickSpeed) but we hope to rectify in the future.

Tags: performance, libraries, javascript, browsers, dom

20 Comments on 'XPath Overnight'

The Performance Paradox

February 7th, 2008

Just because a test is good at measuring performance for one metric, doesn't mean that it's good for all metrics. The other day I posted about some JavaScript Library Loading Speed Tests that were done by the PBWiki team. I made some conclusions about JavaScript Library Loading speed that, I think, were pretty interesting - however, I mentioned some browser load performance results (at the end of the post) which were especially problematic. This brings up an important point from the performance results:

User-generated performance results are a dual-edged sword.

Assuming that there's no cheating involved (which is a big assumption) then quietly collecting data from users can provide interesting results. HOWEVER - how that data is analyzed can wildly effect the quality of your results. Analyzed correctly and you can start to get a picture for how JavaScript libraries perform on page load, incorrectly and you might assume that specific browsers are broken, slow, or providing incorrect results.

There's a ton of examples of misinformation, relating to browsers, within the "Browser Comparison" results. I'm just going to list a bunch of issues - showing how much of a problem user-generated browser performance data can be.

• The results show the numbers for Opera being heavily skewed. At first glance one might assume "oh, that's because Opera is slower at loading JavaScript files" - however this is not the case at all. Instead, a more-plausible answer is that users were testing this site from a copy of Opera Mobile (which performs poorly, compared to a desktop browser).
• Both Safari 2 and Safari 3 are grouped together, which is highly suspect. By a number of measurements Safari 3 is much faster than Safari 2, so having these two merged doesn't do any favors.
• Firefox 3 only has two results. A commenter mentioned that this was because they were being grouped into the "Netscape 6" category - which, in and of itself, is a poor place for conglomeration.
• IE 7 is shown as being faster as IE 6. This may be the case, however it's far more likely that users who are running IE 7 are on newer hardware (think: A new computer with Vista installed), meaning that, on average, IE 7 will run faster than IE 6.
• Firefox, Opera, and Safari for Windows users are, generally, early adopters and technically savvy - meaning that they're, also, more likely to have high performance hardware (giving them an unnecessary advantage in their results).
• No attempt at platform comparison is given (for example, Safari Window vs. Firefox Window and Safari Mac vs. Firefox Mac). Having the results lump together provides an inaccurate view of actual browser performance.

There's one message that should be taken away from this particular case: Don't trust random-user-generated browser performance data. Until you neutralize for outstanding factors like platform, system load, and even hardware it becomes incredibly hard to get meaningful data that is relevant to most users - or even remotely useful to browser vendors.

Tags: performance, browsers

14 Comments on 'The Performance Paradox'

JavaScript Library Loading Speed

February 5th, 2008

There was an interesting piece of JavaScript performance analysis done recently, by the PBWiki team. They wanted to understand a few things about how quickly JavaScript libraries loaded (obviously, their loading speed grossly effecting the total loading speed of a page). They set up a system to gather input from random browsers, aggregating the results into a final breakdown. There's a lot that application, and browser, developers can learn from the results - the total information available is actually quite profound:

JavaScript Packaging Techniques

When distributing a piece of JavaScript code it's traditional to think that the smallest (byte-size) code will download and load the fastest. This is not true - and is a fascinating result of this survey. Looking at the speed of loading jQuery in three forms: normal, minified (using Yahoo Min), and packed (using Packer). By order of file size, packed is the smallest, then minifed, then normal. However, the packed version has an overhead: It must be uncompressed, on the client-side, using a JavaScript decompression algorithm. This unpacking has a tangible cost in load time. This means, in the end, that using a minifed version of the code is much faster than the packed one - even though its file size is quite larger.

Packaging Comparison (loading jquery, all variants)

Next time you pick a compression technique, remember this formula:

Total_Speed = Time_to_Download + Time_to_Evaluate

JavaScript Library Performance

The next nugget of information, that we can unearth, is the total performance of JavaScript libraries, when loading within a page (this includes their transfer time and their evaluation time). Thus, a library that is both smaller and simpler will load faster. Looking at the results you can see a, comparatively, large lead for jQuery (200-400ms - a perceptible difference in speed).

Average Time to Load Toolkit (non cached, gzipped, minified)

Now, some might argue that testing the speed of un-cached pages would be unfair, however according to Yahoo's research on caching, approximately 50% of users will never have the opportunity to have the page contents be cached. Thus, making sure that you page loads quickly both on initial load, and subsequent loads, should be of the utmost importance.

Average Time to Load Toolkit (cached, gzipped, minified)

Once you examine cached speeds the difference becomes much less noticeable (10-30ms difference - with the exception of Prototype/Scriptaculous). Since these results are completely cached we can gauge, roughly, the overhead that's provided by file transfer, in comparison to evaluation speed).

If nothing else, I think this type of analysis warrants further examination. Using user-generated input, against live datasets, to create real-world performance metrics is quite lucrative to everyone involved - to users, framework developers, and browser vendors.

» More information on the web browser performance shown below.

Web Browser Performance

Tags: performance, javascript

34 Comments on 'JavaScript Library Loading Speed'