I don't think there's a single JavaScript developer who isn't excited about the new Selectors API specification (and the upcoming implementations). I've been following the progress of the specification (and implementations) and have been asked to provide some feedback to the Web API working group. What follows is an email that I sent to the public-webapi mailing list.
I just wanted to quickly pull together some of my thoughts concerning querySelectorAll. I've been asked by a number of people to provide my feedback here. Please forgive me if I've missed some previous discussions on the subject matter.
There's three major points that I wanted to discuss:
DOMElement.querySelectorAll returning incorrect elements
This is the most critical issue. As it stands DOM Element-rooted queries are borderline useless to libraries - and users. Their default behavior is unexpected and confusing. Demonstrated with an example, using Dojo:
<div><pid="foo"><span></span></p></div> <scriptsrc="http://o.aolcdn.com/dojo/1.1.0/dojo/dojo.xd.js"></script> <script>
var foo = document.getElementById("foo");
// should return nothing
alert( dojo.query('div span', foo).length );
// will return the SPAN (booo!)
alert( foo.querySelectorAll('div span').length ); </script>
This is due to the fact that element-rooted queries are handled by "finding all the elements that match the given selector -- rooted in the document -- then filtering by the ones that have the specified element as an ancestor." This is completely unacceptable. Not only is it not intuitive (finding elements that don't match the correct expression) but it goes against what every single JavaScript library provides. If there behavior were persisted then there would be serious ramifications for the usefulness of this function in the wild.
I asked some of the other library developers what their thoughts were and they agreed with my conclusion.
Andrew Dupont (creator of Prototype's selector engine):
My issue with this is that it violates principle of least surprise and bears no resemblance to the APIs in the wild.
Alex Russell (creator of Dojo's selector engine):
This is a spec bug.
Combinator-rooted Queries
I read about some prior discussion concerning this (especially in relation to DOMElement.querySelectorAll-style queries). This is an important part of most libraries, as it stands. Maciej's proposed solution of using :root to allow for front-leading combinators is perfectly acceptable to me (where :root is made equivalent to the element, not the document element).
// jQuery
$("#foo").find("> span");
// DOM
document.getElementById("foo").querySelectorAll(":root > span")
This is something that a library can easily detect and inject.
Error-handling
I'm perfectly fine with the proposed try/catch solution however there must be a way of easily determining what the invalid portion of the selector was. Currently the following occurs in Safari:
If there were extra properties to point to what the inappropriate selector was, that'd be fundamentally important. Probably the best solution (for both implementors and JavaScript library authors) would be to simply provide a character index, working something like the following:
There will be some form of a performance hit here but I think, if done correctly, it'll be negligible (especially in comparison to the benefits that are being received).
I hope these proposed changes work well for the members of this group as they will greatly benefit general web developers - and especially library developers.
Today, at SXSW, I hosted a panel discussion on the "Secrets of JavaScript Libraries." We set out to discuss the behind-the-scenes nitty-gritty code that powers the universal features of the most popular JavaScript libraries.
I think the presentation went quite well. I think we could've, easily, done a 2 hour talk with all the content that we had (we had to chop off, at least, two sections at the end in order to take questions). Here are the slides:
Some of the slides will be a little bit tricky considering that they rely on someone talking about them - I'll be sure to do an audio-to-slide synchronization once the presentation podcast is released.
I want to thank all the panelists for participating: Sam Stephenson, Alex Russell, Thomas Fuchs, and Andrew Dupont. Together it helped make for a stand-out panel. Here's hoping that SXSW decides to do more technical panels next year - if this panel was any indicator developers are hungry for more code talks.
A fascinating thing has happened in the world of JavaScript DOM traversal: Over the course of a couple months in 2007 three of the major JavaScript libraries (Prototype, Dojo, and Mootools) all switched their CSS selector engines to using the browser's native XPath functionality, rather than doing traditional DOM traversal. What's interesting about this is that the burden of functionality and performance has, literally, flipped overnight on to the browser's XPath engine, from its pure DOM implementation.
There's some really interesting things about this switch:
Native XPath is blazing fast. For a majority of CSS selectors it completely trumps using native DOM methods (like getElementsByTagName, for example). Sometimes it pays to special-case your code for selectors like #id, but overwhelmingly XPath is the direction in which JavaScript libraries are heading.
Since a large percentage of JavaScript users use JavaScript libraries (and, thus, use the behind-the-scenes XPath, as well) this means that browsers are now spending significantly more time processing XPath queries than they ever were before. This means that the performance field is now, effectively, split between two areas: Traditional DOM querying and XPath.
No one is analyzing the performance of browser XPath queries. Or, if they are, it's certainly not public. I'm working on some new XPath performance tests, in order to bring them some more visibility, and hope to have them released this week.
XPath, while incredibly useful, is a black box. The developer has no control over how fast the results come back - or if they are even correct. Contrast this with traditional DOM scripting (where you can fine-tune your queries to perfection). Browsers will always be bound to have some bugs in their implementations. For example, Safari 3 isn't capable of doing "-of-type" or ":empty" style CSS selectors, nor is any browser able to access the 'checked' property, or namespaced attributes (all noted in Prototype's implementation) which means that they have to fall back to a traditional DOM scripting model.
Internet Explorer is a dead-end. Since most users want a CSS selector implementation that will work against HTML documents - and IE is unable to provide one - all CSS Selector implementations must provide two (2) side-by-side selector engines in order to handle these cases (not to mention the aforementioned cases where browsers provide unexpected behavior).
A couple things to take away from all of this:
XPath (and new methods like querySelector) are the way of the future for a lot of JavaScript libraries - and the next frontier for browser optimization.
These implementations are black boxes that are unable to be modified by the developer (leaving them vulnerable to browser bugs).
A dual DOM-only CSS selector engine must be provided well into the foreseeable future, by libraries, in order to account for browser mis-implementations.
I should, also, probably answer the inevitable question: "Why doesn't jQuery have an XPath CSS Selector implementation?" For now, my answer is: I don't want two selector implementations - it makes the code base significantly harder to maintain, increases the number of possible cross-browser bugs, and drastically increases the filesize of the resulting download. That being said, I'm strongly evaluating XPath for some troublesome selectors that could, potentially, provide some big performance wins to the end user. In the meantime, we've focused on optimizing the actual selectors that most people use (which are poorly represented in speed tests like SlickSpeed) but we hope to rectify in the future.
Update: Olmo and I have been in contact recently, off public-facing sites, but here's a public response that he made, and here's a private one that I gave to him:
Thank you for doing this. I want to, personally, apologize for
everything that I did during the past couple days. It was out of line
- both for myself and in general. I should've done a much-more even
tempered response, I got caught up, it turned into a flame - ugh,
sorry again.
I fully accept your apology and will be adding your comment and
modifying my post to give people more details and apologizing
publicly.
In fact, not only do we steal code, but we're incompetent at it too! Durrrr, let's swap some variables and pray it works! How does animation work? Who knows! It's one of the mysteries of the Intertubes! I mean, it's obvious that we never could've completely rewritten a popular animation library back in 2005 because we just "tweaked some variables" and prayed everyday that it would work.
jQuery hates developers, love companies
We love that sweet, sweet, money that corporations send our way. Forget developers. As far as we're concerned, they can stay in the ditch that we threw them in to drive on the way to moneytown.
File size is the only thing jQuery considers when adding a feature
It doesn't matter how useful it is, or what performance benefits we get. It's file size, over everything else. Carefully balancing facts and weighing advantages is ludicrous and highly overrated.
File size is irrelevant
Making code modular makes file size irrelevant, since you can just pick and choose what you want to use. Nevermind that the sum of the parts is larger, all that matters is that you write code for developers. Developers who don't care about file size.
jQuery code isn't modular
I mean, I think the code base would have to be broken into, at MINIMUM, over 250 modules before we even talked.
Ternary Operators!
Seriously, need I say more? Obviously the mark of incompetence.
My head, literally, just exploded. I'm dead, lying on the floor. Can someone please put "he had a lot of problems because of the closures" on my tombstone? Thanks! (I wish I was making this up, 16:20, check it out.)
Olmo: You seem immensely proud of your ignorance concerning other libraries but love to speak on authority concerning them. Maybe you should take some time to sit down and actually try to use and work with other libraries, because while you were out bashing and slandering us we were writing rock solid code, sharing, collaborating, and discussing the future of the web.
Let us know when you're ready to see the Open Web succeed, we'll be here, getting things done.
This is a presentation that I recently gave at Yahoo for a number of their developers. It was on the importance of JavaScript Libraries, and how their introduction and use changes how JavaScript development works.
Specifically, I discuss some of what I've learned from developing, and working with the users of, jQuery and developing the new FUEL library for Firefox 3.
In the first follow-up to my previous post, on Future-Proofing JavaScript Libraries, I bring you instructions on testing Prototype against the latest alpha builds of Firefox 3!
(Soon, this will be integrated straight into the Mozilla test system, but I needed a good test system to start with, and Prototype is a good candidate to begin with; the Prototype team has been incredibly helpful. Thanks Tobie and Andrew!)
One aspect of the Mozilla test system (the one relevant to testing JavaScript libraries, and similar code) is based on the Mochikit Test Suite, with some heavy modifications.
How it works: All test files, from all projects, are grouped together into a single directory and served up by a temporary web server. A copy of Firefox is spawned, pointing to the local web server. Whenever a test file is run, the results are posted back to the server, which is then able to communicate the status of the tests to the rest of the Mozilla test process.
With that in mind, let's set up Prototype to hook into that process.
cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk
cd mozilla
make -f client.mk checkout MOZ_CO_PROJECT=browser
Note: Do everything but build the browser - we'll be doing that in a later step.
Step 2) Check out Prototype
To start with, we're going to be checking out the bleeding-edge copy of Prototype from SVN.
Make sure that you're in the root Mozilla directory (typically called "mozilla" - you should see a "browser" folder in here, along with many others).
Run the following:
mkdir browser/prototype
cd browser/prototype
svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/trunk .
These instructions also work for any particular version of Prototype (1.5.0 and newer). So, for example, you could checkout Prototype 1.5.1_rc1 by following the above, and changing the svn statement to:
svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/tags/rel_1-5-1_rc1 .
The Prototype code doesn't have to go in "browser" (in fact, the final location of the JavaScript libraries still has yet to be determined), but for now, this should be a suitable location to test from.
Step 3) Download the Test Suite Code
Borrowing some code from Robert Sayre's AJAX Framework Testing instructions, and adding in some Ruby/Rake goodness, we can now combine the Prototype Test Suite with the Mozilla build process.
Make sure that you have ruby, ruby gems, rake, patch, and zip installed on your system. Once you've downloaded the file, and put it in the "browser/prototype" directory, complete the following:
Makefile.in The Makefile for the Prototype test suite. The Mozilla build system recognizes this file and realizes that it needs to attach into the build system and, specifically, run using the central Mochitest test suite that powers all browser-centric tests.
Rakefile.patch This patch adds three important new methods to the Prototype Rakefile (a build file for Ruby applications). moztest, which cleans up, builds Prototype, then builds the Mozilla-centric test suite code. moztest_clean, which cleans up any Mozilla test suite code that was generated. moztest_zip, which builds a zip file of the Mozilla test suite code.
test_Prototype.html, this is the central test suite file, through which all the other Prototype test files are run. Since the Prototype test suite doesn't know how to hook into the Mochitest core, this file helps to manage, and automate, that process.
mozilla/unittest-append.js, this is an addition to Prototype's test/lib/unittest.js code which overrides some of Prototype's test suite functionality, hooking it in to the Mozilla test suite system instead.
mozilla/AJAX_setup.js, mozilla/SimpleTest.js, mozilla/Mochikit_packed.js are the three core JavaScript files that run all the in-browser tests and report the results back to the managing web server (which is then reported by to the main Mozilla test system).
Step 4) Hook the Test Suite In
Now that we have all the appropriate files set up, the last step is to attach the Prototype test suite into the Mozilla build process. This can be done by editing "browser/Makefile.in", making the following change:
Change this line:
DIRS = base components locales themes app
to:
DIRS = base components locales themes app prototype
Step 5) Build Mozilla
Finally, we can build a copy of Firefox. Make sure that you're in the root Mozilla directory, then run the following commands:
make -f client.mk build
perl firefox-objdir/_tests/testing/mochitest/runtests.pl
This will build a copy of Firefox, then run the Mochitest test suite (which the Prototype test suite has hooked in to). (Your copy of Mochitest may not be in firefox-objdir - this depends on how you configured your Mozilla system. If you do see it right there, be sure to do some poking around - it's bound to be somewhere close by.)
We're starting to undertake a few new initiatives, here at Mozilla that attempt to find ways to benefit web developers - and by extension, JavaScript Libraries. I think this is an excellent movement, so I'm doing everything that I can to support it and push it forward. With that in mind, here's an introduction to one of the first initiatives that we're undertaking.
JavaScript libraries can be fickle beasts. Generally speaking, they attempt to pave over browser bugs and interfaces, providing a consistent base-layer that users can build upon. This is a challenging task, as bugs can frequently be nonsensical - and even result in browser crashes.
Additionally, object detection has the ability to completely fail. Safari currently has a super-nasty bugs related to object detection. For example, assuming that you have a variable and you need to determine if it contains a single DOM Element, or a DOM NodeList. One one would think that it would be as simple as:
if( elem.nodeName){ // it's an element }else{ // it's a nodelist }
However, in the current version of Safari, this causes the browser to completely crash, for reasons unknown. (However, I'm fairly certain that this has already been fixed in the nightlies.)
Side Story
I was in the group of JavaScript developers who provided feature/bug fix recommendations to Microsoft for their next version of IE. A huge issue that we were faced with was that we were knowingly asking Microsoft to both break their browser and alienate their existing userbase, in the name of standards.
For example, if Microsoft adds proper DOM Events (addEventListener, etc.) - should they then remove their IE-specific event model (attachEvent, etc.)? Assuming that they do decide to remove the deprecated interfaces, this will have serious effects upon JavaScript developers and libraries (although, in the case of the DOM Event model, object detection is a viable solution and is, therefore, completely future-compatible.)
Additionally, in Internet Explorer, doing object detection checks can, sometimes, cause actual function executions to occur. For example:
if( elem.getAttribute){ // will die in Internet Explorer }
That line will cause problems as Internet Explorer attempts to execute the getAttribute function with no arguments (which is invalid). (The obvious solution is to use "typeof elem.getAttribute == 'undefined'" instead.)
The point of these examples isn't to rag on Safari or Internet Explorer in particular, but to point out that rendering-engine checks can end up becoming very convoluted - and thusly, more vulnerable to future changes within a browser. This is a very important point. A browser deciding to fix bugs can cause more problems for a JavaScript developer than simply adding new features. For every bugfix, there are huge ramifications. Developers expect interfaces to work and behave in very-specific ways.
The recent Internet Explorer 7 release can be seen as a case study in this. They fixed numerous CSS-rendering errors in their engine, which caused an untold number of web sites to render incorrectly. By fixing bugs, shockwaves were sent throughout the entire web development industry.
All of this is just a long-winded way of saying: Browsers will introduce bugs. Either these bugs are going to be legitimate mistakes or unavoidable bug fixes - either way, they'll be regressions that JavaScript developers will have to deal with.
At Mozilla, we've looked at this issue and Mike Shaver came up with an excellent solution: Simply include the test suites of popular JavaScript libraries inside the Mozilla code base.
Doing this will provide, at least two, huge benefits:
Library developers will be able to know about unavoidable regressions and adjust their code before the release even occurs.
Mozilla developers will be able to have a massively-expanded test suite that will help to catch any unintended bugs. In addition to making sure that less, general, bugs will be introduced into the system, library authors and users will be content knowing that their code is already working in the next version of Firefox, without having to do any extra work.
What progress has already been made? Mochikit's test suite (Mochitest) is already a part of Mozilla's official test suite (it's used to test UI-specific features). I've already touched base with Alex Russel, of Dojo, and I'll be working to integrate their test suite once Dojo 0.9 hits. Perhaps unsurprisingly, I'll be working to integrate jQuery's test suite into the core, too. Additionally, I'm also starting to contact other popular library developers attempting to get, at least, a static copy of their test suite in place.
Note: This initiative isn't limited to straight JavaScript libraries. If you have a large, testable, JavaScript-heavy, Open Source project let me know and I'll be sure to start moving things forward. For example, some form of testing for Zimbra will probably come into play.
In all, I think this is a fantastic step forward - and a step that really shows the immediate benefits of having an open development process centered around browser implementations. I hope to see other browser manufacturers catch on too, as having universally-available pre-release library testing will simply help too many users to count.
