August 10th, 2007
Late last night, a commit to the Mozilla trunk caused the tree to "go orange" (signaling that, while the build was successful, a test had failed). Upon inspection it was determined that one of the recently-integrated Prototype tests was failing.
This is particularly interesting because this is the first time that one of the foreign JavaScript library test suites has caught a possible regression. While the true nature of last night's error is still being determined, I'm just pleased that it was able to do something.
However, during the course of that event, some outstanding issues, with how the test suites work, were uncovered. For instance, we determined that we need to be able to answer the following questions easily, and quickly, when an error comes up: Which test failed? and What does the test actually test?. Looking at these issues in depth:
Which test failed? In the case of the Prototype suite, this was actually quite challenging, if not impossible, without changing how the testrunner worked. Presently, the suite will print out any tests that fail, with their error message - but not print out any of the passed tests (giving you no context to know which particular test failed).
To alleviate this problem, I wrote up a quick patch for our copy of the test runner to add line numbers to the end of all the failure messages (giving the developers a rough way to figure out which test was causing problems). And then, today, I upstreamed the patch to Prototype, and after a quick chat with the team, it was committed right away. (This is my first patch to Prototype - it makes me quite happy to be able to contribute a little bit back to the great project.)
What does the test actually test? This is a much more challenging question. Frequently the developers who are reading the test case aren't familiar with the code base (the exception probably being Rob Sayre and the MochiKit tests, since he's a committer). So while there's not much that can be done in this area, at the very least library authors should aspire to write simple, granular, test cases that can be easily digest and analyzed. Although, that's typically a concern for most test suites as it is.
The next step is to patch up the Scriptaculous test suite (since it's using the same testrunner as Prototype) and then do a survey of the other suites to see if their reporting is up-to-par.
Tags: prototype, javascript, mozilla, test
2 Comments on 'First JS Test Suite Bug Caught'
March 19th, 2007
In the first follow-up to my previous post, on Future-Proofing JavaScript Libraries, I bring you instructions on testing Prototype against the latest alpha builds of Firefox 3!
(Soon, this will be integrated straight into the Mozilla test system, but I needed a good test system to start with, and Prototype is a good candidate to begin with; the Prototype team has been incredibly helpful. Thanks Tobie and Andrew!)
One aspect of the Mozilla test system (the one relevant to testing JavaScript libraries, and similar code) is based on the Mochikit Test Suite, with some heavy modifications.
How it works: All test files, from all projects, are grouped together into a single directory and served up by a temporary web server. A copy of Firefox is spawned, pointing to the local web server. Whenever a test file is run, the results are posted back to the server, which is then able to communicate the status of the tests to the rest of the Mozilla test process.
With that in mind, let's set up Prototype to hook into that process.
Step 1) Check out Mozilla from CVS
Typical checkout:
cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk
cd mozilla
make -f client.mk checkout MOZ_CO_PROJECT=browser
Note: Do everything but build the browser - we'll be doing that in a later step.
Step 2) Check out Prototype
To start with, we're going to be checking out the bleeding-edge copy of Prototype from SVN.
Make sure that you're in the root Mozilla directory (typically called "mozilla" - you should see a "browser" folder in here, along with many others).
Run the following:
mkdir browser/prototype
cd browser/prototype
svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/trunk .
These instructions also work for any particular version of Prototype (1.5.0 and newer). So, for example, you could checkout Prototype 1.5.1_rc1 by following the above, and changing the svn statement to:
svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/tags/rel_1-5-1_rc1 .
The Prototype code doesn't have to go in "browser" (in fact, the final location of the JavaScript libraries still has yet to be determined), but for now, this should be a suitable location to test from.
Step 3) Download the Test Suite Code
Borrowing some code from Robert Sayre's AJAX Framework Testing instructions, and adding in some Ruby/Rake goodness, we can now combine the Prototype Test Suite with the Mozilla build process.
Download the testing code here:
http://ejohn.org/files/proto-mozilla-0.1.zip
Make sure that you have ruby, ruby gems, rake, patch, and zip installed on your system. Once you've downloaded the file, and put it in the "browser/prototype" directory, complete the following:
Execute the following to add the testing code:
unzip proto-mozilla-0.1.zip
patch -u < Rakefile.patch
rake moztest
This code package provides the following:
- Makefile.in The Makefile for the Prototype test suite. The Mozilla build system recognizes this file and realizes that it needs to attach into the build system and, specifically, run using the central Mochitest test suite that powers all browser-centric tests.
- Rakefile.patch This patch adds three important new methods to the Prototype Rakefile (a build file for Ruby applications). moztest, which cleans up, builds Prototype, then builds the Mozilla-centric test suite code. moztest_clean, which cleans up any Mozilla test suite code that was generated. moztest_zip, which builds a zip file of the Mozilla test suite code.
- test_Prototype.html, this is the central test suite file, through which all the other Prototype test files are run. Since the Prototype test suite doesn't know how to hook into the Mochitest core, this file helps to manage, and automate, that process.
- mozilla/unittest-append.js, this is an addition to Prototype's test/lib/unittest.js code which overrides some of Prototype's test suite functionality, hooking it in to the Mozilla test suite system instead.
- mozilla/AJAX_setup.js, mozilla/SimpleTest.js, mozilla/Mochikit_packed.js are the three core JavaScript files that run all the in-browser tests and report the results back to the managing web server (which is then reported by to the main Mozilla test system).
Step 4) Hook the Test Suite In
Now that we have all the appropriate files set up, the last step is to attach the Prototype test suite into the Mozilla build process. This can be done by editing "browser/Makefile.in", making the following change:
Change this line:
DIRS = base components locales themes app
to:
DIRS = base components locales themes app prototype
Step 5) Build Mozilla
Finally, we can build a copy of Firefox. Make sure that you're in the root Mozilla directory, then run the following commands:
make -f client.mk build
perl firefox-objdir/_tests/testing/mochitest/runtests.pl
This will build a copy of Firefox, then run the Mochitest test suite (which the Prototype test suite has hooked in to). (Your copy of Mochitest may not be in firefox-objdir - this depends on how you configured your Mozilla system. If you do see it right there, be sure to do some poking around - it's bound to be somewhere close by.)
The Results
Once you've run the test suite you should see a browser pop up. You can now visit the follow URL to see the Prototype test suite run, automated, in the latest Firefox 3 nightly:
http://localhost:8888/tests/browser/prototype/test_Prototype.html
The results will, hopefully, look something like this:
If you have any questions, be sure to let me know, and I'll do my best to answer them.
Tags: firefox, prototype, javascript, libraries, mozilla, testing, test
9 Comments on 'Testing Prototype with Firefox 3'
