Mozilla Build and Test Integration

April 22nd, 2008

Lately I've been investigating how Mozilla's build and test system works (there's a number of pieces that tend to have a pretty tight integration and I wanted to learn more). I asked developer Ben Hearsum and he kindly obliged. I've included the questions and information here in the hopes that others will be able to learn from it, as well.

There's two critical components to Mozilla's build and test infrastructure: Buildbot and Tinderbox - I was wondering if you could tell me about their relationship and integration.

Ben: I'd like to break this down a little bit more. Tinderbox consists of
two parts: Client and Server. The server is essentially just the Waterfall display. It sits on a server somewhere and reacts to incoming e-mail. The client is a set of scripts that knows how to do various interesting things (building, packaging, generating updates, etc) with Mozilla products.

Historically, builds are done on an infinite loop by tinderbox client, reporting back to the server at the start and end of each. It's a completely stateless system; the client sends out a specifically formatted e-mail, and the server acts on it. Because of the simplicity of communicating with the tinderbox server we can use it as a display, with Buildbot driving the builds.

That is only direct communication between the two. In some cases Buildbot does some post-processing of logs to get Tinderbox to display things directly on the Waterfall (f.e., unit test pass/fail numbers).

At this point, Buildbot is responsible for driving almost all of the new build, unit test, and talos infrastructure we bring up. Right now, we'll be sticking with Tinderbox as the main developer frontend. In the future we may want to present developers with the Buildbot Waterfall instead -- but we've got some feature parity to address before that can happen.

How are unit tests integrated into the Buildbot and Tinderbox systems? How are different types of tests handled?

Ben: With the exception of reporting back to the Tinderbox server all of our unit test infrastructure is 100% Buildbot. (I'm not even sure tinderbox client can run unit tests, actually.) We've got custom Buildbot steps that run our various tests and parse the output. These classes deal with the different types of tests - how they run, how to parse the output they generate, etc.

Currently, unit test output is only available to the outside world via Tinderbox. It shows a quick pass/fail/skip for each test. Internally, we mostly watch the Buildbot Waterfall. For the curious, here's what that looks like:

What if a developer wants to test out the implications of a patch before landing it and affecting all other developers?

Ben: The Try Server is exactly what you want here. You can submit a patch to it (or a set of HG repositories, if you're testing Mozilla2 code) - and have it compile, package, and upload a build for you. Recently, we added two new features: rudimentary Talos and a win32 symbol server. For instructions on how to use it swing by the wiki page.

Buildbot appears to be used all throughout the Mozilla infrastructure - is its use applicable to other projects?

Ben: It's true, we use Buildbot a lot here. Off the top of my head we use
it for: Release Automation, Try Server, unit tests, Talos, misc. test infrastructure, l10n, and probably some things I'm unaware of. One of the great things about Buildbot is that it's an active project with a healthy community. This is one of the reasons we want to move to it. It's used by many different projects, including: lists Python, Twisted, KDE, WebKit, Subversion, OpenOffice, Gnome - to name a few.

Buildbot is built in a very extensible and customizable way. It's relatively easy, even for a project like Mozilla (whose build system/process is quite odd) to start driving infrastructure with Buildbot.

Monitoring Tinderbox frequently serves as the heart of Mozilla development. What does this tool provide that makes it so important to developers?

Ben: Because developers still use the Tinderbox server as their source of information there isn't currently much direct benefit to them - other than us Release Engineering folk having more time to do interesting things.

In the future we may want to give developers direct access to the Buildbot waterfall. This is where we can make developers' lives easier. Ideally (depending on what features get implemented) we'd want to provide the following:

• The ability to trigger a clobber build from the web interface (no more CLOBBER files).
• Lots of different ways to display data ("give me the latest build from each builder", "give me a list of builds from builder X", etc).
• Build status via IM/RSS/E-Mail.

Specific to the try server

• Ability to stop a running build (useful when one platform fails quickly - you can save time by canceling the others).
• Ability to submit a patch directly from the Buildbot Waterfall.

--

I want to thank Ben for taking the time to answer these questions for me. I've been really impressed with Mozilla's continuous integration set up - especially the use of Buildbot. I suspect that I'd like to have something similar set up for jQuery (doing automated testing, etc.) but a lot of work is still required to make that happen. At least it doesn't look that hard to get started, which is quite important.

Tags: mozilla, build, testing

2 Comments on 'Mozilla Build and Test Integration'

Dromaeo: JavaScript Performance Testing

April 11th, 2008

Dromaeo is the name that I've given to the JavaScript performance test suite that I've been working on over the past couple months.

I was hoping to hold off on this release for another week or two, while I finished up some final details, but since it's been discovered, and about to hit the Digg front page, there isn't a whole lot that I can do to stop it.

There's a ton of details concerning how it works, and how to use it, on the Dromaeo wiki page. I won't go through too much of it here, but it should clarify most question there.

Probably the most pressing question that'll be encountered (outside of what is answered on the wiki page) is "What is the relation of Dromaeo to SunSpider?" (SunSpider being the WebKit team's JavaScript testing suite).

Right now I'm working very closely with all the browser vendors to make sure that we have a common-ground test suite that is both highly usable and statistically sound (not to mention providing results that are universally interesting). There are a number of outstanding concerns that've been raised by users of the suite, along with a number of concerns that've already been rectified - again, all of this is clarified on the Dromaeo wiki page. It's of the utmost concern that this suite be as applicable as possible. It's very likely that the core suite will be moving to a common working ground where all browser vendors can work on it.

I especially want to thank Allan Branch of LessEverything who provided the awesome design for the site. It's like he tapped into my brain and produced exactly what I wanted - without knowing even it. I highly recommend them, if you have design work that needs to be done.

Tags: testing, performance, javascript, browsers, mozilla

23 Comments on 'Dromaeo: JavaScript Performance Testing'

Acid 3 Tackles ECMAScript

January 10th, 2008

Update: The Acid3 test is now final, I've updated the blog post to reflect this.

The new news on the block is that the upcoming Acid 3 test is in the oven, starting to get baked. Traditionally, the Acid test has served as a way to get browser vendors in line by testing them on really-annoying edge cases. This can, sometimes, get people tied up in knots but it actually serves as a devious way of getting people to meet a large part of a spec.

For example, in order for a browser to have some weird padding/margin test case solved - in CSS - they must also have a working box model. So while an Acid test may not, explicitly, test for a working box model, it will be done implicitly (by testing edge cases that result from it).

With that in mind, it's time to take a look at Acid 3 which primarily focuses on technology that I find to be interesting: ECMAScript and the DOM. Let's dig in and see what exactly is being tested - specifically, relating to ECMAScript.

• Array Elisions - Making sure that stuff like [,,] has a length of 2 and [0,,1] has a length of 3.
• Array Methods - Doing an unshift with multiple arguments .unshift(0, 1, 2), joining with an undefined argument .join(undefined).
• Number Conversion - Banging against .toFixed(), .toExponential(), and .toPrecision() - especially with decimals and negative numbers.
• String Operations - Negative indicies in substr .substr(-7, 3), character access by index "foo"[1] (part of the ECMAScript 4 spec).
• Date - Making sure that certain method calls result in NaN results (like d.setMilliseconds(), with no arguments) and also enforcing +1900 year offsets.
• Unicode in Identifiers - You can't use escaped Unicode in identifiers, for example: eval("test.i\\u002b= 1;"); (that should throw an exception).
• Regular Expressions - /[]/ matches an empty set, /[])]/ should throw an exception, backreferences to non-existent captures, and negative lookaheads /(?!test)(test).exec("test test").
• Enumeration - Make sure that object properties are enumerated in the correct order, make sure that you're able to enumerate properties of certain names (toString, hasOwnProperty, etc.).
• Function Constructors - The user should be able to set custom constructors on the .constructor property, .constructor should not be enumerable, and .prototype.constructor should be deletable.
• Function Expressions - (function test(){ ... })(); You should be able to call the function by name, within the function itself, you can't directly overwrite the function name (only with a function-scoped variable), and 'test' isn't leaked into the parent scope.
• Exception Scope - Variables within the catch(){} should interact with the catch arguments primarily, followed by variables in an outer scope.
• Assignment Expressions - s = a.length = "123"; - a.length has a return value of 123 (the number) which is assigned to 's', rather than the correct result of the string "123".
• Encoding - encodeURI() and encodeURIComponent() must gracefully handle null bytes.

All-in-all it's a comprehensive smattering of weird ECMAScript edge cases - you're bound to find at least one that fails in your favorite browser-of-choice. I'm sure we'll see many more test cases coming in, in the upcoming days in weeks.

I'm looking forward to seeing the final results - and the competitive heat that's been applied to CSS-spec implementors being applied to ECMAScript implementors.

For kicks, here's the current results in a bunch of major browsers (including the correct reference rendering).

These are the preliminary results of the UNCOMPLETED Acid 3 test in UNCOMPLETED versions of major browsers - take with a grain of salt. Go here to view the final version of this test.

Reference Rendering:

Firefox 2:

Firefox 3b2:

Safari 3:

WebKit Nightly:

Opera 9.5b1:

Internet Explorer 7: (thanks chunghe!)

Tags: testing, ecmascript, bugs

55 Comments on 'Acid 3 Tackles ECMAScript'

Testing Prototype with Firefox 3

March 19th, 2007

In the first follow-up to my previous post, on Future-Proofing JavaScript Libraries, I bring you instructions on testing Prototype against the latest alpha builds of Firefox 3!

(Soon, this will be integrated straight into the Mozilla test system, but I needed a good test system to start with, and Prototype is a good candidate to begin with; the Prototype team has been incredibly helpful. Thanks Tobie and Andrew!)

One aspect of the Mozilla test system (the one relevant to testing JavaScript libraries, and similar code) is based on the Mochikit Test Suite, with some heavy modifications.

How it works: All test files, from all projects, are grouped together into a single directory and served up by a temporary web server. A copy of Firefox is spawned, pointing to the local web server. Whenever a test file is run, the results are posted back to the server, which is then able to communicate the status of the tests to the rest of the Mozilla test process.

With that in mind, let's set up Prototype to hook into that process.

Step 1) Check out Mozilla from CVS

• Instructions for checking out the entire Mozilla source tree can be found here:
  http://developer.mozilla.org/en/docs/Mozilla_Source_Code_%28CVS%29
• Full build instructions for your system can be found here:
  http://developer.mozilla.org/en/docs/Build_Documentation

Typical checkout:

cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co mozilla/client.mk
cd mozilla
make -f client.mk checkout MOZ_CO_PROJECT=browser

Note: Do everything but build the browser - we'll be doing that in a later step.

Step 2) Check out Prototype

To start with, we're going to be checking out the bleeding-edge copy of Prototype from SVN.

Make sure that you're in the root Mozilla directory (typically called "mozilla" - you should see a "browser" folder in here, along with many others).

Run the following:

mkdir browser/prototype
cd browser/prototype
svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/trunk .

These instructions also work for any particular version of Prototype (1.5.0 and newer). So, for example, you could checkout Prototype 1.5.1_rc1 by following the above, and changing the svn statement to:

svn co http://svn.rubyonrails.org/rails/spinoffs/prototype/tags/rel_1-5-1_rc1 .

The Prototype code doesn't have to go in "browser" (in fact, the final location of the JavaScript libraries still has yet to be determined), but for now, this should be a suitable location to test from.

Step 3) Download the Test Suite Code

Borrowing some code from Robert Sayre's AJAX Framework Testing instructions, and adding in some Ruby/Rake goodness, we can now combine the Prototype Test Suite with the Mozilla build process.

Download the testing code here:
http://ejohn.org/files/proto-mozilla-0.1.zip

Make sure that you have ruby, ruby gems, rake, patch, and zip installed on your system. Once you've downloaded the file, and put it in the "browser/prototype" directory, complete the following:

Execute the following to add the testing code:

unzip proto-mozilla-0.1.zip
patch -u < Rakefile.patch
rake moztest

This code package provides the following:

• Makefile.in The Makefile for the Prototype test suite. The Mozilla build system recognizes this file and realizes that it needs to attach into the build system and, specifically, run using the central Mochitest test suite that powers all browser-centric tests.
• Rakefile.patch This patch adds three important new methods to the Prototype Rakefile (a build file for Ruby applications). moztest, which cleans up, builds Prototype, then builds the Mozilla-centric test suite code. moztest_clean, which cleans up any Mozilla test suite code that was generated. moztest_zip, which builds a zip file of the Mozilla test suite code.
• test_Prototype.html, this is the central test suite file, through which all the other Prototype test files are run. Since the Prototype test suite doesn't know how to hook into the Mochitest core, this file helps to manage, and automate, that process.
• mozilla/unittest-append.js, this is an addition to Prototype's test/lib/unittest.js code which overrides some of Prototype's test suite functionality, hooking it in to the Mozilla test suite system instead.
• mozilla/AJAX_setup.js, mozilla/SimpleTest.js, mozilla/Mochikit_packed.js are the three core JavaScript files that run all the in-browser tests and report the results back to the managing web server (which is then reported by to the main Mozilla test system).

Step 4) Hook the Test Suite In

Now that we have all the appropriate files set up, the last step is to attach the Prototype test suite into the Mozilla build process. This can be done by editing "browser/Makefile.in", making the following change:

Change this line:

DIRS = base components locales themes app

to:

DIRS = base components locales themes app prototype

Step 5) Build Mozilla

Finally, we can build a copy of Firefox. Make sure that you're in the root Mozilla directory, then run the following commands:

make -f client.mk build
perl firefox-objdir/_tests/testing/mochitest/runtests.pl

This will build a copy of Firefox, then run the Mochitest test suite (which the Prototype test suite has hooked in to). (Your copy of Mochitest may not be in firefox-objdir - this depends on how you configured your Mozilla system. If you do see it right there, be sure to do some poking around - it's bound to be somewhere close by.)

The Results

Once you've run the test suite you should see a browser pop up. You can now visit the follow URL to see the Prototype test suite run, automated, in the latest Firefox 3 nightly:
http://localhost:8888/tests/browser/prototype/test_Prototype.html

The results will, hopefully, look something like this:

If you have any questions, be sure to let me know, and I'll do my best to answer them.

Tags: firefox, prototype, javascript, libraries, mozilla, testing, test

9 Comments on 'Testing Prototype with Firefox 3'

Future-Proofing JavaScript Libraries

March 1st, 2007

We're starting to undertake a few new initiatives, here at Mozilla that attempt to find ways to benefit web developers - and by extension, JavaScript Libraries. I think this is an excellent movement, so I'm doing everything that I can to support it and push it forward. With that in mind, here's an introduction to one of the first initiatives that we're undertaking.

JavaScript libraries can be fickle beasts. Generally speaking, they attempt to pave over browser bugs and interfaces, providing a consistent base-layer that users can build upon. This is a challenging task, as bugs can frequently be nonsensical - and even result in browser crashes.

There are a number of techniques that can be used to know about, and work around, bugs or missing features - but generally speaking, object detection is the safest way to determine is specific feature is available, and usable. Unfortunately, in real-world JavaScript development, object detection can only get you so far. For example, there's no object that you can 'detect' to determine if browsers return inaccurate attribute values from getAttribute, if they execute inline script tags on DOM injection, or if they fail to return correct results from a getElementsByTagName query.

Additionally, object detection has the ability to completely fail. Safari currently has a super-nasty bugs related to object detection. For example, assuming that you have a variable and you need to determine if it contains a single DOM Element, or a DOM NodeList. One one would think that it would be as simple as:

However, in the current version of Safari, this causes the browser to completely crash, for reasons unknown. (However, I'm fairly certain that this has already been fixed in the nightlies.)

Additionally, in Internet Explorer, doing object detection checks can, sometimes, cause actual function executions to occur. For example:

That line will cause problems as Internet Explorer attempts to execute the getAttribute function with no arguments (which is invalid). (The obvious solution is to use "typeof elem.getAttribute == 'undefined'" instead.)

The point of these examples isn't to rag on Safari or Internet Explorer in particular, but to point out that rendering-engine checks can end up becoming very convoluted - and thusly, more vulnerable to future changes within a browser. This is a very important point. A browser deciding to fix bugs can cause more problems for a JavaScript developer than simply adding new features. For every bugfix, there are huge ramifications. Developers expect interfaces to work and behave in very-specific ways.

The recent Internet Explorer 7 release can be seen as a case study in this. They fixed numerous CSS-rendering errors in their engine, which caused an untold number of web sites to render incorrectly. By fixing bugs, shockwaves were sent throughout the entire web development industry.

All of this is just a long-winded way of saying: Browsers will introduce bugs. Either these bugs are going to be legitimate mistakes or unavoidable bug fixes - either way, they'll be regressions that JavaScript developers will have to deal with.

At Mozilla, we've looked at this issue and Mike Shaver came up with an excellent solution: Simply include the test suites of popular JavaScript libraries inside the Mozilla code base.

Doing this will provide, at least two, huge benefits:

1. Library developers will be able to know about unavoidable regressions and adjust their code before the release even occurs.
2. Mozilla developers will be able to have a massively-expanded test suite that will help to catch any unintended bugs. In addition to making sure that less, general, bugs will be introduced into the system, library authors and users will be content knowing that their code is already working in the next version of Firefox, without having to do any extra work.

What progress has already been made? Mochikit's test suite (Mochitest) is already a part of Mozilla's official test suite (it's used to test UI-specific features). I've already touched base with Alex Russel, of Dojo, and I'll be working to integrate their test suite once Dojo 0.9 hits. Perhaps unsurprisingly, I'll be working to integrate jQuery's test suite into the core, too. Additionally, I'm also starting to contact other popular library developers attempting to get, at least, a static copy of their test suite in place.

Note: This initiative isn't limited to straight JavaScript libraries. If you have a large, testable, JavaScript-heavy, Open Source project let me know and I'll be sure to start moving things forward. For example, some form of testing for Zimbra will probably come into play.

In all, I think this is a fantastic step forward - and a step that really shows the immediate benefits of having an open development process centered around browser implementations. I hope to see other browser manufacturers catch on too, as having universally-available pre-release library testing will simply help too many users to count.

Tags: programming, libraries, javascript, browsers, firefox, mozilla, testing

15 Comments on 'Future-Proofing JavaScript Libraries'