Test Swarm Alpha Open

TestSwarm, the project that I’ve been working on over the past 6 months, or so, is now open to the public. Mozilla has been very gracious, allowing me to work on this project exclusively. At the beginning of April I moved from my old position as a JavaScript Evangelist on the Mozilla Evangelism team to that of a JavaScript Tool Developer on the new Developer Tools team (whose other major project is Bespin).

For more information on Test Swarm I’ve written up a detailed explanation of what Test Swarm provides and where it fits into the landscape of JavaScript developer tools.

I’ve also recorded a screencast walkthrough of the TestSwarm site:

Test Swarm Walkthrough from John Resig on Vimeo.

Additionally, there are two previous posts that I’ve made on TestSwarm:

• JavaScript Testing Does Not Scale
• Presentation on TestSwarm at JSConf

There are some additional screenshots of Test Swarm in action on Flickr.

More Information:

• Test Swarm Source Code
• Test Swarm Bug Tracker
• Test Swarm Discussion Group
• Test Swarm Twitter Updates

If you’re interested in using Test Swarm I strongly recommend reading the project overview first. If you have any further questions please direct them towards the Test Swarm discussion Group.

TestSwarm ended up being a very challenging project to get to an alpha state (and probably will be even more challenging to get to a final release state). Dealing with cross-browser incompatibilities, cross-domain test suite execution, and asynchronous, distributed, client execution has been more than enough to make for a surprisingly difficult project. It’s mostly written in PHP and uses MySQL as a back end (allowing it to run in virtually any environment). Patches will absolutely be appreciated.

This project has been a long time coming now, the first inklings started back in 2007. Some of us on the jQuery team were discussing ways to distribute the test suite load to multiple browsers in an automated fashion. Andy Kent came along and proposed a participatory application for testing visual code (such as jQuery UI). We worked on that code base for a while but it didn’t get off the ground. Eventually I decided to re-tackle the problem early on in 2009. Even in its rough alpha state we’ve already been able to make great use of TestSwarm. For example, here’s a view of jQuery commits run in TestSwarm:

The vertical axis is SVN commits to jQuery (newer commits at the top), the horizontal axis are all the different browsers that we target. Using TestSwarm we’ve been able to easily spot regressions and fix them with a minimum amount of hassle (especially since all the results are logged).

And this is only the beginning. There are so many different directions in which Test Swarm can be taken. For example:

• A pastebin-like service where you can drop in code and see the results come back, from many browsers, in real-time.
• IDE integration for sending minor changes out for quick testing.
• Manual testing of user interface code. Pushing manual tests, with instructions, to users for them to walk through.
• Distributing tests to any number of browsers, rather than a specific sub-set. (You could use this to embed a tiny iframe in your site to collect test results from a small sampling of our users.)
• The ability to drive and test browser code or extensions.

And the list goes on. I’m definitely curious to see what directions the community is interested in driving the code base. I’ve gotten it to a level where it’s particularly useful for me and the jQuery team – where should we go from here?

Posted: August 26th, 2009

35 Comments (Show Comments)

1. Timothy (August 26, 2009 at 9:49 am)

   Once again, good work Resig.

2. James Brooks (August 26, 2009 at 9:51 am)

   I’m joining this!

3. James Brooks (August 26, 2009 at 9:51 am)

   Well I was but apparently you don’t need my help..?

4. John Resig (August 26, 2009 at 9:54 am)

   @James Brooks: It’s going to depend upon which browser you’re using – if you’re not using a browser that’s needed by the swarm then there isn’t much need to join :-) As I mentioned in the “don’t need help note” please send your useragent info to the Google Group so that we can look over it, in case there’s a mistake.

5. James Brooks (August 26, 2009 at 9:59 am)

   Sure thing John. Any browsers in particular you need help with?

6. John Resig (August 26, 2009 at 10:12 am)

   @James: All the ones listed on the Test Swarm home page.

7. Gappa (August 26, 2009 at 10:40 am)

   Nice. I like the idea!

   However, as I am using the RC version of Windows 7, all my browsers were rejected. The only one to join the swarm is the IE6 from Virtual XP Mode – so I could try TestSwarm out.

   Is Win7 support planned in the near future?

8. John Resig (August 26, 2009 at 11:25 am)

   @James & Gappa: I’ve added in Windows 7 support.

9. Gappa (August 26, 2009 at 11:43 am)

   Great, joined the swarm with more browsers.

10. Michaël (August 26, 2009 at 12:11 pm)

    Congratulations, John!

    Do you plan to add Chromium and/or Opera on OS X to the list of browsers? Firefox and Safari are currently the only browsers listed for this OS.

11. Elbrinner (August 26, 2009 at 12:28 pm)

    Muy interesante.

12. John Resig (August 26, 2009 at 1:24 pm)

    @Michaël: Opera is open on OS X, just scroll down. Unfortunately Opera doesn’t announce what version of OS X it’s on so it’s in a generic “OS X” bin (neither does Firefox 2.0). I’m going to wait until there’s an alpha or beta of Chromium before opening it up on the swarm.

13. Chris Wanstrath (August 26, 2009 at 1:28 pm)

    Oh man this is awesome. JavaScript is going cool places and this should really strengthen the ecosystem. Great work John!

14. Edwin Khodabakchian (August 26, 2009 at 3:37 pm)

    Congratulations John: this is really well designed!

15. Svenne (August 26, 2009 at 5:27 pm)

    John – excellent work!

    Our company would like to use spare CPU cycles on our machines to the benefit of Testswarm. My idea is to simply put an 1×1 pixel IFRAME including http://testswarm.com/run/username on our intranet homepage (which is automatically loaded on bootup) – would that be sufficient to have the test start running? Any drawbacks with this approach. Could IFRAME:ing affect the test correctness in any way?

    Great stuff!

16. John Resig (August 26, 2009 at 7:52 pm)

    @Svenne: I’d imagine that that technique would work – I doubt iframeing the tests will have a severe effect – especially considering that they’re iframed already. Any help would be appreciated – thanks!

17. rdza (August 26, 2009 at 8:43 pm)

    wonderful!

    would be great to see iphone support & to convert bespin to use this.

18. Richard (August 27, 2009 at 4:33 am)

    I’ve been looking in to the source code, and while checking it out I noticed there’s quite a lot of $_REQUEST calls. These are dangerous, because they’re coming from three places (Get, Post, Cookie). It’s always better to specify the origin of the variable.

    And furthermore, ereg* functions are deprecated, you should use preg* everywhere. They’re faster, and there’s a lot of goodies that come along with it. :-)

19. Dmitry Chaplinsky (August 27, 2009 at 5:54 am)

    Great job!

    John, what about time measurements for each test? I know, in such distributed testing systems it’s hard to made correct measurements because environment for each browsers are quite different, but it’s very attractive to use similar system for performance testing.

20. John Resig (August 27, 2009 at 10:26 am)

    @Richard: I was very deliberate in my use $_REQUEST. Note that I only use $_POST when I explicitly want a POST request coming in and use $_REQUEST for other cases where non-destructive operations are occurring (and where it doesn’t matter if a GET or POST occurs). I did another pass through but didn’t notice any cases where $_REQUEST was being used for a destructive operation.

    As far as ereg/preg goes, good catch. I’ll definitely check in to that.

    @Dmitry: I actually record the time information for the tests but I don’t have it displayed anywhere yet. I’ll see if I can find a useful way to show that data.

21. Richard (August 27, 2009 at 10:52 am)

    $_REQUEST is the most common source of XSS-attacks, due to the fact that $_COOKIE values are the last values to be copied into it. If you would want just post and get values, just use a simple function:
    

    That would zero out the possibility of XSS attacks via cookies...

22. Dmitry Chaplinsky (August 27, 2009 at 1:50 pm)

    @John: Just an idea, it’s possible to run small common performance test to determine overall environment performance and then use this info to normalize full execution time somehow (and make it less specific to different environment).

23. ratbeard (August 27, 2009 at 11:12 pm)

    You’ve engineered a great solution to this problem, John :)

    I signed up but haven’t been given any tasks to do yet. Is there a way I can manually kick off to run the tests for a commit? Even if it doesn’t contribute anything, I’d like to run them to see that cool grid. Its a little bit of a bummer get excited at the video and sign up, but not have anything to do :]

    I imagine it could work like:

    – go to my testswarm page
    – no new tests to run. I’m useless :(
    – … but on the page is a list of the projects in the system. I can click on jquery
    – now I see a list of the recent commits for jquery. I can click a button to run the last commit, but instead I click on a commit link.
    – now I see the msg and diff for the commit. I can click a button to run the tests for this commit.

    This seems like a great way to get people aware of what active development is going on in their library, and hence more likely to contribute to it in some way.

    I think a simple way to chat with other logged on users would also be very cool. In my above scenario, the jquery project could have its own irc-like chat room, though much less daunting for new users! People can talk about their library there, ask questions, watch as new commits come in, use the performance data to brag about how fast their computer is.

    As you can see I think this can have a lot of potential even beyond testing – which already is exciting enough!

    ???

24. Alex (August 28, 2009 at 12:38 am)

    My cpu is just sitting here waiting to be used john!! Where are my tests?! =)

    Very cool stuff tho! Great work as always! Just out of curiosity, have you considered using http push? I’m not familiar with the implications of using it but this seems like it might be a good application for it.

25. Dmitry Chaplinsky (August 28, 2009 at 4:17 am)

    @ratbeard, @Alex, I suppose there will be way more projects to test, because of (un)expected success of this app and lot of volunteers :)

26. Asbjørn Ulsberg (August 28, 2009 at 10:00 am)

    Excellent work, awesome tool, incredible execution! You are my personal hero, John! :-) I’ve joined the swarm, but unfortunately there’s no tests for me to execute. Gief moar tests plx!

27. Michaël (September 1, 2009 at 2:09 pm)

    Just a detail, John: I see you already replaced Opera’s icon with the one from Opera 10 on the Test Swarm main page. But Firefox still uses the icon from Firefox 3 and not the new, better one from Firefox 3.5. It’s a little bit funny because, since you work for Mozilla, I would have bet that Firefox would have been the first to have its icon updated on the Test Swarm website :)

28. David (September 10, 2009 at 1:06 pm)

    Awesome work, John.

    My user page says that I connected 3 hours ago…Its 2:01pm (GMT-5:00) Eastern Standard right now and I first connected at 1:51pm. Is there some sort of time machine involved here? haha.

29. rajakvk (September 13, 2009 at 1:41 am)

    Awesome!! I’m going to share
    IE6, IE7, IE8, FF2, FF3, FF3.5, Opera on Win XP and Safari, FF3.5 on Leopard.

30. Dmitry Chaplinsky (September 15, 2009 at 5:56 am)

    John, another proposition is to show count of connected browsers (with same version as your) right on test page. Something like:
    http://img.skitch.com/20090915-djh8nd4mskt3yu3qxx65cj2fn4.png

31. Dmitry Chaplinsky (September 21, 2009 at 2:49 pm)

    Looks like your routine for scores calculation is very non-efficient and overloading DB a lot. At least, it’s hard to obtain the page with scores in reasonable amount of time.

32. Joel (September 28, 2009 at 11:35 pm)

    Haven’t been able to access the front page of the site for days on end :-/

33. foo (November 11, 2009 at 10:39 pm)

    how about mobile browser support?

34. Cheap home insurance quotes >> Tips on getting home insurance quotes ... (November 27, 2009 at 8:49 am)

    ..]one useful source of tips on this issueis ,ejohn.org,..]

35. Mike (October 9, 2010 at 6:19 am)

    Have not been able to use this in a while.

Comments are closed.
Comments are automatically turned off two weeks after the original post. If you have a question concerning the content of this post, please feel free to contact me.