It was pretty hard to get work done yesterday. My feed reader was filled with stories of how Apple is pulling out of Macworld and my IM and email were bursting with people asking me what I think about FireUnit.

I’m the co-author of two tools that overlap with FireUnit, Windmill and MozMill. I say overlap and not “compete” because I think the tools differ enough in their goals that they don’t directly compete.

FireUnit has some limitations. It’s an extension to Firebug which means it will only run on Firefox. It’s focused on providing a simple API for people who are competent JavaScript programmers. I think it’s safe to say that there is an emphasis on unittesting and not functional testing, and certainly an emphasis on testing content and not chrome.

Something really great happens when you create a tool that embraces limitations instead of trying to be too much to too many people. You tend to be able to create better smaller purpose tools for a selection of people that have been beating themselves up trying to make tools with less limitations work for a smaller use case.

The API

// Simple true-like/false-like testing
fireunit.ok( true, "I'm going to pass!" );
fireunit.ok( false, "I'm going to fail!" );
 
// Compare two strings - shows a diff of the
// results if they're different
fireunit.compare(
  "The lazy fox jumped over the log.",
  "The lazy brown fox jumped the log.",
  "Are these two strings the same?"
);
 
// Compare a string using a regular expression
fireunit.reCompare(
  /The .* fox jumped the log./,
  "The lazy brown fox jumped the log.",
  "Compare a string using a RegExp."
);
 
// Display the total results
fireunit.testDone();

This is the example from John Resig’s post about FireUnit. I think it illustrates how awesome the API is. I’ve always been a huge fan of this “ok” style API instead of the larger jUnit style API I keep being talked in to implementing. Clearly FireUnit assumes it’s audience is JavaScript programmers that can write normal expressions to handle their pass/fail logic without the hand holding of a bigger API.

The beauty of the comparison API you really don’t appreciate until you look at the UI.

Debugging

I’m most critical of tools that don’t provide sufficient debugging tools. The test debugging environment is even more important than the test authoring environment in the long run.

There is no better tool for debugging the web than Firebug. A big part of Windmill’s debugging strategy is it’s integration of Firebug and Firebug Lite because there is just no better tool in the world. Firebug ROCKS!

FireUnit is built as an extension of Firebug, it’s integrated in a beautiful and intuitive way so they’ve totally dodged the need for any additional debugging work outside of creating a good UI for the test bits. I’m calling this out because I think as time goes on people aren’t going to talk about the debugging experience in FireUnit because they are so used to already using Firebug for debugging their regular development and won’t recognize how good they have it debugging their tests.

The UI

Another benefit of embracing limitations is that you trim down your feature set to only what is actually necessary. The upside to having a simple API is that you reduce the complexity of test data you need to represent in the UI. With all that going for it I expected the FireUnit UI to be pretty good but I was still blown away.

Look at that comparison UI!

The Bad

Within the limitations described above FireUnit is great but as soon as you start to cross them you’ll be disappointed.

Although they have some helper APIs for doing some functional testing it’s not a good functional testing tool. The tests are async which is a real pain for functional testing which is incredibly synchronous. But I think if you want to do functional testing multiple browser support is a big concern and will already preclude you from using FireUnit.

Although you can test Chrome (Mozilla Chrome not Google Chrome) I wouldn’t venture outside of unittests. If you need to start getting at anonymous nodes and simulating events on them you’re gonna see your code get about 4 times larger than a typical mozmill test.

Right now there doesn’t seem to be a way to run FireUnit tests in continuous integration (tinderbox, buildbot, hudson, etc). But this could easily be overcome using jsbridge and I’m happy to point the FireUnit guys in the right direction

At the end of the day I think there is a huge need for a tool like FireUnit that just does what it does well and doesn’t try to be too much to too many people. A tool like this is guaranteed to make it’s users very happy and I think those users are currently unhappy with the existing set of automation tools that are trying to fill a larger feature set.

One of the big questions we’ve been getting since we outlined how to write unittests in mozmill is how to serve out local files over HTTP. For quite a while I’ve had a port of mochitest’s js http server in the mozmill source tree but hadn’t written a nice API for you to be able to use it inside of your mozmill tests.

Ideally what I wanted was an API that worked a lot like how the dependencies work, which could serve out local files relative to the tests location on the filesystem. This way we don’t rely on any Python end to tell use what to serve out and can handle starting and stopping the server inside of the framework in the extension without any extra work from the test author. I finished the implementation a few weeks back but just documented it today.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var url = collector.addHttpResource('./files');
 
var setupModule = function(module) {
  module.controller = mozmill.getBrowserController();
}
 
var testFoo = function(){
  controller.open(url+'index.html');
}

This made it in before the last release so it’s available for use right away. Enjoy

In jsbridge we have a socket that stays open constantly and the JavaScript end streams JSON objects for all the events we care about in the browser back over to Python. In jsbridge this happens asynchronously and we use asyncore since it’s available in stdlib and is pretty lightweight, but the bulk of the parsing code is universal.

import simplejson
decoder = simplejson.JSONDecoder()
 
class StreamReader(object):
    sbuffer = ''
    def process_read(self, data):
        """Parse out json objects and fire callbacks."""
        self.sbuffer += data
        self.parsing = True
        while self.parsing:
            # Remove erroneus data in front of callback object
            index = self.sbuffer.find('{')
            if index is not -1 and index is not 0:
                self.sbuffer = self.sbuffer[index:]
            # Try to get a json object from the data stream
            try:
                obj, index = decoder.raw_decode(self.sbuffer)
            except Exception, e:
                self.parsing = False
            # If we got an object fire the callback infra
            if self.parsing:
                self.fire_callbacks(obj)
                self.sbuffer = self.sbuffer[index:]

The process_read function gets called every time a block of data gets read from the stream so it needs to be able to handle multiple JSON objects in a single read, objects that span multiple reads, and junk (usually spaces and return carriers) showing up between JSON objects.

If you wanna see this code in the wild you can check out the jsbridge source.

One of the first pieces of feedback we got on MozMill was that there didn’t seem to be any way to click on system menus.

System menus are defined in XUL using a special set of elements. Those are then displayed differently on every operating system and as such we can’t use tools like the MozMill Inspector to get repeatable elements to click on.

After digging in to this problem a bit I figured out that the elements defining menus can be traversed and that they even respond properly to .click(). I then wrote thin API around the elements so that they can be easily looked up by name the way they are displayed in the menus. You can see the full documentation here.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var setupModule = function (module) {
  module.controller = mozmill.getBrowserController();
}
 
var testMenu = function () {
  // Example 1: Click menus without testing the click
  controller.menus.File["New Tab"].click();
  controller.menus.View.Sidebar.Bookmarks.click();
  controller.menus.View["Page Style"]["No Style"].click();
  // Example 2: Click menus and test the click
  controller.click(new elementslib.Elem(controller.menus.File["New Tab"]));
  controller.click(new elementslib.Elem(controller.menus.View.Sidebar.Bookmarks));
  controller.click(new elementslib.Elem(controller.menus.View["Page Style"]["No Style"]));
}

As you can see the API is a simple chain of objects by name cascading down the different sub menus until a menu element is reached. Attribute access is just plain old JavaScript so you only need to use ["name"] syntax when looking up menus with names that contain spaces. The API is most definitely case sensitive.

MozMill already has some big advantages over mochitest. For one, you don’t need a special test build, you can write and run tests against any build of a Mozilla Application. You can even write and run tests just using the extension and not worry about any Python stuff until you set it up in continuous integration.

MozMill already runs in “chrome”, or what Adam and I have been referring to as the “trusted” environment. All the tests that get run in MozMill are imported in this environment, which means you have access to pretty much everything in the Mozilla environment. We also have a dependency system and a setup/teardown system for tests.

All we were missing was a simple unittest API that can trigger failures without causing exceptions like the MozMill functional testing APIs. I looked at what others have done including dojo, mochikit, and jsunit.

Fortunately MozMill gets out of any test registration and staggering stuff, it already has a well defined way of defining tests and dependencies and a clear way to run tests for debugging and continuous integration. Of all the different jum (JavaScript Unittest Module) APIs I liked jsunit the most except for the incredibly stupid idea of passing the comment argument first instead of making it an optional last argument.

In the end MozMill got it’s own jum which I’ve just finished up documenting.

var jum = {}; Components.utils.import('resource://mozmill/modules/jum.js', jum);
 
var testAsserts = function() {
  jum.assert(true);
  jum.assertTrue(true);
  jum.assertFalse(false);
  jum.assertEquals('asdf', 'asdf');
  jum.assertNotEquals('asdf', 'fdsa');
  jum.assertNull(null);
  jum.assertNotNull(true);
  jum.assertUndefined({}.asdf);
  jum.assertNotUndefined('asdf');
  jum.assertNaN('a');
  jum.assertNotNaN(4);
  jum.pass();
}
 
var testNotAsserts = function() {
  // All of these calls should fail
  jum.assert(false);
  jum.assertTrue(false);
  jum.assertTrue('asf');
  jum.assertFalse(true);
  jum.assertFalse('asdf');
  jum.assertEquals('asdf', 'fdsa');
  jum.assertNotEquals('asdf', 'asdf');
  jum.assertNull(true);
  jum.assertNotNull(null);
  jum.assertUndefined('asdf');
  jum.assertNotUndefined({}.asdf);
  jum.assertNaN(4);
  jum.assertNotNaN('f');
  jum.fail();
}

All of these functions take an optional comment argument which can be any JSON serializable value (string, int, obj, array, null, bool).

What we’re missing right now is the http local file server. I’ve already ported mochikit’s web server to be a resource but I’m still working on a better API for using it. What we want is for tests to simply define local relative paths that should be mounted at different local url namespaces and the framework can handle starting and stopping the server if necessary. This API will be finished in the next few weeks and ready for MozMill 1.0 (which we’re looking at pushing out before January 1st).

Last week we finished up the 1.0RC2 release of MozMill.

The biggest new feature was is the Test Recorder. You can now hit the Record button and use the Application normally and create a test outline. I say “test outline” because it’s usually necessary after creating a recorded test to go in and fill it up with a few conditional wait calls, like waitForElement and waitForPageLoad. If you don’t the test runs too fast and ends up failing pretty quickly. We’re working on some future code to pick up things like new page loads so that we can add those waits in a recorded test but for now you’ll have to do them by hand.

We also have significantly improved some of the documentation.

• http://code.google.com/p/mozmill/wiki/MozMillTestBasics
• http://code.google.com/p/mozmill/wiki/CommandLine
• http://code.google.com/p/mozmill/wiki/MenuAPI

I’ll be writing some more blog entries this week on using MozMill for writing and running different kinds of tests.

I started working on this library months ago and it’s been getting some use already. I spoke about it a bit at Mozilla Summit and as a result there was a post on Ajaxian, but for the most part I’ve been holding off on promoting it until I solidified the API and pushed out a solid release.

jsbridge is a Python to JavaScript translation interface for Mozilla applications. It’s implemented as an extension and can be dynamically installed and launched in a new “clean” profile using mozrunner.

You can call functions and set variables in any window in any scope from Python and those objects are returned and represented as Python objects (which are just classes that inherit from Python base types). There are obvious limitations for assignment and representations in Python, you can really only send objects over the bridge that can be JSON serialized, but it does support assignments by reference so you can move around and manipulate anything that’s already there.

But the most interesting feature is the event system. The event system is available as a resource from inside the Mozilla environment (in other words you can import it from your Firefox extension). Events fired on the JavaScript end are serialized as JSON and sent to the Python side. This means you can add listeners in Python to events fired from the Mozilla JavaScript environment.

This is a big deal. It means we can write significantly cleaner tools. We can now create tools solely as Firefox extensions that don’t require any special Python code to run, but have code that checks if jsbridge is installed and send pertinent events over the bridge for continuous integration. No more printing to console and parsing in Python, no more outputting to log files and parsing them, no more scripts calling scripts and OH MAN MY HEAD HURTS!

The documentation has some great examples.

For a little snippet of the events system in the wild here is some new mozmill Python code that sends all the events to the Python logger.

import logging
logger = logging.getLogger("mozmill")
 
from jsbridge import events
 
class LoggerListener(object):
    cases = {
        'mozmill.pass':   lambda obj: logger.debug('Test Pass: '+repr(obj)),
        'mozmill.fail':   lambda obj: logger.error('Test Failure: '+repr(obj)),
    }
 
    class default(object):
        def __init__(self, eName): self.eName = eName
        def __call__(self, obj): logger.info(self.eName+' :: '+repr(obj))
 
    def __call__(self, eName, obj):
        if self.cases.has_key(eName):
            self.cases[eName](obj)
        else:
            self.cases[eName] = self.default(eName)
            self.cases[eName](obj)
 
events.add_global_listener(LoggerListener())
logging.basicConfig(level=logging.ERROR)

Ubiquity rocks! A month or so ago I completely removed the search box from Firefox and started solely using ubiquity for searches.

The one thing that I really missed was a python doc search that I used to have for the search box. For a while I just used the awesome bar to perform google and history searches for python docs but that basically broke when Python 2.6 was released since all the old docs.python.org urls that google returns break because 2.6 has completely new docs provided by Sphinx.

So today I took a break from my “normal” work to write a ubiquity command to bring up the search page from docs.python.org for your input text. You can see the source here and you should be able to install it by viewing this blog post.

In the future I’d like to do better previews of the searches but I can’t seem to find any web API for searching docs.python.org.

Last week we finished up the Beta2 Release for MozMill.

You can install it from AMO.

We’ve been gathering feedback from our Beta1 release and pushed out another incremental beta release before the big push for MozMill 1.0 RC1.

One important thing we did was improve the inspector workflow. It’s much smother clicking on an element and coming back to the Inspector. The inspector also gives you much more information now, including what document you should be passing the Elem you create and what method you should use to create a controller for the window you’re testing.

Lot’s of bug fixes, special thanks to Tony, Tracy and a big hug to Farhad for feature suggestions and filing bugs against beta1.

This release does not include the new framework that hooks up to jsbridge, and therefor isn’t ready for full continuous integration, that’s now in trunk getting worked on for the RC1 release.

I didn’t think we’d be doing any notable windmill releases until 1.0. Boy was I wrong!

Seriously Faster

On Wednesday Adam messaged me and said that the windmill startup time was too slow. He was right, we’ve know about this for a while but hadn’t put a lot of serious thought in to how we could reduce it.
The issue here was was that we have about 50 JavaScript files that need to get loaded for windmill to start.

Enter windmill-compressor, a new url namespace we added that concats all the js windmill needs in to one file and minifies it. We do this dynamically when windmill starts up, because adding a “build” step would just be too….. Java.

That reduced the startup time from 5-10 seconds to around 2 seconds. But that wasn’t good enough. We saw that most of that startup time was blocking waiting for the compressor to finish, so I threaded it when we start the windmill service and it does the compression while we wait for the browser to start up.

In all, windmill startup times are about 10x faster than 0.9!

Adam also decided to make our page load wait code a bit more aggressive which made not only startup times faster but all of our tests!

Native JavaScript Test Framework

We tied up the loose ends on the JavaScript testing framework and it can now shutdown all of windmill in it’s own teardown, hello continuous integration for native JavaScript tests

This was the last thing holding us back from promoting this along side the Python test authoring library.

We Love Firebug

In 0.9 we introduced Firebug Lite integration for windmill on all browsers. But when you’re using Firefox you probably still want to use the full Firebug extension, which is easy enough to integrate since we use mozrunner for Firefox launching.

Windmill now has a “firebug” command line argument that installs the full Firebug
extension when launching Firefox.