It was pretty hard to get work done yesterday. My feed reader was filled with stories of how Apple is pulling out of Macworld and my IM and email were bursting with people asking me what I think about FireUnit.

I’m the co-author of two tools that overlap with FireUnit, Windmill and MozMill. I say overlap and not “compete” because I think the tools differ enough in their goals that they don’t directly compete.

FireUnit has some limitations. It’s an extension to Firebug which means it will only run on Firefox. It’s focused on providing a simple API for people who are competent JavaScript programmers. I think it’s safe to say that there is an emphasis on unittesting and not functional testing, and certainly an emphasis on testing content and not chrome.

Something really great happens when you create a tool that embraces limitations instead of trying to be too much to too many people. You tend to be able to create better smaller purpose tools for a selection of people that have been beating themselves up trying to make tools with less limitations work for a smaller use case.

The API

// Simple true-like/false-like testing
fireunit.ok( true, "I'm going to pass!" );
fireunit.ok( false, "I'm going to fail!" );
 
// Compare two strings - shows a diff of the
// results if they're different
fireunit.compare(
  "The lazy fox jumped over the log.",
  "The lazy brown fox jumped the log.",
  "Are these two strings the same?"
);
 
// Compare a string using a regular expression
fireunit.reCompare(
  /The .* fox jumped the log./,
  "The lazy brown fox jumped the log.",
  "Compare a string using a RegExp."
);
 
// Display the total results
fireunit.testDone();

This is the example from John Resig’s post about FireUnit. I think it illustrates how awesome the API is. I’ve always been a huge fan of this “ok” style API instead of the larger jUnit style API I keep being talked in to implementing. Clearly FireUnit assumes it’s audience is JavaScript programmers that can write normal expressions to handle their pass/fail logic without the hand holding of a bigger API.

The beauty of the comparison API you really don’t appreciate until you look at the UI.

Debugging

I’m most critical of tools that don’t provide sufficient debugging tools. The test debugging environment is even more important than the test authoring environment in the long run.

There is no better tool for debugging the web than Firebug. A big part of Windmill’s debugging strategy is it’s integration of Firebug and Firebug Lite because there is just no better tool in the world. Firebug ROCKS!

FireUnit is built as an extension of Firebug, it’s integrated in a beautiful and intuitive way so they’ve totally dodged the need for any additional debugging work outside of creating a good UI for the test bits. I’m calling this out because I think as time goes on people aren’t going to talk about the debugging experience in FireUnit because they are so used to already using Firebug for debugging their regular development and won’t recognize how good they have it debugging their tests.

The UI

Another benefit of embracing limitations is that you trim down your feature set to only what is actually necessary. The upside to having a simple API is that you reduce the complexity of test data you need to represent in the UI. With all that going for it I expected the FireUnit UI to be pretty good but I was still blown away.

Look at that comparison UI!

The Bad

Within the limitations described above FireUnit is great but as soon as you start to cross them you’ll be disappointed.

Although they have some helper APIs for doing some functional testing it’s not a good functional testing tool. The tests are async which is a real pain for functional testing which is incredibly synchronous. But I think if you want to do functional testing multiple browser support is a big concern and will already preclude you from using FireUnit.

Although you can test Chrome (Mozilla Chrome not Google Chrome) I wouldn’t venture outside of unittests. If you need to start getting at anonymous nodes and simulating events on them you’re gonna see your code get about 4 times larger than a typical mozmill test.

Right now there doesn’t seem to be a way to run FireUnit tests in continuous integration (tinderbox, buildbot, hudson, etc). But this could easily be overcome using jsbridge and I’m happy to point the FireUnit guys in the right direction

At the end of the day I think there is a huge need for a tool like FireUnit that just does what it does well and doesn’t try to be too much to too many people. A tool like this is guaranteed to make it’s users very happy and I think those users are currently unhappy with the existing set of automation tools that are trying to fill a larger feature set.

One of the big questions we’ve been getting since we outlined how to write unittests in mozmill is how to serve out local files over HTTP. For quite a while I’ve had a port of mochitest’s js http server in the mozmill source tree but hadn’t written a nice API for you to be able to use it inside of your mozmill tests.

Ideally what I wanted was an API that worked a lot like how the dependencies work, which could serve out local files relative to the tests location on the filesystem. This way we don’t rely on any Python end to tell use what to serve out and can handle starting and stopping the server inside of the framework in the extension without any extra work from the test author. I finished the implementation a few weeks back but just documented it today.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var url = collector.addHttpResource('./files');
 
var setupModule = function(module) {
  module.controller = mozmill.getBrowserController();
}
 
var testFoo = function(){
  controller.open(url+'index.html');
}

This made it in before the last release so it’s available for use right away. Enjoy

Everyday I wake up and wonder why I own such a large car. Don’t get me wrong, I love my car, it’s a black 2008 touring edition Prius with the full geek package (bluetooth, GPS, wireless key). But I really don’t drive it much.

Most days I ride my bike and when I go to San Francisco it’s just easier to take the Bart most of the time. I rarely fill my car with any additional passengers unless you count my dog. All said and done I’d say the car only leaves it’s parking spot about 3 days a week, and most of the drives are less than 50 miles total. Sure, once a quarter we have an onsite at Mozilla and I drive to Mountain View 4 days in a row, but other than that I don’t take many long trips.

For the most part, my life doesn’t require such a large car. I do drive just a little too often to use a flex car, but almost all my driving is short distance and with only one or two people in the car. If I had to rethink my purchase it would have been sufficient to buy a smart car.

This is my transit story, but this is quickly becoming the transit story for many Americans. I know very few people that have decided to buy homes far from urban areas, and only one of them from my generation.

Thinking forward

The costs of commuting to work will only increase in the next 5 years. Gas prices seem lower now, but remember that there is only so much oil in the world and the closer we get to running out of it the more expensive it becomes.

My generation never seemed to think of employment as having much permanence which underlined the need to live in a more urban area with greater chance of finding new employment. The current economic crisis is obviously increasing the lack of confidence in holding on to any current employment so there is very little incentive to move away from employment centers.

The more people live in urban areas the less need they have for single commuter vehicles and the more fuel prices increase the smaller any single commuter vehicles will need to be.

American cars are notoriously larger than they need to be. Vehicle efficiency is about size and weight just as much as it is about the fuel source. Yet all the plans for more environmentally friendly vehicles from the American automakers seem to be identical cars to their current line with electric/hybrid/flex fuel engines for higher prices. Meanwhile, a new set of companies is making inroads in alternate vehicles that are a closer match to our new transit patterns.

Add to this the expansive transit plans of the Obama administration and you have far less need for vehicles even as large as a Japanese sedan. I know of only one small electric vehicle from a traditional automaker and that’s the Mini EV which is just now entering the public trial phase.

Being green only takes you so far

Morals are a distant second to economic necessities when making wide scale changes in human behavior. Thinking that environmentalism can change transportation behavior requires that the majority of people driving put their personal responsibility for future generations and the planet ahead of any social and economic realities they face. Not gonna happen.

Fortunately the social and economic realities are driving us towards more efficiency and sustainability in our transit patterns which, if we do this right, might just end up saving the planet.

We have to let them fail

Our problem is that we continue to turn to old institutions to create transformative products. Ford and GM can’t make a car that is efficient from the ground up, there are just too many incentives for them to do things the same way they always have even if forced to changes things a bit by congress.

These companies are going to fail, big time. It’s an inevitability. People can’t afford to buy what they are selling and they just can’t make what people need.

Transformational products are already coming from a bunch of new companies. Companies that, believe it or not, employ people. The more their products are bought and manufacturing of their vehicles increases the more workers they will need. In fact, medium sized business tend to be much better employers than large corporations, so a new movement of medium sized auto companies in the long term would be much better for workers than trying to save the current lot.

An yes, steel production will decrease, steel companies will fail and people will lose their jobs, because we need to use less steel. The suppliers of the big auto companies will also fail, because we don’t need their parts, we need new parts from new companies. All those jobs will be replaced by new companies and new jobs because you know what, people still need to get around.

It took a few months but the Broakland guys built me a new custom track bike.

I’m still getting used to it, evidenced by the fact that I took huge dive the other day, but it rides beautifully.

In jsbridge we have a socket that stays open constantly and the JavaScript end streams JSON objects for all the events we care about in the browser back over to Python. In jsbridge this happens asynchronously and we use asyncore since it’s available in stdlib and is pretty lightweight, but the bulk of the parsing code is universal.

import simplejson
decoder = simplejson.JSONDecoder()
 
class StreamReader(object):
    sbuffer = ''
    def process_read(self, data):
        """Parse out json objects and fire callbacks."""
        self.sbuffer += data
        self.parsing = True
        while self.parsing:
            # Remove erroneus data in front of callback object
            index = self.sbuffer.find('{')
            if index is not -1 and index is not 0:
                self.sbuffer = self.sbuffer[index:]
            # Try to get a json object from the data stream
            try:
                obj, index = decoder.raw_decode(self.sbuffer)
            except Exception, e:
                self.parsing = False
            # If we got an object fire the callback infra
            if self.parsing:
                self.fire_callbacks(obj)
                self.sbuffer = self.sbuffer[index:]

The process_read function gets called every time a block of data gets read from the stream so it needs to be able to handle multiple JSON objects in a single read, objects that span multiple reads, and junk (usually spaces and return carriers) showing up between JSON objects.

If you wanna see this code in the wild you can check out the jsbridge source.

One of the first pieces of feedback we got on MozMill was that there didn’t seem to be any way to click on system menus.

System menus are defined in XUL using a special set of elements. Those are then displayed differently on every operating system and as such we can’t use tools like the MozMill Inspector to get repeatable elements to click on.

After digging in to this problem a bit I figured out that the elements defining menus can be traversed and that they even respond properly to .click(). I then wrote thin API around the elements so that they can be easily looked up by name the way they are displayed in the menus. You can see the full documentation here.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var setupModule = function (module) {
  module.controller = mozmill.getBrowserController();
}
 
var testMenu = function () {
  // Example 1: Click menus without testing the click
  controller.menus.File["New Tab"].click();
  controller.menus.View.Sidebar.Bookmarks.click();
  controller.menus.View["Page Style"]["No Style"].click();
  // Example 2: Click menus and test the click
  controller.click(new elementslib.Elem(controller.menus.File["New Tab"]));
  controller.click(new elementslib.Elem(controller.menus.View.Sidebar.Bookmarks));
  controller.click(new elementslib.Elem(controller.menus.View["Page Style"]["No Style"]));
}

As you can see the API is a simple chain of objects by name cascading down the different sub menus until a menu element is reached. Attribute access is just plain old JavaScript so you only need to use ["name"] syntax when looking up menus with names that contain spaces. The API is most definitely case sensitive.

MozMill already has some big advantages over mochitest. For one, you don’t need a special test build, you can write and run tests against any build of a Mozilla Application. You can even write and run tests just using the extension and not worry about any Python stuff until you set it up in continuous integration.

MozMill already runs in “chrome”, or what Adam and I have been referring to as the “trusted” environment. All the tests that get run in MozMill are imported in this environment, which means you have access to pretty much everything in the Mozilla environment. We also have a dependency system and a setup/teardown system for tests.

All we were missing was a simple unittest API that can trigger failures without causing exceptions like the MozMill functional testing APIs. I looked at what others have done including dojo, mochikit, and jsunit.

Fortunately MozMill gets out of any test registration and staggering stuff, it already has a well defined way of defining tests and dependencies and a clear way to run tests for debugging and continuous integration. Of all the different jum (JavaScript Unittest Module) APIs I liked jsunit the most except for the incredibly stupid idea of passing the comment argument first instead of making it an optional last argument.

In the end MozMill got it’s own jum which I’ve just finished up documenting.

var jum = {}; Components.utils.import('resource://mozmill/modules/jum.js', jum);
 
var testAsserts = function() {
  jum.assert(true);
  jum.assertTrue(true);
  jum.assertFalse(false);
  jum.assertEquals('asdf', 'asdf');
  jum.assertNotEquals('asdf', 'fdsa');
  jum.assertNull(null);
  jum.assertNotNull(true);
  jum.assertUndefined({}.asdf);
  jum.assertNotUndefined('asdf');
  jum.assertNaN('a');
  jum.assertNotNaN(4);
  jum.pass();
}
 
var testNotAsserts = function() {
  // All of these calls should fail
  jum.assert(false);
  jum.assertTrue(false);
  jum.assertTrue('asf');
  jum.assertFalse(true);
  jum.assertFalse('asdf');
  jum.assertEquals('asdf', 'fdsa');
  jum.assertNotEquals('asdf', 'asdf');
  jum.assertNull(true);
  jum.assertNotNull(null);
  jum.assertUndefined('asdf');
  jum.assertNotUndefined({}.asdf);
  jum.assertNaN(4);
  jum.assertNotNaN('f');
  jum.fail();
}

All of these functions take an optional comment argument which can be any JSON serializable value (string, int, obj, array, null, bool).

What we’re missing right now is the http local file server. I’ve already ported mochikit’s web server to be a resource but I’m still working on a better API for using it. What we want is for tests to simply define local relative paths that should be mounted at different local url namespaces and the framework can handle starting and stopping the server if necessary. This API will be finished in the next few weeks and ready for MozMill 1.0 (which we’re looking at pushing out before January 1st).