It was pretty hard to get work done yesterday. My feed reader was filled with stories of how Apple is pulling out of Macworld and my IM and email were bursting with people asking me what I think about FireUnit.
I’m the co-author of two tools that overlap with FireUnit, Windmill and MozMill. I say overlap and not “compete” because I think the tools differ enough in their goals that they don’t directly compete.
FireUnit has some limitations. It’s an extension to Firebug which means it will only run on Firefox. It’s focused on providing a simple API for people who are competent JavaScript programmers. I think it’s safe to say that there is an emphasis on unittesting and not functional testing, and certainly an emphasis on testing content and not chrome.
Something really great happens when you create a tool that embraces limitations instead of trying to be too much to too many people. You tend to be able to create better smaller purpose tools for a selection of people that have been beating themselves up trying to make tools with less limitations work for a smaller use case.
The API
// Simple true-like/false-like testing fireunit.ok( true, "I'm going to pass!" ); fireunit.ok( false, "I'm going to fail!" ); // Compare two strings - shows a diff of the // results if they're different fireunit.compare( "The lazy fox jumped over the log.", "The lazy brown fox jumped the log.", "Are these two strings the same?" ); // Compare a string using a regular expression fireunit.reCompare( /The .* fox jumped the log./, "The lazy brown fox jumped the log.", "Compare a string using a RegExp." ); // Display the total results fireunit.testDone();
This is the example from John Resig’s post about FireUnit. I think it illustrates how awesome the API is. I’ve always been a huge fan of this “ok” style API instead of the larger jUnit style API I keep being talked in to implementing. Clearly FireUnit assumes it’s audience is JavaScript programmers that can write normal expressions to handle their pass/fail logic without the hand holding of a bigger API.
The beauty of the comparison API you really don’t appreciate until you look at the UI.
Debugging
I’m most critical of tools that don’t provide sufficient debugging tools. The test debugging environment is even more important than the test authoring environment in the long run.
There is no better tool for debugging the web than Firebug. A big part of Windmill’s debugging strategy is it’s integration of Firebug and Firebug Lite because there is just no better tool in the world. Firebug ROCKS!
FireUnit is built as an extension of Firebug, it’s integrated in a beautiful and intuitive way so they’ve totally dodged the need for any additional debugging work outside of creating a good UI for the test bits. I’m calling this out because I think as time goes on people aren’t going to talk about the debugging experience in FireUnit because they are so used to already using Firebug for debugging their regular development and won’t recognize how good they have it debugging their tests.
The UI
Another benefit of embracing limitations is that you trim down your feature set to only what is actually necessary. The upside to having a simple API is that you reduce the complexity of test data you need to represent in the UI. With all that going for it I expected the FireUnit UI to be pretty good but I was still blown away.
Look at that comparison UI!
The Bad
Within the limitations described above FireUnit is great but as soon as you start to cross them you’ll be disappointed.
Although they have some helper APIs for doing some functional testing it’s not a good functional testing tool. The tests are async which is a real pain for functional testing which is incredibly synchronous. But I think if you want to do functional testing multiple browser support is a big concern and will already preclude you from using FireUnit.
Although you can test Chrome (Mozilla Chrome not Google Chrome) I wouldn’t venture outside of unittests. If you need to start getting at anonymous nodes and simulating events on them you’re gonna see your code get about 4 times larger than a typical mozmill test.
Right now there doesn’t seem to be a way to run FireUnit tests in continuous integration (tinderbox, buildbot, hudson, etc). But this could easily be overcome using jsbridge and I’m happy to point the FireUnit guys in the right direction 
At the end of the day I think there is a huge need for a tool like FireUnit that just does what it does well and doesn’t try to be too much to too many people. A tool like this is guaranteed to make it’s users very happy and I think those users are currently unhappy with the existing set of automation tools that are trying to fill a larger feature set.
