The new year is bringing some big changes for me. A few weeks back I accepted a position at Relaxed Inc. and notified Mozilla that I would be leaving at the end of the year.

Mozilla

I started working at Mozilla 2 years ago. I started the day after my employment at the Open Source Applications Foundation ended. At this point I already took for granted some of the best parts of working at Mozilla; working for a public benefit organization, spending 100% of my time working on Open Source, working with very smart people in the open (lists, IRC, etc.).

But Mozilla is even more than all that. Succeeding at Mozilla means something more than a pat on the back and a good end of the year review. When you succeed at Mozilla you impact one of the most important products on the internet. You reach hundreds of millions of users and contribute to keeping the web an open and free (as in speech) world. There is no other place in the world you can work where you can conceivably have this kind of impact.

Mozilla as an organization is truly unique. Last year was the hardest I’ve ever had, i suffered a huge loss in my personal life and Mozilla was as supportive during this time as any of my friends or family. There are a lot of places that let you put so much of yourself in to the organization to help it attain it’s goals but there are only a handful that are there to support you when you need it.

Relaxed Inc.

I started using CouchDB in 2008 after a great talk by Jan Lehnardt at OSCON. I started using it right away and over the next year it re-shaped how I think about web development and applications. In the last 6 months my group at Mozilla has become a heavy CouchDB user and not just because of my own interest but because CouchDB was the only solution for some of the harder problems we needed to solve with our results storage.

As I’ve used CouchDB more and more and become a part of the CouchDB community I’ve had the pleasure of knowing some of the core contributors, three of which have decided to found a new startup around CouchDB; Jan Lehnardt, J Chris Anderson, and the creator of CouchDB Damien Katz. Shorty after they received their funding they made an offer. It’s an amazing opportunity and while the decision to leave Mozilla is one of the hardest I’ve ever had to make I’m very excited about my future at Relaxed.

The Future

I’m really looking forward to working with everyone at Relaxed. It’s an exciting time and I’m not 100% sure yet which projects I currently work on that I will still have time to maintain. In the next week or so I’ll be doing a blog post on all the libraries I currently work on and maintain (it’s a long list) and what their status is moving forward. I still maintain code I wrote long before I worked at Mozilla and have every intention of continuing to work on some of the projects I started at Mozilla.

One thing is certain. I’m not the guy who figures out how to test the browser any more. Windmill and Mozmill are important projects that I have every intention of supporting by making time for code reviews and community support but I won’t be available to put time in to new feature work and refactoring like I have in the past. Luckily there are solid communities behind both of these projects and I’m confident that there are people who can continue to drive them in the future.

I don’t know what is going to happen next, all I know is that it should be fun, it won’t be like anything I’ve done before, and will certainly continue to include lots JavaScript and Python.

For everyone who depends on me and the code I’ve written over the last few years I’ll be sure to keep you all up to date. And one thing I can promise is that if you want to fix anything in one my projects, fork it on github and send me a pull request and I will always find time to look at it

We’re gearing up for some heavy CouchDB usage in a new automation system and it has fallen upon me to do some performance benchmarking.

The most important thing for us to figure out was whether or not the CentOS virtual machine we’re currently running CouchDB on is going to be enough even in the short term. Until today we’ve been running 0.9 and have encountered performance problems.

Our main bottleneck is, and has always been, view generation and update performance. We tend to have medium to large size documents (jobs are relatively small but results from test runs can be incredibly large).

View generation of large documents has typically been our biggest issue which we have previously mitigated by refreshing all views after any large write but that isn’t going to work for the amount of results that we plan on pouring in to the new system.

Last weekend I wrote a Python view server for CouchDB. couchdb-python includes a view server but in the past I’ve heard complaints about performance (although none recently). In addition, the view server in couchdb-python only supports map and reduce, which is only about 1/5 of the current view server spec which includes handlers for update, show, list, filter, and validate which provide the groundwork for CouchDB as an application platform. As of Sunday my view server passes all of the current CouchDB spec and initial performance tests showed it faster than the JavaScript view server.

Below are the performance graphs for CouchDB trunk running on a CentOS virtual machine. I’m using Python 2.6 with the default stdlib json library. The spidermonkey core is 1.7 (I don’t know what the status of using 1.8 with CouchDB is but as we’ll see below, this won’t improve performance too much for these tests).

These graphs show view generation time for a given number of documents in a new database. The design doc I used had two views, one does emit(doc['type'],doc), the other emit(doc['_id'], 1).

The graphs support zooming, mouseover and all kinds of flot goodness

JavaScript is the yellow line. Python is the Blue line.

This is a test of moderately sized documents, what we normally expect the size of a job or build description. Each document is identical and fairly simple with a size of ~1,588 bytes.

These documents were incredibly large, they were taken from a full fennec mochitest run. Each document is identical and while large it consists mostly of small sized JSON objects inside a much larger JSON object coming in at ~139,096 bytes.

I had also intended to chart the reduce performance with a simple sum operation but all the results were sub-second regardless of the amount of documents I threw at it with Python being only a little faster than JavaScript.

The nearly identical reduce time tells me that the actual code processing time inside the view functions are hardly different which means that the large difference in performance during view generation is most likely due to JSON serialization time. This also explains why larger documents cause an even greater difference in performance between Python and JavaScript.

Improving Performance

The Python view server is already as optimized as I can imagine for processing time inside the views. Since CouchDB doesn’t provide a way for the view server to support it’s own concurrency we’ve basically hit the wall here on what Python can provide. If we increased the complexity of the view functions I think that Python would start to show better than Spidermonkey 1.7, but 1.8 with traceing enabled would likely bridge that gap, possibly even showing JavaScript faster than Python.

The big problem is JSON serialization. We can make Python faster by compiling simplejson with C speedups. But using the C based JSON parser in newer versions of Spidermonkey requires some other changes to CouchDB since there are differences in the encoding of undefined.

At the end of the day though, this all looks great. CouchDB trunk (pre-0.11) is going to run fast enough for what we need right now even on a virtual machine and if we start to see view generation bottlenecks on views that aren’t hit as often and have to update a large number of documents we can just move those views to Python and the performance should be back down to sub-second.

I’m in London for the next few days and would love to grab a drink with any community members be you Mozilla, CouchDB, Python, Windmill, JavaScript or just plain old coffee, whisky or beer geeks

I had some fun this weekend with Python, <video>, CouchDB and Brett Gaylor’s [RiP: A Remix Manifesto](http://www.ripremix.com). In just a few hours I was able to crank out a little [annotations remix](http://ripannotations.pythonesque.org) which allows anyone to add annotations to the film that are displayed as people view it.

I’m hosting it on my little mac mini (currently hidden in a data-center) so hopefully it doesn’t fall over pushing so much video

I’ve posted all the code up on [github](http://github.com/mikeal/ripannotations/tree/master). The more I use <video> and CouchDB the more excited I get about the future of web applications. This entire project was done in little chunks of spare time over the weekend and most of that was me messing around with styling. To get the data stored, queried, and displayed took less than 2 hours.

Hope you all enjoy the annotations remix and if you haven’t already go and pay what you want for a terrific copy of [RiP: A Remix Manifesto](http://www.ripremix.com). It’s worth it.

The Rails community seems to be on the verge of imploding, not necessarily because of a [particular event](http://www.ultrasaurus.com/sarahblog/2009/04/gender-and-sex-at-gogaruco/) but because of the [response by Rails leadership](http://afreshcup.com/2009/04/28/a-painful-decision/) to the event and their lack of meaningful reflection on why this happened.

This event has been well commented on, so I won’t go in to all the details, but unsurprisingly the best comments I’ve read so far have come from women, in particular [Audrey Eschright](http://dyepot-teapot.com/2009/04/25/dear-fellow-rubyists/). It’s good to see meaningful discussion in that community but I think the proposed solution of increased “professionalism” is counterproductive and generating an even more hostile response from [Rails leadership](http://loudthinking.com/posts/39-im-an-r-rated-individual).

Professionalism doesn’t have a good history of promoting equality. After segregation “professional” was one of many code words for “white male” and even after many years when non-white males began entering the work force the idea of professionalism served to strip people of their culture and identity to better fit in to a white male dominated environment. For those of us in open source, and especially those who have left traditional corporations, we identify “professionalism” as the way corporations turn people in to plastic dolls in suites that can talk to other plastic dolls from other corporations at trade shows without getting “off message”. Luckily, we aren’t talking about a corporation here, we’re talking about a community and community isn’t about professionals it’s about people.

I think people see professionalism as a solution to this problem because it provides a way for us to change people’s conduct that we disagree with without actually changing people or how they think they should conduct themselves in a community. The real problem here is that some **people** think it’s just fine to engage in public conduct that knowingly excludes people and doesn’t help include or encourage any new participation.

When news of this little outburst broke I have to say I wasn’t surprised that it happened in the Rails community. All tech communities have a low number of female participants, this is something nobody is great at yet, but for some time now Rails has been the worst offender. In the early days Rails was promoted by DHH in what, at the time, sounded like a great strategy: Attack dominant technologies (Java) and piss people off that use them long enough to check out your stuff. This strategy generated huge buzz and lead to one of the fastest adoptions of a new technology I’ve ever seen and it’s worth noting that the technology was actually so good that people could go from being a pissed off Java developer to a Rails developer in a relatively short amount of time.

But the biggest problem with DHH’s strategy was that it was hostile and it intentionally excluded a large number of people who were invested in the old technology and couldn’t get over being pissed off at DHH. The first thing DHH will say is “well, we don’t want or care about those people”, and he’s right, Rails doesn’t need those people. But when you, as the leader of a project, are hostile and partake in action that **intentionally excludes people**, regardless of who those people are, it creates an environment where others see no need to alter their behavior not to exclude people.

I’m really disheartened by [DHH's "I'm an R rated individual" post](http://www.loudthinking.com/posts/39-im-an-r-rated-individual), because it’s not that he’s wrong, he’s actually right, he should be able to be an individual in the community he’s built, but if he doesn’t intentionally curb some of his behavior he’s going to encourage exaggerations of the worst parts of his personality by the example he sets. It’s great that DHH loves Louis C.K and his dick jokes, but that doesn’t mean he would intentionally start a keynote with them, he knows better, and that’s what makes the difference and that’s what is so wrong with Matt’s talk, he knew before he went up there that this would make a lot of people uncomfortable and discourage participation from women and he did it anyway.

In order to lead an inclusive community you have to encourage participation and allow everyone to be individuals. If you treat everyone with respect, no matter who they are and no matter what they do, others will conduct themselves the same way. It’s always easier to respond to hostility with hostility but all you’re doing is creating an environment where hostility is the norm and it’s alright to exclude people.

The kind of communities we want are not impossible, in fact we get closer and closer every day. I’m lucky enough to be at [#moz09](http://search.twitter.com/search?q=%23moz09) this week, and I’m looking around at over 200 people from different cultures from around the world none of which seem to be trying to hide who they are or where they came from for some notion of “professionalism” and everyone treats each other with respect and every day we gather more participation from a broader set of individuals. We aren’t perfect, we’re never where we want to be, but we should be confident we’re going in the right direction.

I’ve been writing some automated tests for the new <video> work in Firefox 3.5 and it’s been incredibly fun. I spent some time in the trenches at Real Networks and dealt with accessing embedded video in the browser so I really appreciate how much nicer <video> is than embedded players like Real and Flash.

People talk a lot about embedded plugins being *”stuck in a box”* and while this is partially true I don’t think it’s descriptive enough. Embedded players like RealPlayer and Flash open up methods that can be called from javascript, so you do have some ways of interacting with the box, but it’s limited to somewhat rigid pre-defined interfaces. It’s also worth noting that using these interfaces just doesn’t **feel** like web development the way doing regular JavaScript and DOM feels. It’s not dynamic and you’re limited to only doing **what the plugin authors have already thought of**. The beautiful thing about the browser as a platform is that it enables usage far beyond what the original creators have intended.

Using <video> **feels** like the web. You resize it by setting size attributes the same way you would an interface node. You seek by setting the currentTime attribute, you can listen for the time changes to create your own interface by adding a listener for “timeupdate” events. You can write interfaces around <video> without switching contexts because it’s not a foreign technology jammed in to the browser. Here is some javascript that implements a few video controls.

var v = document.getElementById("v1");
document.getElementById("playDiv").onclick = function(e) {v.play()}
document.getElementById("pauseDiv").onclick = function(e) {v.pause()}
document.getElementById("reloadDiv").onclick = function(e) {v.load()}
document.getElementById("seekDiv").onclick = function(e) {
    v.currentTime = document.getElementById('seekField').value;
    }

Isn’t that beautiful! Maybe I’m overly excited because I’ve had to deal with this using older technologies but this just **feels** good, and if it feels good then I know people are going to get excited about it and build cool stuff.

One of the big questions we’ve been getting since we outlined how to write unittests in mozmill is how to serve out local files over HTTP. For quite a while I’ve had a port of mochitest’s js http server in the mozmill source tree but hadn’t written a nice API for you to be able to use it inside of your mozmill tests.

Ideally what I wanted was an API that worked a lot like how the dependencies work, which could serve out local files relative to the tests location on the filesystem. This way we don’t rely on any Python end to tell use what to serve out and can handle starting and stopping the server inside of the framework in the extension without any extra work from the test author. I finished the implementation a few weeks back but just documented it today.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var url = collector.addHttpResource('./files');
 
var setupModule = function(module) {
  module.controller = mozmill.getBrowserController();
}
 
var testFoo = function(){
  controller.open(url+'index.html');
}

This made it in before the last release so it’s available for use right away. Enjoy

One of the first pieces of feedback we got on MozMill was that there didn’t seem to be any way to click on system menus.

System menus are defined in XUL using a special set of elements. Those are then displayed differently on every operating system and as such we can’t use tools like the MozMill Inspector to get repeatable elements to click on.

After digging in to this problem a bit I figured out that the elements defining menus can be traversed and that they even respond properly to .click(). I then wrote thin API around the elements so that they can be easily looked up by name the way they are displayed in the menus. You can see the full documentation here.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var setupModule = function (module) {
  module.controller = mozmill.getBrowserController();
}
 
var testMenu = function () {
  // Example 1: Click menus without testing the click
  controller.menus.File["New Tab"].click();
  controller.menus.View.Sidebar.Bookmarks.click();
  controller.menus.View["Page Style"]["No Style"].click();
  // Example 2: Click menus and test the click
  controller.click(new elementslib.Elem(controller.menus.File["New Tab"]));
  controller.click(new elementslib.Elem(controller.menus.View.Sidebar.Bookmarks));
  controller.click(new elementslib.Elem(controller.menus.View["Page Style"]["No Style"]));
}

As you can see the API is a simple chain of objects by name cascading down the different sub menus until a menu element is reached. Attribute access is just plain old JavaScript so you only need to use ["name"] syntax when looking up menus with names that contain spaces. The API is most definitely case sensitive.

MozMill already has some big advantages over mochitest. For one, you don’t need a special test build, you can write and run tests against any build of a Mozilla Application. You can even write and run tests just using the extension and not worry about any Python stuff until you set it up in continuous integration.

MozMill already runs in “chrome”, or what Adam and I have been referring to as the “trusted” environment. All the tests that get run in MozMill are imported in this environment, which means you have access to pretty much everything in the Mozilla environment. We also have a dependency system and a setup/teardown system for tests.

All we were missing was a simple unittest API that can trigger failures without causing exceptions like the MozMill functional testing APIs. I looked at what others have done including dojo, mochikit, and jsunit.

Fortunately MozMill gets out of any test registration and staggering stuff, it already has a well defined way of defining tests and dependencies and a clear way to run tests for debugging and continuous integration. Of all the different jum (JavaScript Unittest Module) APIs I liked jsunit the most except for the incredibly stupid idea of passing the comment argument first instead of making it an optional last argument.

In the end MozMill got it’s own jum which I’ve just finished up documenting.

var jum = {}; Components.utils.import('resource://mozmill/modules/jum.js', jum);
 
var testAsserts = function() {
  jum.assert(true);
  jum.assertTrue(true);
  jum.assertFalse(false);
  jum.assertEquals('asdf', 'asdf');
  jum.assertNotEquals('asdf', 'fdsa');
  jum.assertNull(null);
  jum.assertNotNull(true);
  jum.assertUndefined({}.asdf);
  jum.assertNotUndefined('asdf');
  jum.assertNaN('a');
  jum.assertNotNaN(4);
  jum.pass();
}
 
var testNotAsserts = function() {
  // All of these calls should fail
  jum.assert(false);
  jum.assertTrue(false);
  jum.assertTrue('asf');
  jum.assertFalse(true);
  jum.assertFalse('asdf');
  jum.assertEquals('asdf', 'fdsa');
  jum.assertNotEquals('asdf', 'asdf');
  jum.assertNull(true);
  jum.assertNotNull(null);
  jum.assertUndefined('asdf');
  jum.assertNotUndefined({}.asdf);
  jum.assertNaN(4);
  jum.assertNotNaN('f');
  jum.fail();
}

All of these functions take an optional comment argument which can be any JSON serializable value (string, int, obj, array, null, bool).

What we’re missing right now is the http local file server. I’ve already ported mochikit’s web server to be a resource but I’m still working on a better API for using it. What we want is for tests to simply define local relative paths that should be mounted at different local url namespaces and the framework can handle starting and stopping the server if necessary. This API will be finished in the next few weeks and ready for MozMill 1.0 (which we’re looking at pushing out before January 1st).

Last week we finished up the 1.0RC2 release of MozMill.

The biggest new feature was is the Test Recorder. You can now hit the Record button and use the Application normally and create a test outline. I say “test outline” because it’s usually necessary after creating a recorded test to go in and fill it up with a few conditional wait calls, like waitForElement and waitForPageLoad. If you don’t the test runs too fast and ends up failing pretty quickly. We’re working on some future code to pick up things like new page loads so that we can add those waits in a recorded test but for now you’ll have to do them by hand.

We also have significantly improved some of the documentation.

• http://code.google.com/p/mozmill/wiki/MozMillTestBasics
• http://code.google.com/p/mozmill/wiki/CommandLine
• http://code.google.com/p/mozmill/wiki/MenuAPI

I’ll be writing some more blog entries this week on using MozMill for writing and running different kinds of tests.