Mozilla

I started working at Mozilla 2 years ago. I started the day after my employment at the Open Source Applications Foundation ended. At this point I already took for granted some of the best parts of working at Mozilla; working for a public benefit organization, spending 100% of my time working on Open Source, working with very smart people in the open (lists, IRC, etc.).

But Mozilla is even more than all that. Succeeding at Mozilla means something more than a pat on the back and a good end of the year review. When you succeed at Mozilla you impact one of the most important products on the internet. You reach hundreds of millions of users and contribute to keeping the web an open and free (as in speech) world. There is no other place in the world you can work where you can conceivably have this kind of impact.

Mozilla as an organization is truly unique. Last year was the hardest I’ve ever had, i suffered a huge loss in my personal life and Mozilla was as supportive during this time as any of my friends or family. There are a lot of places that let you put so much of yourself in to the organization to help it attain it’s goals but there are only a handful that are there to support you when you need it.

Relaxed Inc.

I started using CouchDB in 2008 after a great talk by Jan Lehnardt at OSCON. I started using it right away and over the next year it re-shaped how I think about web development and applications. In the last 6 months my group at Mozilla has become a heavy CouchDB user and not just because of my own interest but because CouchDB was the only solution for some of the harder problems we needed to solve with our results storage.

As I’ve used CouchDB more and more and become a part of the CouchDB community I’ve had the pleasure of knowing some of the core contributors, three of which have decided to found a new startup around CouchDB; Jan Lehnardt, J Chris Anderson, and the creator of CouchDB Damien Katz. Shorty after they received their funding they made an offer. It’s an amazing opportunity and while the decision to leave Mozilla is one of the hardest I’ve ever had to make I’m very excited about my future at Relaxed.

The Future

I’m really looking forward to working with everyone at Relaxed. It’s an exciting time and I’m not 100% sure yet which projects I currently work on that I will still have time to maintain. In the next week or so I’ll be doing a blog post on all the libraries I currently work on and maintain (it’s a long list) and what their status is moving forward. I still maintain code I wrote long before I worked at Mozilla and have every intention of continuing to work on some of the projects I started at Mozilla.

One thing is certain. I’m not the guy who figures out how to test the browser any more. Windmill and Mozmill are important projects that I have every intention of supporting by making time for code reviews and community support but I won’t be available to put time in to new feature work and refactoring like I have in the past. Luckily there are solid communities behind both of these projects and I’m confident that there are people who can continue to drive them in the future.

I don’t know what is going to happen next, all I know is that it should be fun, it won’t be like anything I’ve done before, and will certainly continue to include lots JavaScript and Python.

For everyone who depends on me and the code I’ve written over the last few years I’ll be sure to keep you all up to date. And one thing I can promise is that if you want to fix anything in one my projects, fork it on github and send me a pull request and I will always find time to look at it

I’ve had a great run with WordPress but I have a few ideas I want to experiment with and I also want to dogfood couchdb-pythonviews a little more.

This blog is hosted on Dreamhost. Dreamhost has been a great host for a low impact blog, the uptime hasn’t been 100% but all the maintenance has been easy and it’s also remained dirt cheap for the last few years.

I need to find a new hosting provider. I have one dedicated server but I don’t plan on running a blog there because that server is a little busy.

I need something cheap. I need root (or some kind of sudo jail) where I can run CouchDB and nginx and manage Python. Preferably Debian. Definitely Linux. Decent uptime.

I’ve considered EC2 but for a low impact site it’s actually quite expensive (~30 dollars a month before bandwidth) and the performance I’m told is about 5x slower than a Macbook.

Backups aren’t necessary since I have CouchDB replication for backing up all the important bits.

I’m open to any and all suggestions.

The most important thing for us to figure out was whether or not the CentOS virtual machine we’re currently running CouchDB on is going to be enough even in the short term. Until today we’ve been running 0.9 and have encountered performance problems.

Our main bottleneck is, and has always been, view generation and update performance. We tend to have medium to large size documents (jobs are relatively small but results from test runs can be incredibly large).

View generation of large documents has typically been our biggest issue which we have previously mitigated by refreshing all views after any large write but that isn’t going to work for the amount of results that we plan on pouring in to the new system.

Last weekend I wrote a Python view server for CouchDB. couchdb-python includes a view server but in the past I’ve heard complaints about performance (although none recently). In addition, the view server in couchdb-python only supports map and reduce, which is only about 1/5 of the current view server spec which includes handlers for update, show, list, filter, and validate which provide the groundwork for CouchDB as an application platform. As of Sunday my view server passes all of the current CouchDB spec and initial performance tests showed it faster than the JavaScript view server.

Below are the performance graphs for CouchDB trunk running on a CentOS virtual machine. I’m using Python 2.6 with the default stdlib json library. The spidermonkey core is 1.7 (I don’t know what the status of using 1.8 with CouchDB is but as we’ll see below, this won’t improve performance too much for these tests).

These graphs show view generation time for a given number of documents in a new database. The design doc I used had two views, one does emit(doc['type'],doc), the other emit(doc['_id'], 1).

The graphs support zooming, mouseover and all kinds of flot goodness

JavaScript is the yellow line. Python is the Blue line.

This is a test of moderately sized documents, what we normally expect the size of a job or build description. Each document is identical and fairly simple with a size of ~1,588 bytes.

These documents were incredibly large, they were taken from a full fennec mochitest run. Each document is identical and while large it consists mostly of small sized JSON objects inside a much larger JSON object coming in at ~139,096 bytes.

I had also intended to chart the reduce performance with a simple sum operation but all the results were sub-second regardless of the amount of documents I threw at it with Python being only a little faster than JavaScript.

The nearly identical reduce time tells me that the actual code processing time inside the view functions are hardly different which means that the large difference in performance during view generation is most likely due to JSON serialization time. This also explains why larger documents cause an even greater difference in performance between Python and JavaScript.

Improving Performance

The Python view server is already as optimized as I can imagine for processing time inside the views. Since CouchDB doesn’t provide a way for the view server to support it’s own concurrency we’ve basically hit the wall here on what Python can provide. If we increased the complexity of the view functions I think that Python would start to show better than Spidermonkey 1.7, but 1.8 with traceing enabled would likely bridge that gap, possibly even showing JavaScript faster than Python.

The big problem is JSON serialization. We can make Python faster by compiling simplejson with C speedups. But using the C based JSON parser in newer versions of Spidermonkey requires some other changes to CouchDB since there are differences in the encoding of undefined.

At the end of the day though, this all looks great. CouchDB trunk (pre-0.11) is going to run fast enough for what we need right now even on a virtual machine and if we start to see view generation bottlenecks on views that aren’t hit as often and have to update a large number of documents we can just move those views to Python and the performance should be back down to sub-second.

Volunteers are people who give their time/effort to an institution or group at some cost to them for little benefit in return. They usually do this as a labor of love, and therefor the argument can be made they receive some kind of emotional satisfaction from the exchange but it’s fundamentally categorized as a one way exchange where the volunteer is **giving** and the institution or group is **taking**.

Open source contributions are not market transactions. In a market there is a producer and a consumer, the transactions between them are to the benefit of either the producer or the consumer or both. Producer makes something, consumer evaluates the product and decides to give capital to the producer. Volunteer is a term used to describe an actor that is working for the benefit of the producer without reciprocal benefit to themselves to the extent that they benefit the producer.

Capital is the driving force in a market, it’s what enables the transactions. Workers are paid for contributions to product so that institutions can enable transactions where consumers are given a product in return for more capital. Capital is certainly a factor in open source, but it’s auxiliary. Capital may drive one side of the transaction, an actor paid to contribute to a product or consumption of a product used by the consumer to generate capital, but it does not drive each side of the transaction.

Contributors are not driven by a need to benefit a particular producer and are rarely driven by capital. In fact I can’t think of a way to describe open source contributions in terms of a market. In open source there is the **product**. The product exists almost as it’s own entity outside of the producers that created it or the consumers that use it. Because if it’s transparency and it’s ease of access and manipulation it cannot be viewed as a unit in a transaction within a market. Instead, all interactions need to be described in **relation to the product**. Contributors, institutions and individuals, that take part in production and those that take part in consumption take part in transactions with the **product**. This is an open source community, a group of actors taking part in transactions with a product.

The communities that thrive are the ones that remove barriers to these transactions and create tools that enable new transactions for more diverse contributions to the product. The transactions are not one-way, each transaction is two-way, benefiting both the product and the actor.

Rather than capital, mutual benefit seems to drive open source transactions. The product and the actor benefit from every transaction, with only a small portion of those transactions seeing capital as the benefit. The notion of a volunteer simply doesn’t exist in this model because there are rarely, if ever, transactions that only benefit a product at a cost to the actor. Actors **must** to be motivated and products are not “owned” in the traditional sense of ownership since their production is taken on by a community motivated by mutual benefit which tears down the relationship traditional market producers have with products.

Tools that are built to enable one sided transactions to a product usually fail because actors aren’t motivated by transactions that aren’t mutually beneficial.

A quick look at Firefox shows a very broad and diverse number of tools that enable mutually beneficial transactions. Although we often think about the new kinds of contributions that additions to Firefox itself will enable like [Personas](http://www.getpersonas.com/) or [Jetpack](https://jetpack.mozillalabs.com/) we also have a variety of tools that enable non-code contributions. Everything from that little button that reports a crash (benefits the product’s stability and improves your browser experience) to [SUMO](http://support.mozilla.com/en-US/kb/) (users seek resolution to support issues while providing the product with immeasurable usage feedback and bugs) are examples of tools that enable new transactions with the product that are mutually beneficial to the product and the actor.

One of my favorite things about working at Mozilla is being able to think about new kinds of contributions and how to enable them. There are few products with such a large and diverse ecosystem of users so the opportunities for new contribution is uniquely large so long as we create tools that are mutually beneficial.

Any new technology can have a side effect of making an entire industry irrelevant. Every time an industry is on the brink they try to plead for their own survival. Their tone and message is predictable, “you’ll miss our profession when it’s gone”. But of course, the profession never disappears only the institutions that used to enable it which are no longer relevant. These Institutions cannot survive in a new world so they scream to save their business model and claim that their profession is what is at risk.

I love journalism, and music, and I’m a total font geek but I just can’t stand newspapers, record labels, and these god forsaken font ["foundries"](http://blog.mozilla.com/rob-sayre/2009/07/19/broken-record/).

Let me get this straight. Your business model is to create fonts, **once**, and then license them on a per-use basis. I can’t be the first one to tell you **THAT ISN’T GONNA FUCKING WORK ANYMORE**.

@font-face has the side effect of invalidating your whole business model by driving demand for ubiquitous free and open fonts. I’m sorry. I wish it didn’t have to be this way. The world is cruel, and you’re fucked.

Print publications bought your fonts because they had fairly high production costs already and adding a little on top wasn’t going to break them. But this is the web, content is ubiquitous and free so the tools to create it have to be ubiquitous and free as well.

To steal a little from [Clay Shirky](http://www.shirky.com/), your industry is now going to suffer from mass-amatuerization. Font creation will not die with the foundries, there is going to be more font creation than there ever has been in human history, it’s just going to be open and free. The cost of producing a font is nearly zero. There is labor involved but the tools are ubiquitous and mostly free while distribution is a non-issue. Now that **anyone** can create a font and quality is determined solely by the creators talent, and we can actually **use** open fonts in the web content we create, it is safe to expect an explosion in the creation of fonts.

There is no technology that can save your business model because it pre-dates the web. The web changed the world you live in and you don’t get to change it back. If you don’t believe me you should have a chat with some former stock photographers.

Duke Nukem OS is a lightweight high performance open source operating system. The first release, codename “Forever”, is due out in late 2010. A promotional comic book will accompany the release and to insure it is available by the delivery date I plan to hire [Kevin Smith](http://en.wikipedia.org/wiki/Kevin_Smith) and [Alex Ross](http://en.wikipedia.org/wiki/Alex_Ross) to write and illustrate it.

Although no screenshots, specifications, or source code will be available until release consider this a call to action for the open source community to get involved in the project.

This new operating system will be built on the Linux kernel but I will be throwing out the bloated window managers Linux is currently known for and building a next generating interface. I’ll be using XULRunner, the Mozilla runtime used to create Firefox, as the basis for this window manager but the tools to develop applications are exclusively web3.0 “semantic web” standards. [RDFa](http://www.w3.org/TR/xhtml-rdfa-primer/), a next generation semantic markup language known for it’s simplicity and rapid rate of adoption, will be the primary language for building in this next generation operating system.

More updates in the future.

I’m putting together a new [Windmill talk](http://opensourcebridge.org/sessions/36) that tries to incorporate all the feedback we’ve received over the last year of speaking which I’ll be presenting on Thursday.

Mozilla is also a [sponsoring](http://opensourcebridge.org/sponsors/) the conference and there is going to be some great [Firefox related sprints in the hacker lounge](http://opensourcebridge.org/wiki/Hacker_Lounge). Dietrich is also giving what sounds like an awesome talk on extending Firefox called [Firefox Switchblade](http://opensourcebridge.org/sessions/251).

Hope to see you all there!

PS. I’ll also be at EuroPython and the Community Leadership Summit, more on those later

I’m hosting it on my little mac mini (currently hidden in a data-center) so hopefully it doesn’t fall over pushing so much video

I’ve posted all the code up on [github](http://github.com/mikeal/ripannotations/tree/master). The more I use <video> and CouchDB the more excited I get about the future of web applications. This entire project was done in little chunks of spare time over the weekend and most of that was me messing around with styling. To get the data stored, queried, and displayed took less than 2 hours.

Hope you all enjoy the annotations remix and if you haven’t already go and pay what you want for a terrific copy of [RiP: A Remix Manifesto](http://www.ripremix.com). It’s worth it.

This event has been well commented on, so I won’t go in to all the details, but unsurprisingly the best comments I’ve read so far have come from women, in particular [Audrey Eschright](http://dyepot-teapot.com/2009/04/25/dear-fellow-rubyists/). It’s good to see meaningful discussion in that community but I think the proposed solution of increased “professionalism” is counterproductive and generating an even more hostile response from [Rails leadership](http://loudthinking.com/posts/39-im-an-r-rated-individual).

Professionalism doesn’t have a good history of promoting equality. After segregation “professional” was one of many code words for “white male” and even after many years when non-white males began entering the work force the idea of professionalism served to strip people of their culture and identity to better fit in to a white male dominated environment. For those of us in open source, and especially those who have left traditional corporations, we identify “professionalism” as the way corporations turn people in to plastic dolls in suites that can talk to other plastic dolls from other corporations at trade shows without getting “off message”. Luckily, we aren’t talking about a corporation here, we’re talking about a community and community isn’t about professionals it’s about people.

I think people see professionalism as a solution to this problem because it provides a way for us to change people’s conduct that we disagree with without actually changing people or how they think they should conduct themselves in a community. The real problem here is that some **people** think it’s just fine to engage in public conduct that knowingly excludes people and doesn’t help include or encourage any new participation.

When news of this little outburst broke I have to say I wasn’t surprised that it happened in the Rails community. All tech communities have a low number of female participants, this is something nobody is great at yet, but for some time now Rails has been the worst offender. In the early days Rails was promoted by DHH in what, at the time, sounded like a great strategy: Attack dominant technologies (Java) and piss people off that use them long enough to check out your stuff. This strategy generated huge buzz and lead to one of the fastest adoptions of a new technology I’ve ever seen and it’s worth noting that the technology was actually so good that people could go from being a pissed off Java developer to a Rails developer in a relatively short amount of time.

But the biggest problem with DHH’s strategy was that it was hostile and it intentionally excluded a large number of people who were invested in the old technology and couldn’t get over being pissed off at DHH. The first thing DHH will say is “well, we don’t want or care about those people”, and he’s right, Rails doesn’t need those people. But when you, as the leader of a project, are hostile and partake in action that **intentionally excludes people**, regardless of who those people are, it creates an environment where others see no need to alter their behavior not to exclude people.

I’m really disheartened by [DHH's "I'm an R rated individual" post](http://www.loudthinking.com/posts/39-im-an-r-rated-individual), because it’s not that he’s wrong, he’s actually right, he should be able to be an individual in the community he’s built, but if he doesn’t intentionally curb some of his behavior he’s going to encourage exaggerations of the worst parts of his personality by the example he sets. It’s great that DHH loves Louis C.K and his dick jokes, but that doesn’t mean he would intentionally start a keynote with them, he knows better, and that’s what makes the difference and that’s what is so wrong with Matt’s talk, he knew before he went up there that this would make a lot of people uncomfortable and discourage participation from women and he did it anyway.

In order to lead an inclusive community you have to encourage participation and allow everyone to be individuals. If you treat everyone with respect, no matter who they are and no matter what they do, others will conduct themselves the same way. It’s always easier to respond to hostility with hostility but all you’re doing is creating an environment where hostility is the norm and it’s alright to exclude people.

The kind of communities we want are not impossible, in fact we get closer and closer every day. I’m lucky enough to be at [#moz09](http://search.twitter.com/search?q=%23moz09) this week, and I’m looking around at over 200 people from different cultures from around the world none of which seem to be trying to hide who they are or where they came from for some notion of “professionalism” and everyone treats each other with respect and every day we gather more participation from a broader set of individuals. We aren’t perfect, we’re never where we want to be, but we should be confident we’re going in the right direction.

People talk a lot about embedded plugins being *”stuck in a box”* and while this is partially true I don’t think it’s descriptive enough. Embedded players like RealPlayer and Flash open up methods that can be called from javascript, so you do have some ways of interacting with the box, but it’s limited to somewhat rigid pre-defined interfaces. It’s also worth noting that using these interfaces just doesn’t **feel** like web development the way doing regular JavaScript and DOM feels. It’s not dynamic and you’re limited to only doing **what the plugin authors have already thought of**. The beautiful thing about the browser as a platform is that it enables usage far beyond what the original creators have intended.

Using <video> **feels** like the web. You resize it by setting size attributes the same way you would an interface node. You seek by setting the currentTime attribute, you can listen for the time changes to create your own interface by adding a listener for “timeupdate” events. You can write interfaces around <video> without switching contexts because it’s not a foreign technology jammed in to the browser. Here is some javascript that implements a few video controls.

var v = document.getElementById("v1");
document.getElementById("playDiv").onclick = function(e) {v.play()}
document.getElementById("pauseDiv").onclick = function(e) {v.pause()}
document.getElementById("reloadDiv").onclick = function(e) {v.load()}
document.getElementById("seekDiv").onclick = function(e) {
    v.currentTime = document.getElementById('seekField').value;
    }

Isn’t that beautiful! Maybe I’m overly excited because I’ve had to deal with this using older technologies but this just **feels** good, and if it feels good then I know people are going to get excited about it and build cool stuff.

This kind of thing continually reinforces something I’ve thought about a lot since the App store was released, which sounds horrible to say but it might be true: Apple is creating an ecosystem of the kind of customers I don’t want. With the ridiculous approval process leaving bugfixes to take over a week to show up, with prices being driven down to nothing by farting apps… it just feels hostile to me. While I have plenty of great customers who have been raving about the app, all it takes is one little issue and it all comes crashing down.

It’s no longer acceptable for consumer applications to “blame the user”. Sure, many people still do it, but if a blog post like this just blamed individual users for their complaints without this closing paragraph it wouldn’t be gaining so much steam and I probably never would have read about it. Instead Garrett blames the *community* of users, and by extension Apple for process that encourages this kind of community.

The first rule of community leadership in open source is simple; **”Don’t be an asshole.”**. While simple, sometimes it’s the hardest rule to follow. Many people in the community can be difficult, some of them downright hostile, but you can’t be hostile back and you can’t be an asshole. When you’re hostile or an asshole in your community you give license to the community to do the same. You cultivate the behavior that antagonized you in the first place. This is the rule when trying to lead contributors and I have to assume it’s the rule when trying to lead users and posts like this violate the rule by blaming users for their own expectations and then dismissing them altogether.

He tries to deflect attention to Apple since they are the gate keeper. Dismissing the bulk of your user community due to the gate keeper is logic that is so convenient you know it has to be flawed. The truth is **most** platforms have some kind of gate keeper, this is the world that we live in, but your complaints about your users have nothing to do with Apple.

iFart is 1 dollar but you’re giving it way too much credit if you think that’s why iPhone users have lower price expectations. You know what else is 1 dollar, Ocarina. Applications and games like Ocarina are larger triumphs of creativity than they are of engineering time. Not to say they are easy, or simple, but they are priced cheaply because the creators are confident in their product and because they didn’t spend a year developing it. This is where iPhone users have cultivated their expectations about interface, graceful failure, and low prices, because of your competitors not because of the gate keeper.

But even if all the user community’s expectations and demeanor is Apple’s fault, even if your users are cheap and vicious and given way too big a soap box, it’s still your fault.

As an example. Firefox get’s requests for features from Windows users to make certain things be “more like Windows”. If you look at the Firefox interface over 3 operating systems you’ll notice that the preferences menu item moves around based on which platform it’s on. The truth is, if you designed an interface from the ground up in some ideal world there would be one place that is best to put the preferences item, but that’s not the world we live in, we live in a world of expectations from user communities. The easiest thing to do is to say “we don’t need those users” or “those users are stupid and don’t understand proper design” but it’s not the **right** thing to do because it’s not what **they** want.

What your users are really saying with those complaints is that this failure frustrated them to the point of opening up iTunes and writing a complaint, not just that something broke. You could have failed more gracefully, you could have presented them with a button that opened a support web page with more detailed information and assurances that it was being resolved. Then after some time the bad reviews would get swallowed up by the positive ones you mentioned. But now you don’t have to do anything because you’ve convinced yourself that you did nothing wrong and it’s just a bunch of users you don’t want complaining about something they are too dumb to understand.

I hope some day Garrett finds his perfect user community with expectations that match his own. I’ll stick to working on products for everyone else, listen to their feedback seriously and try to improve them.

The biggest new features are:

* [django management command](http://trac.getwindmill.com/wiki/WindmillAndDjango) for running windmill tests ([Jacob](http://jacobian.org/) said the existing django support wasn’t good enough and I agreed so I wrote this during the PyCon Sprints)
* new [nose plugin](http://trac.getwindmill.com/wiki/BookChapter-5-RunningTests#RunningTestsfromNose)
* cygwin support contributed by [Simon Law](http://sfllaw.livejournal.com/) (he went and wrote an implemenation of [winreg for cygwin](http://pypi.python.org/pypi/cygwinreg) to get this to work).

There were also some really good bug fixes that landed:

* much better unicode handling and serialization (adam)
* fix for POST to foreign domains ([Anthony Lenton](http://anthony.lenton.com.ar))
* continued improvements to click simulation (adam)

The release is [up on PyPI](http://cheeseshop.python.org/pypi/windmill) and you can install/update with:

$ easy_install -U windmill

For anyone interesting in working **on** windmill we’re having a Sprint in #windmill on irc.freenode.net tomorrow April 8th, 2008 for pretty much all day. We’re going to be improving the unittests for Windmill itself.

The next planned major release will be 1.2 which will include the much anticipated SSL support, courtesy of some great work being done by Anthony.

The latest RCs have seen bigger improvements than we thought would make it. A new wave of contributions has given us some great new features in Django support and test serialization and [Adam](http://www.adamchristian.com) has done some great UI work as well.

Big thanks to everyone who contributed to the release and I hope to see many of you at PyCon. Adam is doing a [talk on windmill](http://us.pycon.org/2009/conference/schedule/event/9/) which I’ll be helping out with and Adam and I will also be on a [functional testing tools panel](http://us.pycon.org/2009/conference/schedule/event/88/) which should be a lot of fun. And if you’re staying around for the sprint days of the conference we’ll be leading a [windmill sprint](http://us.pycon.org/2009/sprints/projects/windmill/).

Feeds are an obvious and helpful tool in helping *normal* people automate the tasks of keeping up with information streams. Many users are finding ways to use feeds to **follow** a particular site or person and using new tools and services that work better for them than the traditional big RSS Readers. Personally, I think Firefox3 has the best workflows for integrating with any of these tools. Let’s work through the subscribe workflow for my blog.

* Click on the feed button.

* Select or configure your client and click subscribe.

This is a very simple workflow and it makes web based tools first class citizens where previous versions of Firefox and other browsers only support desktop client subscriptions. You don’t need to make a lot of decisions or know anything about the feed technology or protocols behind the scenes.

But the workflow for my blog is an ideal scenario where only one rel=”alternate” feed has been parsed from the page. This workflow becomes more complicated when multiple rel=”alternate” links are in the page.

Is this really necessary? 90% of the people who click this want to follow new posts and nothing else. More importantly what feed would **you** like people to subscribe to? You can make a decision here to push people in the direction you would like and somewhere else on the page you can offer them these more complicated options if they are seeking out alternate subscriptions.

This is terrible. You’re basically undercutting all the work browser vendors have done to let users start to integrate feeds in to their workflows without understanding feed protocols.

If we want to encourage adoption of these technologies we have to help eliminate this kind of extra confusion from basic user workflows. Feeds aren’t just being used in traditionally feed readers now, those readers de-contextualize the information so that power users can burn through information quicker. New tools are enabling users to use feeds to “follow” a site, they way they might follow someone on twitter. These unnecessary subscription options discourage that kind of adoption because it presents choices that fall outside of the bounds of that workflow.

The UI has been totally re-worked by Adam based mainly of design session feedback. We ditched the old xul UI and are just using html loaded via chrome url, this made development an order of magnitude easier and allowed us to use a great syntax highlighting library for the editor that we could never quite get working in xul.

The release also has lots of new features like automatically binding the mozmill window to the side of a test window and editing of multiple tests at once. A whole slew of bugfixes made it in, mostly logged by Henrik, including fixes to mulit-window recording.

At the suggestion of Dan Mosedale we also added new asynchronous test support for more unittesting capabilities. Documentation is still on it’s way but you can refer to the mozmill unittest for now.

MozMill can be downloaded a few ways, the extension is available on google code downloads or the Python package can be installed via PyPI

$ easy_install -U mozmill

Clint also just finished up a big round of new end user docs and API documentation cleanup and reformatting on QMO.

I used to be really in to public bookmark services like delicious and magnolia. Then Firefox3 was released and the new bookmarks UI combined with the awesomebar was just much easier to use than any of the sharing extensions.

I never took the time to sit down and write a simple sync/push extension because I just figured that someone else would do it

It’s been a year now, it’s time for PushMarks. https://addons.mozilla.org/en-US/firefox/addon/10806/

PushMarks is a simple firefox extension that pushes new bookmarks to your delicious account (more services coming). You can also initiate a “sync” which will go through all your local bookmarks and push them to delicious and also make sure any bookmarks you have on delicious are in your local bookmarks.

If you do a big sync, please be patient. It takes a while for all the bookmarks to get up to delicious because I have to throttle the connections for fear of being locked out of the delicious API.

Of course it’s all open source and up on github http://github.com/mikeal/pushmarks/tree/master .

The lesson starts next monday and is all being organized through a Google Group. It is open to anyone who would like to participate so if you want to learn Python come on over.

I posted about about the project on QMO and you can see the discussion surrounding the lesson outline on the Google Group.

I’m the co-author of two tools that overlap with FireUnit, Windmill and MozMill. I say overlap and not “compete” because I think the tools differ enough in their goals that they don’t directly compete.

FireUnit has some limitations. It’s an extension to Firebug which means it will only run on Firefox. It’s focused on providing a simple API for people who are competent JavaScript programmers. I think it’s safe to say that there is an emphasis on unittesting and not functional testing, and certainly an emphasis on testing content and not chrome.

Something really great happens when you create a tool that embraces limitations instead of trying to be too much to too many people. You tend to be able to create better smaller purpose tools for a selection of people that have been beating themselves up trying to make tools with less limitations work for a smaller use case.

The API

// Simple true-like/false-like testing
fireunit.ok( true, "I'm going to pass!" );
fireunit.ok( false, "I'm going to fail!" );
 
// Compare two strings - shows a diff of the
// results if they're different
fireunit.compare(
  "The lazy fox jumped over the log.",
  "The lazy brown fox jumped the log.",
  "Are these two strings the same?"
);
 
// Compare a string using a regular expression
fireunit.reCompare(
  /The .* fox jumped the log./,
  "The lazy brown fox jumped the log.",
  "Compare a string using a RegExp."
);
 
// Display the total results
fireunit.testDone();

This is the example from John Resig’s post about FireUnit. I think it illustrates how awesome the API is. I’ve always been a huge fan of this “ok” style API instead of the larger jUnit style API I keep being talked in to implementing. Clearly FireUnit assumes it’s audience is JavaScript programmers that can write normal expressions to handle their pass/fail logic without the hand holding of a bigger API.

The beauty of the comparison API you really don’t appreciate until you look at the UI.

Debugging

I’m most critical of tools that don’t provide sufficient debugging tools. The test debugging environment is even more important than the test authoring environment in the long run.

There is no better tool for debugging the web than Firebug. A big part of Windmill’s debugging strategy is it’s integration of Firebug and Firebug Lite because there is just no better tool in the world. Firebug ROCKS!

FireUnit is built as an extension of Firebug, it’s integrated in a beautiful and intuitive way so they’ve totally dodged the need for any additional debugging work outside of creating a good UI for the test bits. I’m calling this out because I think as time goes on people aren’t going to talk about the debugging experience in FireUnit because they are so used to already using Firebug for debugging their regular development and won’t recognize how good they have it debugging their tests.

The UI

Another benefit of embracing limitations is that you trim down your feature set to only what is actually necessary. The upside to having a simple API is that you reduce the complexity of test data you need to represent in the UI. With all that going for it I expected the FireUnit UI to be pretty good but I was still blown away.

Look at that comparison UI!

The Bad

Within the limitations described above FireUnit is great but as soon as you start to cross them you’ll be disappointed.

Although they have some helper APIs for doing some functional testing it’s not a good functional testing tool. The tests are async which is a real pain for functional testing which is incredibly synchronous. But I think if you want to do functional testing multiple browser support is a big concern and will already preclude you from using FireUnit.

Although you can test Chrome (Mozilla Chrome not Google Chrome) I wouldn’t venture outside of unittests. If you need to start getting at anonymous nodes and simulating events on them you’re gonna see your code get about 4 times larger than a typical mozmill test.

Right now there doesn’t seem to be a way to run FireUnit tests in continuous integration (tinderbox, buildbot, hudson, etc). But this could easily be overcome using jsbridge and I’m happy to point the FireUnit guys in the right direction

At the end of the day I think there is a huge need for a tool like FireUnit that just does what it does well and doesn’t try to be too much to too many people. A tool like this is guaranteed to make it’s users very happy and I think those users are currently unhappy with the existing set of automation tools that are trying to fill a larger feature set.

Ideally what I wanted was an API that worked a lot like how the dependencies work, which could serve out local files relative to the tests location on the filesystem. This way we don’t rely on any Python end to tell use what to serve out and can handle starting and stopping the server inside of the framework in the extension without any extra work from the test author. I finished the implementation a few weeks back but just documented it today.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var url = collector.addHttpResource('./files');
 
var setupModule = function(module) {
  module.controller = mozmill.getBrowserController();
}
 
var testFoo = function(){
  controller.open(url+'index.html');
}

This made it in before the last release so it’s available for use right away. Enjoy

import simplejson
decoder = simplejson.JSONDecoder()
 
class StreamReader(object):
    sbuffer = ''
    def process_read(self, data):
        """Parse out json objects and fire callbacks."""
        self.sbuffer += data
        self.parsing = True
        while self.parsing:
            # Remove erroneus data in front of callback object
            index = self.sbuffer.find('{')
            if index is not -1 and index is not 0:
                self.sbuffer = self.sbuffer[index:]
            # Try to get a json object from the data stream
            try:
                obj, index = decoder.raw_decode(self.sbuffer)
            except Exception, e:
                self.parsing = False
            # If we got an object fire the callback infra
            if self.parsing:
                self.fire_callbacks(obj)
                self.sbuffer = self.sbuffer[index:]

The process_read function gets called every time a block of data gets read from the stream so it needs to be able to handle multiple JSON objects in a single read, objects that span multiple reads, and junk (usually spaces and return carriers) showing up between JSON objects.

If you wanna see this code in the wild you can check out the jsbridge source.

System menus are defined in XUL using a special set of elements. Those are then displayed differently on every operating system and as such we can’t use tools like the MozMill Inspector to get repeatable elements to click on.

After digging in to this problem a bit I figured out that the elements defining menus can be traversed and that they even respond properly to .click(). I then wrote thin API around the elements so that they can be easily looked up by name the way they are displayed in the menus. You can see the full documentation here.

var mozmill = {}; 
Components.utils.import('resource://mozmill/modules/mozmill.js', mozmill);
var elementslib = {}; 
Components.utils.import('resource://mozmill/modules/elementslib.js', elementslib);
 
var setupModule = function (module) {
  module.controller = mozmill.getBrowserController();
}
 
var testMenu = function () {
  // Example 1: Click menus without testing the click
  controller.menus.File["New Tab"].click();
  controller.menus.View.Sidebar.Bookmarks.click();
  controller.menus.View["Page Style"]["No Style"].click();
  // Example 2: Click menus and test the click
  controller.click(new elementslib.Elem(controller.menus.File["New Tab"]));
  controller.click(new elementslib.Elem(controller.menus.View.Sidebar.Bookmarks));
  controller.click(new elementslib.Elem(controller.menus.View["Page Style"]["No Style"]));
}

As you can see the API is a simple chain of objects by name cascading down the different sub menus until a menu element is reached. Attribute access is just plain old JavaScript so you only need to use ["name"] syntax when looking up menus with names that contain spaces. The API is most definitely case sensitive.

MozMill already runs in “chrome”, or what Adam and I have been referring to as the “trusted” environment. All the tests that get run in MozMill are imported in this environment, which means you have access to pretty much everything in the Mozilla environment. We also have a dependency system and a setup/teardown system for tests.

All we were missing was a simple unittest API that can trigger failures without causing exceptions like the MozMill functional testing APIs. I looked at what others have done including dojo, mochikit, and jsunit.

Fortunately MozMill gets out of any test registration and staggering stuff, it already has a well defined way of defining tests and dependencies and a clear way to run tests for debugging and continuous integration. Of all the different jum (JavaScript Unittest Module) APIs I liked jsunit the most except for the incredibly stupid idea of passing the comment argument first instead of making it an optional last argument.

In the end MozMill got it’s own jum which I’ve just finished up documenting.

var jum = {}; Components.utils.import('resource://mozmill/modules/jum.js', jum);
 
var testAsserts = function() {
  jum.assert(true);
  jum.assertTrue(true);
  jum.assertFalse(false);
  jum.assertEquals('asdf', 'asdf');
  jum.assertNotEquals('asdf', 'fdsa');
  jum.assertNull(null);
  jum.assertNotNull(true);
  jum.assertUndefined({}.asdf);
  jum.assertNotUndefined('asdf');
  jum.assertNaN('a');
  jum.assertNotNaN(4);
  jum.pass();
}
 
var testNotAsserts = function() {
  // All of these calls should fail
  jum.assert(false);
  jum.assertTrue(false);
  jum.assertTrue('asf');
  jum.assertFalse(true);
  jum.assertFalse('asdf');
  jum.assertEquals('asdf', 'fdsa');
  jum.assertNotEquals('asdf', 'asdf');
  jum.assertNull(true);
  jum.assertNotNull(null);
  jum.assertUndefined('asdf');
  jum.assertNotUndefined({}.asdf);
  jum.assertNaN(4);
  jum.assertNotNaN('f');
  jum.fail();
}

All of these functions take an optional comment argument which can be any JSON serializable value (string, int, obj, array, null, bool).

What we’re missing right now is the http local file server. I’ve already ported mochikit’s web server to be a resource but I’m still working on a better API for using it. What we want is for tests to simply define local relative paths that should be mounted at different local url namespaces and the framework can handle starting and stopping the server if necessary. This API will be finished in the next few weeks and ready for MozMill 1.0 (which we’re looking at pushing out before January 1st).

The biggest new feature was is the Test Recorder. You can now hit the Record button and use the Application normally and create a test outline. I say “test outline” because it’s usually necessary after creating a recorded test to go in and fill it up with a few conditional wait calls, like waitForElement and waitForPageLoad. If you don’t the test runs too fast and ends up failing pretty quickly. We’re working on some future code to pick up things like new page loads so that we can add those waits in a recorded test but for now you’ll have to do them by hand.

We also have significantly improved some of the documentation.

• http://code.google.com/p/mozmill/wiki/MozMillTestBasics
• http://code.google.com/p/mozmill/wiki/CommandLine
• http://code.google.com/p/mozmill/wiki/MenuAPI

I’ll be writing some more blog entries this week on using MozMill for writing and running different kinds of tests.

jsbridge is a Python to JavaScript translation interface for Mozilla applications. It’s implemented as an extension and can be dynamically installed and launched in a new “clean” profile using mozrunner.

You can call functions and set variables in any window in any scope from Python and those objects are returned and represented as Python objects (which are just classes that inherit from Python base types). There are obvious limitations for assignment and representations in Python, you can really only send objects over the bridge that can be JSON serialized, but it does support assignments by reference so you can move around and manipulate anything that’s already there.

But the most interesting feature is the event system. The event system is available as a resource from inside the Mozilla environment (in other words you can import it from your Firefox extension). Events fired on the JavaScript end are serialized as JSON and sent to the Python side. This means you can add listeners in Python to events fired from the Mozilla JavaScript environment.

This is a big deal. It means we can write significantly cleaner tools. We can now create tools solely as Firefox extensions that don’t require any special Python code to run, but have code that checks if jsbridge is installed and send pertinent events over the bridge for continuous integration. No more printing to console and parsing in Python, no more outputting to log files and parsing them, no more scripts calling scripts and OH MAN MY HEAD HURTS!

The documentation has some great examples.

For a little snippet of the events system in the wild here is some new mozmill Python code that sends all the events to the Python logger.

import logging
logger = logging.getLogger("mozmill")
 
from jsbridge import events
 
class LoggerListener(object):
    cases = {
        'mozmill.pass':   lambda obj: logger.debug('Test Pass: '+repr(obj)),
        'mozmill.fail':   lambda obj: logger.error('Test Failure: '+repr(obj)),
    }
 
    class default(object):
        def __init__(self, eName): self.eName = eName
        def __call__(self, obj): logger.info(self.eName+' :: '+repr(obj))
 
    def __call__(self, eName, obj):
        if self.cases.has_key(eName):
            self.cases[eName](obj)
        else:
            self.cases[eName] = self.default(eName)
            self.cases[eName](obj)
 
events.add_global_listener(LoggerListener())
logging.basicConfig(level=logging.ERROR)

Ubiquity rocks! A month or so ago I completely removed the search box from Firefox and started solely using ubiquity for searches.

The one thing that I really missed was a python doc search that I used to have for the search box. For a while I just used the awesome bar to perform google and history searches for python docs but that basically broke when Python 2.6 was released since all the old docs.python.org urls that google returns break because 2.6 has completely new docs provided by Sphinx.

So today I took a break from my “normal” work to write a ubiquity command to bring up the search page from docs.python.org for your input text. You can see the source here and you should be able to install it by viewing this blog post.

In the future I’d like to do better previews of the searches but I can’t seem to find any web API for searching docs.python.org.

You can install it from AMO.

We’ve been gathering feedback from our Beta1 release and pushed out another incremental beta release before the big push for MozMill 1.0 RC1.

One important thing we did was improve the inspector workflow. It’s much smother clicking on an element and coming back to the Inspector. The inspector also gives you much more information now, including what document you should be passing the Elem you create and what method you should use to create a controller for the window you’re testing.

Lot’s of bug fixes, special thanks to Tony, Tracy and a big hug to Farhad for feature suggestions and filing bugs against beta1.

This release does not include the new framework that hooks up to jsbridge, and therefor isn’t ready for full continuous integration, that’s now in trunk getting worked on for the RC1 release.

Seriously Faster

On Wednesday Adam messaged me and said that the windmill startup time was too slow. He was right, we’ve know about this for a while but hadn’t put a lot of serious thought in to how we could reduce it.
The issue here was was that we have about 50 JavaScript files that need to get loaded for windmill to start.

Enter windmill-compressor, a new url namespace we added that concats all the js windmill needs in to one file and minifies it. We do this dynamically when windmill starts up, because adding a “build” step would just be too….. Java.

That reduced the startup time from 5-10 seconds to around 2 seconds. But that wasn’t good enough. We saw that most of that startup time was blocking waiting for the compressor to finish, so I threaded it when we start the windmill service and it does the compression while we wait for the browser to start up.

In all, windmill startup times are about 10x faster than 0.9!

Adam also decided to make our page load wait code a bit more aggressive which made not only startup times faster but all of our tests!

Native JavaScript Test Framework

We tied up the loose ends on the JavaScript testing framework and it can now shutdown all of windmill in it’s own teardown, hello continuous integration for native JavaScript tests

This was the last thing holding us back from promoting this along side the Python test authoring library.

We Love Firebug

In 0.9 we introduced Firebug Lite integration for windmill on all browsers. But when you’re using Firefox you probably still want to use the full Firebug extension, which is easy enough to integrate since we use mozrunner for Firefox launching.

Windmill now has a “firebug” command line argument that installs the full Firebug
extension when launching Firefox.

We hit a major milestone last week when I finished implementation of the “Lookup” object, which uses a custom expression syntax that can define lookup for any element, even anonymous elements created by XBL that aren’t in the DOM. We also made some crucial additions to the Inspector UI;  generation of expressions for the new Lookup object, better XPath, validation for all the expressions the inspector generates, and enhanced fallback logic. After about an hour of testing I couldn’t find a single element the Inspector couldn’t generate an expression for testing, which means we can simulate all events on all elements in any Mozilla application.

Late last week we finally pushed our Beta release, and now the extension is up on AMO and ready for anyone interested to install and start poking around.

Clint has also written some great documentation; a great tutorial and a fairly complete API reference.

For some time now we’ve always had a shell in Windmill for debugging but it was never as nice as the Firebug shell and maintaining that code was kind of a pain. In the recent 0.8.2 release of Windmill we dropped our old shell code and fully integrated Firebug Lite so that whenever you need to debug a test in any browser you can open Firebug Lite in the target application by clicking a single link in the Windmill IDE.

I can’t even begin to explain how much easier this is going to make writing and debugging tests in Windmill.

The task of just starting and stopping the Mozilla Applications is harder than it would seem, particularly when it comes to stopping. There’s also a bunch of other things you wanna do to Mozilla Applications when you’re launching them; set preferences, install plugins, build new clean profiles and use them, add command line arguments to the launch, etc.

mozrunner is a Python library that is independent of any particular Mozilla application source repository and is hosted at http://code.google.com/p/mozrunner . The library takes on the task of starting, configuring and stopping Mozilla Applications (only tested against Firefox right now but it should work with Thunderbird, Songbird, etc. and patches are welcome). We used this library when developing the new Gristmill suite of test automation tools and it’s been wonderful to work with.

The current documentation is over on the google code wiki; http://code.google.com/p/mozrunner/wiki/UsingMozRunner , there is a lot more that you can do with mozrunner that still isn’t up in the documentation and I plan on just adding it as people ask me how to do things.

If you have Python and setuptools installed you can just install it via easy_install.

$ easy_install mozrunner

It should find your local Firefox install by default, so just running mozrunner from the command line should launch Firefox. You can also check out the –help for some simple configuration and testing of the launcher script.

$ mozrunner --help

But of course, you shouldn’t be executing the script from within Python, and you’ll find some information in the docs about how to simply import the mozrunner Python library and embed the launching and configuration in to your tool.