It’s no secret that shortly after I started at Mozilla I began working on an ambitious automation project. The project, called MozMill, is a tool that can be used for full UI automation of all Mozilla applications. In time this means not just Firefox but Thunderbird, Flock, Songbird, etc.

We hit a major milestone last week when I finished implementation of the “Lookup” object, which uses a custom expression syntax that can define lookup for any element, even anonymous elements created by XBL that aren’t in the DOM. We also made some crucial additions to the Inspector UI;  generation of expressions for the new Lookup object, better XPath, validation for all the expressions the inspector generates, and enhanced fallback logic. After about an hour of testing I couldn’t find a single element the Inspector couldn’t generate an expression for testing, which means we can simulate all events on all elements in any Mozilla application.

Late last week we finally pushed our Beta release, and now the extension is up on AMO and ready for anyone interested to install and start poking around.

Clint has also written some great documentation; a great tutorial and a fairly complete API reference.

I’ve been a bad open source developer. I’ve had some really useful code sitting on my laptop for nearly 8 months now without publishing it. It’s a pretty simple mako plugin for django that makes using mako templates as simple and integrated as django’s regular template system.

A week or so ago I finally got off my ass and put it up, but it took me until today to write proper documentation.

The google code project is here, http://code.google.com/p/django-mako/ .

The documentation is here, http://code.google.com/p/django-mako/wiki/Usage .

Basically all you need to do is enable the middleware and all the Mako lookup stuff is handled for you and you can use new shortcut functions that mimic django’s regular render_to_string and render_to_response functions.

During OSCON this year I attended Jan Lehnardt’s talk on CouchDB. Somewhere in the middle of the talk I realized how much time I spend “dealing” with RDBMS and what a pain in the ass it is.

That weekend, in between OSCON and Mozilla Summit, I started playing with CouchDB and totally fell in love. Certainly my favorite part of CouchDB is that I no longer feel like I need huge abstractions between me and the datastore. The datastore is simple enough to get your head around and it comes with a beautiful web interface and it makes me feel like I don’t need a heavy duty ORM that handles everything for me, I can have a more intimate relationship with my datastore.

At some point I decided to seek out the Python library for interfacing with CouchDB, couchdb-python, and wasn’t too pleased. It’s a big library and a lot of great work was put in to it, but it just wasn’t what I wanted at all.

It feels like an ORM for RDBMS in that it does a lot of work to be a complete abstraction between you and CouchDB, which I just don’t want. The “schema” stuff, which is it’s mapper classes, feels a lot like RDBMS ORM. One of the great things about CouchDB is that there is no “schema” and that all your documents can fully mutate. This is how I want to define most of my “models”.

class User(Model): pass

So on the plane to Mozilla Summit I wrote a CouchDB library and spent free minutes here and there tweaking it. It’s incredibly small, about 243 lines with whitespace and comments, which is a testament to how much easier a REST API is to program for than SQL. You can create “Restrictions” which can do type checking and other validation before an attribute is set for a given property, and you can also use a Restriction to define custom marshalling before it hits the JSON parser. But all Models are fully mutatable and no validation is done at read time from the datastore, this way the ORM won’t ever get in your way as you change you data definitions over time in your application.

The whole thing is hosted on Google Code at http://code.google.com/p/pouch/ .

The docs right now are short and sweet http://code.google.com/p/pouch/wiki/Usage .

There are some features I’d like to add, it currently uses httplib and I’d like the option of using httplib2 so that you could do caching pretty seamlessly. Any other features or patches are welcome. I’ll be pushing it to the cheeseshop later today when I get a chance. Here’s a taste;

import pouch
 
pouch.set_globals('http://127.0.0.1:5984', 'test')
 
class TestModel(pouch.Model): pass
 
class TestModelTwo(pouch.Model):
   unicode_test = pouch.Unicode()
   int_test = pouch.Int()
   float_test = pouch.Float()
   list_test = pouch.List()
   dict_test = pouch.Dict()
   bool_test = pouch.Bool()
 
test = TestModel(anything="yes")
test.another = "ok"
test.save()
 
test == TestModel.get(test.id)
 
result = pouch.GLOBAL_DB.views.company.all() # Retrieves the response for view I have named company.

I never thought this would be the kind of blog that talks about politics too much or posts funny videos, but this must be shared.

Last Wednesday Adam and I had our work laptops stolen out of the back of my car. It sucked, and I was stupid about information security so all my data was open to whomever decided to steal it.

For a variety of reasons I decided to buy a MacBook Air out of pocket, which I intend to do all my work on. I shelled out the extra money for the SSD drive after looking at the read speed numbers.

After a few days I’ve got everything up and running, and all the files I need restored from a TimeMachine backup of my old laptop. I have to say, this computer is fast enough to do the vast majority of tasks most people do, including developers.

Even though I love this machine and am perfectly able to get everything I need done, I do think there are some work situations in which you just cannot use this machine on a daily basis.

• You work on a large compiled application
• You write Java
• You spend a sizable portion of your time in VMWare

The write speeds are pretty slow, and it maxes out at the OOTB 2GB of memory. BUT, the read speeds are astonishing. In fact I’m finding most of what I do on this laptop to be FASTER than my MacBook Pro.

You read from your drive a lot more than you write to it, and seeing 18x improvements in read performance has some stunning consequences. Firefox is noticeably more responsive, bootup times are really fast, Mail.app is usable again, and switching applications is instantaneous.

Not to mention, this machine is BEAUTIFUL. I never thought I’d be staring over at a MacBook Pro and thinking it looked “dated”. I really like this keyboard better than the MBP, I know it’s the same keyboard that’s been on the Macbook for a while but I’m happy to finally have one in my hands, it’s a big improvement.

Everyone knows how awesome Firebug is. But for those times you have the unfortunate task of debugging in a browser that isn’t Firefox the Firebug team recently put out a new release of Firebug Lite.

For some time now we’ve always had a shell in Windmill for debugging but it was never as nice as the Firebug shell and maintaining that code was kind of a pain. In the recent 0.8.2 release of Windmill we dropped our old shell code and fully integrated Firebug Lite so that whenever you need to debug a test in any browser you can open Firebug Lite in the target application by clicking a single link in the Windmill IDE.

I can’t even begin to explain how much easier this is going to make writing and debugging tests in Windmill.

I had the pleasure of attending the very first DjangoCon this last weekend.

I’ve attended a lot of conferences; big conferences, small conferences, very targetted conferences and very broad conferences. I think DjangoCon may have been the best conference I’ve ever been to.

Over the years what I’ve learned to truly value about a conference is; meeting with people in the community, learning and understanding things I care about that would be harder to learn from reading than listening to a speaker, and just generally having a good time. DjangoCon was fantastic on all fronts. Being so targeted (people who use Django) there weren’t many talks that I didn’t care about and the people speaking were in all cases the best person to be speaking about a particular topic. The Django community is my favorite open source community, beating out the larger Python community by a small margin. Everyone is nice, helpful, and rarely suffer from even a marginal amount of smugness. Add food, decent wifi, and Cal’s talk “Why I hate Django” and you definitely cover the “having a good time” bit.

AppEngine

I messed with AppEngine for a couple weeks when it was first released. There were some serious rough edges that I decided I couldn’t deal with and I really didn’t completely get my head around the datastore.

Seeing Guido’s talk I noticed that those rough edges have been smoothed out and the development environment seems nicer, not to mention the documentation seems to have significantly improved. I’ve also been doing a lot of CouchDB work lately so I’ve internalized the “Document Store” concept and once I saw the Expando class provided by AppEngine I felt like I was now in a place where I can really understand Google’s datastore. It’s also not documented very well but Google’s datastore supports some geo data, which has always been a concern of mine putting too many eggs in the CouchDB basket right now. Add the fact that I absolutely hate managing infrastructure and I made a quick decision to invest once again in AppEngine or an upcoming project.

Why I Hate Django

I don’t think anyone will disagree that the highlight of Day One was Cal Henderson talk “Why I Hate Django”, he was absolutely hilarious. The video will be up eventually and I highly suggest everyone watch because it’s just too damn entertaining. I pulled a series of CC pictures from flickr in the meantime that illustrate the humor of the talk.

Live Testing

Something like two years ago I wrote a patch to add live server support to the Django Test Framework (based on unittest). That patch fell out of sync with trunk and on the floor at some point. Eventually an intern at Disqus came along and got it working again and made some really nice improvements.

Although the patch didn’t make it in to Django 1.0 I’m told it should land soon and will be in 3.1, but Disqus has been running with it all summer, using it for some Selenium tests. Well I couldn’t stand idle while Selenium test were running with Django and Windmill tests weren’t so I put my head down for about an hour during one of the talks and committed Django support to Windmill trunk. I ported the current patch for live server support in Django’s Trac so that it dynamically adds what is needed to Django and you can use this with Django 1.0 unmodified, and created a TestCase class that will launch your Windmill tests from within the normal Django python mange.py test runner. I wrote up a bit more about this in a post to the windmill-dev list.

Adam wrote a great post recently on some code he wrote that outputs his test results in an xml format that Hudson can digest. It serves as an example of a fairly underutilized feature in functest for reporting. It’s incredibly simple and powerful, you add a simple class to the highest level __init__.py with your report code and it will be called when all the tests have finished.

Ideally you want to checkin your reporting mechanism with the source for all your tests but only run the reporter when it runs in continuous integration. Another functest feature is that any unrecognized command line = arguments are stuffed in to dictionary at functest.registry, and there is code in the windmill command line that will do the same.

$ windmill shell report=true                              [14:15]
report is not a windmill argument. Sticking in functest registry.
 
In [1]: import functest
In [2]: functest.registry
Out[2]: {'report': 'true'}

You can use this in your Report class to decide when you should actually report, you can even do different types of reporting based on what you pass. I wrote an example by refactoring Adam’s earlier Hudson example, I’ve also sped up some of the code and use ElementTree to write the xml instead of writing the string output by hand.

import functest
from functest import reports
from xml.etree import ElementTree
 
class JUnitReporter(reports.FunctestReportInterface):
    def summary(self, test_list, totals_dict, stdout_capture):
        if functest.registry.get('report', False):
            total_sec = reduce(lambda x, y: (y.endtime - y.starttime).seconds + x, test_list, 0)
            e = ElementTree.Element('testsuite')
            e.attrib['errors'] = e.attrib['failures'] = str(totals_dict['fail'])
            e.attrib['tests'] = str(len(test_list))
            e.attrib['name'] = 'windmill.functional'
            e.attrib['time'] = str(total_sec)
            for entry in test_list:
                t = entry.endtime - entry.starttime
                test = ElementTree.Element('testcase')
                test.attrib['classname'] = test.attrib['name'] = entry.__name__
                test.attrib['time'] = str(t.seconds)+'.'+str(t.microseconds)
                if entry.result is not True:
                    failure = ElementTree.Element('failure')
                    failure.attrib['type'] = entry.tb[-1].split(':')[0]
                    failure.text = '\n'.join(entry.tb)
                    test.append(failure)
                e.append(test)
 
            if len(stdout_capture):
                # ElementTree doesn't support CDATA so we need to hack it a little
                replace = '#$!#$!replace#@!$'
                sysout = ElementTree.Element('system-out')
                sysout.text = replace
                e.append(sysout)
                outs = ElementTree.tostring(e).replace(replace, '<!--[CDATA['+stdout_capture+']]-->')
            else:
                outs = ElementTree.tostring(e)
 
            f = open('continuous_test.log','w')
            f.write(outs) ; f.close()
 
reports.register_reporter(JUnitReporter())