This question has come up a few times; “How do I do atomic increments with CouchDB”.

The existing document update semantics aren’t intended for operations that work on the “current” document but instead on a specific document revision in order to avoid conflicts. However, update functions always get the current document along with the request object which allows you to do update operations without any specific rev.

The update function below should increase or decrease any integer value in any document given just an array of properties that resolve the proper attribute in the document.

function (doc, request) {
  var a = JSON.parse(request.body);
  var attr = doc;
  if (typeof a.lookup == "string") { last = a.lookup }
  else {
    var last = a.lookup.pop();
    a.lookup.forEach(function (name) { attr = attr[name]; })
  }
  if (a.increase) { attr[last] + a.increase }
  if (a.decrease) { attr[last] + a.decrease }
  return [doc, "Updated, new value is "+attr[last] ]
}

With this function you can easily increase or decrease any value using the HTTP API for this update function.

PUT /dbname/_design/designdoc/_show/atomic/docid
{”lookup”:”name”,”increase”:1}

PUT /dbname/_design/designdoc/_show/atomic/docid
{”lookup”:["attr1","attr2"],”decrease”:1}

For more on update functions check out the wiki page

Enjoy!

One of the best features in CouchDB is the change notifications.

Basically, it’s a Comet style push service that informs you of updates to a database. This is great when you’re building Browser apps because you get Comet for free!

However, one thing that’s missing is being able to write some code in your design document that consumes these changes on the backend and doesn’t depend on another active client. To remedy this situation I decided to write a generic change consumer that you could point at CouchDB and it would find any change handlers in any of the design documents and keep them running, consuming changes, and removing or replacing them when the design document changed.

The best part is that it’s written in node.js, which means you can also make HTTP connections out to the rest of the world and put data back in to CouchDB. And since it’s a service that is intended to stay up indefinitely with only one instance, you can also use setInterval to take care of tasks you would have previously put in cron

All the code is over on github.

To start:

node service http://localhost:5984

To define a listener in your design document.

{ ....

"changes":"var sys = require('sys');
           var listener = function (change) {
             sys.puts(JSON.stringify(change));
           }
           exports.listener = listener;",
}

The new year is bringing some big changes for me. A few weeks back I accepted a position at Relaxed Inc. and notified Mozilla that I would be leaving at the end of the year.

Mozilla

I started working at Mozilla 2 years ago. I started the day after my employment at the Open Source Applications Foundation ended. At this point I already took for granted some of the best parts of working at Mozilla; working for a public benefit organization, spending 100% of my time working on Open Source, working with very smart people in the open (lists, IRC, etc.).

But Mozilla is even more than all that. Succeeding at Mozilla means something more than a pat on the back and a good end of the year review. When you succeed at Mozilla you impact one of the most important products on the internet. You reach hundreds of millions of users and contribute to keeping the web an open and free (as in speech) world. There is no other place in the world you can work where you can conceivably have this kind of impact.

Mozilla as an organization is truly unique. Last year was the hardest I’ve ever had, i suffered a huge loss in my personal life and Mozilla was as supportive during this time as any of my friends or family. There are a lot of places that let you put so much of yourself in to the organization to help it attain it’s goals but there are only a handful that are there to support you when you need it.

Relaxed Inc.

I started using CouchDB in 2008 after a great talk by Jan Lehnardt at OSCON. I started using it right away and over the next year it re-shaped how I think about web development and applications. In the last 6 months my group at Mozilla has become a heavy CouchDB user and not just because of my own interest but because CouchDB was the only solution for some of the harder problems we needed to solve with our results storage.

As I’ve used CouchDB more and more and become a part of the CouchDB community I’ve had the pleasure of knowing some of the core contributors, three of which have decided to found a new startup around CouchDB; Jan Lehnardt, J Chris Anderson, and the creator of CouchDB Damien Katz. Shorty after they received their funding they made an offer. It’s an amazing opportunity and while the decision to leave Mozilla is one of the hardest I’ve ever had to make I’m very excited about my future at Relaxed.

The Future

I’m really looking forward to working with everyone at Relaxed. It’s an exciting time and I’m not 100% sure yet which projects I currently work on that I will still have time to maintain. In the next week or so I’ll be doing a blog post on all the libraries I currently work on and maintain (it’s a long list) and what their status is moving forward. I still maintain code I wrote long before I worked at Mozilla and have every intention of continuing to work on some of the projects I started at Mozilla.

One thing is certain. I’m not the guy who figures out how to test the browser any more. Windmill and Mozmill are important projects that I have every intention of supporting by making time for code reviews and community support but I won’t be available to put time in to new feature work and refactoring like I have in the past. Luckily there are solid communities behind both of these projects and I’m confident that there are people who can continue to drive them in the future.

I don’t know what is going to happen next, all I know is that it should be fun, it won’t be like anything I’ve done before, and will certainly continue to include lots JavaScript and Python.

For everyone who depends on me and the code I’ve written over the last few years I’ll be sure to keep you all up to date. And one thing I can promise is that if you want to fix anything in one my projects, fork it on github and send me a pull request and I will always find time to look at it

We’re gearing up for some heavy CouchDB usage in a new automation system and it has fallen upon me to do some performance benchmarking.

The most important thing for us to figure out was whether or not the CentOS virtual machine we’re currently running CouchDB on is going to be enough even in the short term. Until today we’ve been running 0.9 and have encountered performance problems.

Our main bottleneck is, and has always been, view generation and update performance. We tend to have medium to large size documents (jobs are relatively small but results from test runs can be incredibly large).

View generation of large documents has typically been our biggest issue which we have previously mitigated by refreshing all views after any large write but that isn’t going to work for the amount of results that we plan on pouring in to the new system.

Last weekend I wrote a Python view server for CouchDB. couchdb-python includes a view server but in the past I’ve heard complaints about performance (although none recently). In addition, the view server in couchdb-python only supports map and reduce, which is only about 1/5 of the current view server spec which includes handlers for update, show, list, filter, and validate which provide the groundwork for CouchDB as an application platform. As of Sunday my view server passes all of the current CouchDB spec and initial performance tests showed it faster than the JavaScript view server.

Below are the performance graphs for CouchDB trunk running on a CentOS virtual machine. I’m using Python 2.6 with the default stdlib json library. The spidermonkey core is 1.7 (I don’t know what the status of using 1.8 with CouchDB is but as we’ll see below, this won’t improve performance too much for these tests).

These graphs show view generation time for a given number of documents in a new database. The design doc I used had two views, one does emit(doc['type'],doc), the other emit(doc['_id'], 1).

The graphs support zooming, mouseover and all kinds of flot goodness

JavaScript is the yellow line. Python is the Blue line.

This is a test of moderately sized documents, what we normally expect the size of a job or build description. Each document is identical and fairly simple with a size of ~1,588 bytes.

These documents were incredibly large, they were taken from a full fennec mochitest run. Each document is identical and while large it consists mostly of small sized JSON objects inside a much larger JSON object coming in at ~139,096 bytes.

I had also intended to chart the reduce performance with a simple sum operation but all the results were sub-second regardless of the amount of documents I threw at it with Python being only a little faster than JavaScript.

The nearly identical reduce time tells me that the actual code processing time inside the view functions are hardly different which means that the large difference in performance during view generation is most likely due to JSON serialization time. This also explains why larger documents cause an even greater difference in performance between Python and JavaScript.

Improving Performance

The Python view server is already as optimized as I can imagine for processing time inside the views. Since CouchDB doesn’t provide a way for the view server to support it’s own concurrency we’ve basically hit the wall here on what Python can provide. If we increased the complexity of the view functions I think that Python would start to show better than Spidermonkey 1.7, but 1.8 with traceing enabled would likely bridge that gap, possibly even showing JavaScript faster than Python.

The big problem is JSON serialization. We can make Python faster by compiling simplejson with C speedups. But using the C based JSON parser in newer versions of Spidermonkey requires some other changes to CouchDB since there are differences in the encoding of undefined.

At the end of the day though, this all looks great. CouchDB trunk (pre-0.11) is going to run fast enough for what we need right now even on a virtual machine and if we start to see view generation bottlenecks on views that aren’t hit as often and have to update a large number of documents we can just move those views to Python and the performance should be back down to sub-second.

I’ll be leaving tomorrow morning for [Open Source Bridge](http://opensourcebridge.org/) in Portland, Oregon.

I’m putting together a new [Windmill talk](http://opensourcebridge.org/sessions/36) that tries to incorporate all the feedback we’ve received over the last year of speaking which I’ll be presenting on Thursday.

Mozilla is also a [sponsoring](http://opensourcebridge.org/sponsors/) the conference and there is going to be some great [Firefox related sprints in the hacker lounge](http://opensourcebridge.org/wiki/Hacker_Lounge). Dietrich is also giving what sounds like an awesome talk on extending Firefox called [Firefox Switchblade](http://opensourcebridge.org/sessions/251).

Hope to see you all there!

PS. I’ll also be at EuroPython and the Community Leadership Summit, more on those later

I had some fun this weekend with Python, <video>, CouchDB and Brett Gaylor’s [RiP: A Remix Manifesto](http://www.ripremix.com). In just a few hours I was able to crank out a little [annotations remix](http://ripannotations.pythonesque.org) which allows anyone to add annotations to the film that are displayed as people view it.

I’m hosting it on my little mac mini (currently hidden in a data-center) so hopefully it doesn’t fall over pushing so much video

I’ve posted all the code up on [github](http://github.com/mikeal/ripannotations/tree/master). The more I use <video> and CouchDB the more excited I get about the future of web applications. This entire project was done in little chunks of spare time over the weekend and most of that was me messing around with styling. To get the data stored, queried, and displayed took less than 2 hours.

Hope you all enjoy the annotations remix and if you haven’t already go and pay what you want for a terrific copy of [RiP: A Remix Manifesto](http://www.ripremix.com). It’s worth it.

I’ve been writing some automated tests for the new <video> work in Firefox 3.5 and it’s been incredibly fun. I spent some time in the trenches at Real Networks and dealt with accessing embedded video in the browser so I really appreciate how much nicer <video> is than embedded players like Real and Flash.

People talk a lot about embedded plugins being *”stuck in a box”* and while this is partially true I don’t think it’s descriptive enough. Embedded players like RealPlayer and Flash open up methods that can be called from javascript, so you do have some ways of interacting with the box, but it’s limited to somewhat rigid pre-defined interfaces. It’s also worth noting that using these interfaces just doesn’t **feel** like web development the way doing regular JavaScript and DOM feels. It’s not dynamic and you’re limited to only doing **what the plugin authors have already thought of**. The beautiful thing about the browser as a platform is that it enables usage far beyond what the original creators have intended.

Using <video> **feels** like the web. You resize it by setting size attributes the same way you would an interface node. You seek by setting the currentTime attribute, you can listen for the time changes to create your own interface by adding a listener for “timeupdate” events. You can write interfaces around <video> without switching contexts because it’s not a foreign technology jammed in to the browser. Here is some javascript that implements a few video controls.

var v = document.getElementById("v1");
document.getElementById("playDiv").onclick = function(e) {v.play()}
document.getElementById("pauseDiv").onclick = function(e) {v.pause()}
document.getElementById("reloadDiv").onclick = function(e) {v.load()}
document.getElementById("seekDiv").onclick = function(e) {
    v.currentTime = document.getElementById('seekField').value;
    }

Isn’t that beautiful! Maybe I’m overly excited because I’ve had to deal with this using older technologies but this just **feels** good, and if it feels good then I know people are going to get excited about it and build cool stuff.

Just in time for PyCon we’re releasing [Windmill 1.0](http://www.getwindmill.com/archives/412). It’s been almost 3 years of development and I’m both excited and relieved to have something we’re comfortable calling 1.0.

The latest RCs have seen bigger improvements than we thought would make it. A new wave of contributions has given us some great new features in Django support and test serialization and [Adam](http://www.adamchristian.com) has done some great UI work as well.

Big thanks to everyone who contributed to the release and I hope to see many of you at PyCon. Adam is doing a [talk on windmill](http://us.pycon.org/2009/conference/schedule/event/9/) which I’ll be helping out with and Adam and I will also be on a [functional testing tools panel](http://us.pycon.org/2009/conference/schedule/event/88/) which should be a lot of fun. And if you’re staying around for the sprint days of the conference we’ll be leading a [windmill sprint](http://us.pycon.org/2009/sprints/projects/windmill/).

I used to be really in to public bookmark services like delicious and magnolia. Then Firefox3 was released and the new bookmarks UI combined with the awesomebar was just much easier to use than any of the sharing extensions.

I never took the time to sit down and write a simple sync/push extension because I just figured that someone else would do it

It’s been a year now, it’s time for PushMarks. https://addons.mozilla.org/en-US/firefox/addon/10806/

PushMarks is a simple firefox extension that pushes new bookmarks to your delicious account (more services coming). You can also initiate a “sync” which will go through all your local bookmarks and push them to delicious and also make sure any bookmarks you have on delicious are in your local bookmarks.

If you do a big sync, please be patient. It takes a while for all the bookmarks to get up to delicious because I have to throttle the connections for fear of being locked out of the delicious API.

Of course it’s all open source and up on github http://github.com/mikeal/pushmarks/tree/master .

It was pretty hard to get work done yesterday. My feed reader was filled with stories of how Apple is pulling out of Macworld and my IM and email were bursting with people asking me what I think about FireUnit.

I’m the co-author of two tools that overlap with FireUnit, Windmill and MozMill. I say overlap and not “compete” because I think the tools differ enough in their goals that they don’t directly compete.

FireUnit has some limitations. It’s an extension to Firebug which means it will only run on Firefox. It’s focused on providing a simple API for people who are competent JavaScript programmers. I think it’s safe to say that there is an emphasis on unittesting and not functional testing, and certainly an emphasis on testing content and not chrome.

Something really great happens when you create a tool that embraces limitations instead of trying to be too much to too many people. You tend to be able to create better smaller purpose tools for a selection of people that have been beating themselves up trying to make tools with less limitations work for a smaller use case.

The API

// Simple true-like/false-like testing
fireunit.ok( true, "I'm going to pass!" );
fireunit.ok( false, "I'm going to fail!" );
 
// Compare two strings - shows a diff of the
// results if they're different
fireunit.compare(
  "The lazy fox jumped over the log.",
  "The lazy brown fox jumped the log.",
  "Are these two strings the same?"
);
 
// Compare a string using a regular expression
fireunit.reCompare(
  /The .* fox jumped the log./,
  "The lazy brown fox jumped the log.",
  "Compare a string using a RegExp."
);
 
// Display the total results
fireunit.testDone();

This is the example from John Resig’s post about FireUnit. I think it illustrates how awesome the API is. I’ve always been a huge fan of this “ok” style API instead of the larger jUnit style API I keep being talked in to implementing. Clearly FireUnit assumes it’s audience is JavaScript programmers that can write normal expressions to handle their pass/fail logic without the hand holding of a bigger API.

The beauty of the comparison API you really don’t appreciate until you look at the UI.

Debugging

I’m most critical of tools that don’t provide sufficient debugging tools. The test debugging environment is even more important than the test authoring environment in the long run.

There is no better tool for debugging the web than Firebug. A big part of Windmill’s debugging strategy is it’s integration of Firebug and Firebug Lite because there is just no better tool in the world. Firebug ROCKS!

FireUnit is built as an extension of Firebug, it’s integrated in a beautiful and intuitive way so they’ve totally dodged the need for any additional debugging work outside of creating a good UI for the test bits. I’m calling this out because I think as time goes on people aren’t going to talk about the debugging experience in FireUnit because they are so used to already using Firebug for debugging their regular development and won’t recognize how good they have it debugging their tests.

The UI

Another benefit of embracing limitations is that you trim down your feature set to only what is actually necessary. The upside to having a simple API is that you reduce the complexity of test data you need to represent in the UI. With all that going for it I expected the FireUnit UI to be pretty good but I was still blown away.

Look at that comparison UI!

The Bad

Within the limitations described above FireUnit is great but as soon as you start to cross them you’ll be disappointed.

Although they have some helper APIs for doing some functional testing it’s not a good functional testing tool. The tests are async which is a real pain for functional testing which is incredibly synchronous. But I think if you want to do functional testing multiple browser support is a big concern and will already preclude you from using FireUnit.

Although you can test Chrome (Mozilla Chrome not Google Chrome) I wouldn’t venture outside of unittests. If you need to start getting at anonymous nodes and simulating events on them you’re gonna see your code get about 4 times larger than a typical mozmill test.

Right now there doesn’t seem to be a way to run FireUnit tests in continuous integration (tinderbox, buildbot, hudson, etc). But this could easily be overcome using jsbridge and I’m happy to point the FireUnit guys in the right direction

At the end of the day I think there is a huge need for a tool like FireUnit that just does what it does well and doesn’t try to be too much to too many people. A tool like this is guaranteed to make it’s users very happy and I think those users are currently unhappy with the existing set of automation tools that are trying to fill a larger feature set.