MozMill already has some big advantages over mochitest. For one, you don’t need a special test build, you can write and run tests against any build of a Mozilla Application. You can even write and run tests just using the extension and not worry about any Python stuff until you set it up in continuous integration.
MozMill already runs in “chrome”, or what Adam and I have been referring to as the “trusted” environment. All the tests that get run in MozMill are imported in this environment, which means you have access to pretty much everything in the Mozilla environment. We also have a dependency system and a setup/teardown system for tests.
All we were missing was a simple unittest API that can trigger failures without causing exceptions like the MozMill functional testing APIs. I looked at what others have done including dojo, mochikit, and jsunit.
Fortunately MozMill gets out of any test registration and staggering stuff, it already has a well defined way of defining tests and dependencies and a clear way to run tests for debugging and continuous integration. Of all the different jum (JavaScript Unittest Module) APIs I liked jsunit the most except for the incredibly stupid idea of passing the comment argument first instead of making it an optional last argument.
In the end MozMill got it’s own jum which I’ve just finished up documenting.
var jum = {}; Components.utils.import('resource://mozmill/modules/jum.js', jum); var testAsserts = function() { jum.assert(true); jum.assertTrue(true); jum.assertFalse(false); jum.assertEquals('asdf', 'asdf'); jum.assertNotEquals('asdf', 'fdsa'); jum.assertNull(null); jum.assertNotNull(true); jum.assertUndefined({}.asdf); jum.assertNotUndefined('asdf'); jum.assertNaN('a'); jum.assertNotNaN(4); jum.pass(); } var testNotAsserts = function() { // All of these calls should fail jum.assert(false); jum.assertTrue(false); jum.assertTrue('asf'); jum.assertFalse(true); jum.assertFalse('asdf'); jum.assertEquals('asdf', 'fdsa'); jum.assertNotEquals('asdf', 'asdf'); jum.assertNull(true); jum.assertNotNull(null); jum.assertUndefined('asdf'); jum.assertNotUndefined({}.asdf); jum.assertNaN(4); jum.assertNotNaN('f'); jum.fail(); }
All of these functions take an optional comment argument which can be any JSON serializable value (string, int, obj, array, null, bool).
What we’re missing right now is the http local file server. I’ve already ported mochikit’s web server to be a resource but I’m still working on a better API for using it. What we want is for tests to simply define local relative paths that should be mounted at different local url namespaces and the framework can handle starting and stopping the server if necessary. This API will be finished in the next few weeks and ready for MozMill 1.0 (which we’re looking at pushing out before January 1st).
You don’t need a special test build for mochitest either. You do need to point the build being tested at the tests (which are not “built by default”) and you do need to run against a special profile (which the python script that starts the test creates).
And the reason for that is that mochitest aims to test regular webpages (in particular to test the security restrictions that we impose on those webpages), except in cases where it needs to do something special (in which you have to explicitly request privileges). If you want something that runs as chrome, then either the jssh unit test framework (probably abandoned by now) or the mochichrome and browserchrome frameworks do fine.
For that matter, how _is_ this different from browserchrome in practice?
I guess my real question is what problems mozmill is trying to solve. I can think of a few we have in unit-test land (lack of file:// mochitests, mochichrome running in content docshells, lack of unit tests for signed jars, lack of decent JS (or DOM, for that matter) performance regression tests, not having the JS regression test suite hooked up to automation yet, etc), but none of them are obviously addressed here.
To my knowledge, you cannot download a nightly or release build of Firefox and run mochittests against it. You also cannot run the tests without running the Python script.
MozMill tests don’t need to be defined “as chrome” (in the chrome manifest) in order to run in the privileged environment, the scripts are imported and run in that environment dynamically.
MozMill was originally designed for funcitonal testing, it has a very large API set for simulating user interactions. Taking on unittesting is new to MozMill, this is our first set of features specifically targeted at unittesting. You can also mix and match unittesting and fucntional testing in MozMill (simulate some user interactions to trigger a certain code path, then use the jum to assert various properties).
The main advantage to using MozMill is that anyone in the community can install the extension and write/run/debug test scripts with ease against any release, debug, or nightly build. When tests are run in continuous integration the Python side (using jsbridge) integrates seamlessly but Python is not required to write/run/debug tests.
I’m not familiar enough with the problems you’ve described with mochitests to say whether MozMill handles them or not. MozMill’s environment is intended to be simple and flexible in order to avoid most of the problems that come with creating big “framework” environments for testing, so it’s possible that the boundaries you’re seeing with mochitest don’t exist in MozMill but I can’t tell without more knowledge about the depth of the problems.
mikeal, as long as you have the tests themselves (which does require a build tree), you can in fact point a nightly at them (by having the python script just run it). All the python script is needed for, really, is setting up the special profile. We could get an effect similar to what you describe in terms of write/run/debug against any build at all by packaging up the tests and the profile creator (or even a created profile) and making those available. If we package the profile creator that still requires python; if we don’t, it just requires starting with the -profile option pointing to the given profile.
That said, simulating user interaction well is in fact something that we’re missing from our testing story (it’s possible to do a bit of it in mochitest, but it’s cumbersome, often buggy, and certainly painful to use for things like popups or menus), and it’s great to hear that this will fill that gap! This certainly answers my question about what the goal is.
As far as covering the gaps in mochitest and our other tests… those gaps aren’t due to boundaries per se, certainly not in mochitest. All sorts of things _can_ be accomplished in mochitest right now (e.g. one could create signed jars on the fly). The real problem is the lack of helpers to make those tasks easy to accomplish, which prevents effective writing of tests that exercise certain parts of the codebase.