Adam wrote a great post recently on some code he wrote that outputs his test results in an xml format that Hudson can digest. It serves as an example of a fairly underutilized feature in functest for reporting. It’s incredibly simple and powerful, you add a simple class to the highest level __init__.py with your report code and it will be called when all the tests have finished.
Ideally you want to checkin your reporting mechanism with the source for all your tests but only run the reporter when it runs in continuous integration. Another functest feature is that any unrecognized command line = arguments are stuffed in to dictionary at functest.registry, and there is code in the windmill command line that will do the same.
$ windmill shell report=true [14:15]
report is not a windmill argument. Sticking in functest registry.
In [1]: import functest
In [2]: functest.registry
Out[2]: {'report': 'true'}You can use this in your Report class to decide when you should actually report, you can even do different types of reporting based on what you pass. I wrote an example by refactoring Adam’s earlier Hudson example, I’ve also sped up some of the code and use ElementTree to write the xml instead of writing the string output by hand.
import functest from functest import reports from xml.etree import ElementTree class JUnitReporter(reports.FunctestReportInterface): def summary(self, test_list, totals_dict, stdout_capture): if functest.registry.get('report', False): total_sec = reduce(lambda x, y: (y.endtime - y.starttime).seconds + x, test_list, 0) e = ElementTree.Element('testsuite') e.attrib['errors'] = e.attrib['failures'] = str(totals_dict['fail']) e.attrib['tests'] = str(len(test_list)) e.attrib['name'] = 'windmill.functional' e.attrib['time'] = str(total_sec) for entry in test_list: t = entry.endtime - entry.starttime test = ElementTree.Element('testcase') test.attrib['classname'] = test.attrib['name'] = entry.__name__ test.attrib['time'] = str(t.seconds)+'.'+str(t.microseconds) if entry.result is not True: failure = ElementTree.Element('failure') failure.attrib['type'] = entry.tb[-1].split(':')[0] failure.text = '\n'.join(entry.tb) test.append(failure) e.append(test) if len(stdout_capture): # ElementTree doesn't support CDATA so we need to hack it a little replace = '#$!#$!replace#@!$' sysout = ElementTree.Element('system-out') sysout.text = replace e.append(sysout) outs = ElementTree.tostring(e).replace(replace, '<!--[CDATA['+stdout_capture+']]-->') else: outs = ElementTree.tostring(e) f = open('continuous_test.log','w') f.write(outs) ; f.close() reports.register_reporter(JUnitReporter())
[...] UPDATE: Mikeal has posted some additions/revisions to this code sample, here. [...]
[...] from functest import reports from datetime import datetime class JUnitReport(reports.FunctestReportInterface): def summary(self, test_list, totals_dict, stdout_capture): total_sec = 0 for entry in test_list: time_delta = entry.endtime - entry.starttime total_sec += time_delta.seconds out = ‘n’ out += ‘n’ for entry in test_list: if entry.result is not True: entry_time = entry.endtime - entry.starttime out += ‘n’ out += ‘n’ #out += str(stdout_capture) #until I can figure out how to get the traceback out += ‘There was an error in ‘+ entry.name out += ‘nn’ out += ” else: entry_time = entry.endtime - entry.starttime out += ‘n’ out += ‘nn’ out += ” f=open(’continuous_test.log’,'w’) f.write(out); f.close() reports.register_reporter(JUnitReport()) Happy automating! UPDATE: Mikeal has posted some additions/revisions to this code sample, here. [...]
[...] You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here. [...]
[...] You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here. [...]
Hi,
I am not fluent at all in python so I don t know where I should put this init.py
I try putting it in the same folder than my test but it did not work.
The response is probably here “you add a simple class to the highest level init.py with your report code” but I don t understand the meaning of this sentence.
So functest (the framework that windmill embeds) traverses up all your test modules. Any directory with init.py is a valid test module. You can use the directory/module hierarchy to separate out different test functionality. If you’re writing a reporter class you probably want it to work with all your tests so you would want to stick it at the top of the module hierarchy.
But regardless of where you put the class, if it’s in a test module that gets run it will be registered when the module is loaded. So if you put it in the folder your test is in, it should work, if not then something is wrong.
If something seems very off you can come on to IRC in #windmill on irc.freenode.net and someone should be around to help you debug it. Although right now during the holidays people seem to have been away a bit more.