Regression testing

One of the great advantages of open source for library authors is that you get access to your users' unit test suites. Ensuring that your library passes its own test suite before release is good, but sometimes your users are relying on features (or undocumented behaviour) you didn't test. Running their tests too can give you extra confidence, and it's easy to make this happen automatically in your release script.

I got to thinking about this because Plash 1.18 was released a few days ago, with a bug that causes 0launch to hang (and this is detected by 0launch's fairly comprehensive test suite). By using your user's test suites, you can avoid this.

You can even test against multiple versions (if your installation system supports parallel installs). As a demonstration, I've written an example script, multi-tester, which downloads and runs the test suite of every version of a chosen program. There's a setting near the start where you give the Zero Install URI of the program to test and the name of the test script inside it. e.g. with this configuration:

test_iface = 'http://0install.net/2007/interfaces/0publish-gui.xml'
test_main = 'tests/testxml.py'

It prints this output:

$ ./multi-tester 
Testing 0.7-post
testAfter (__main__.TestXML) ... ok
testBefore (__main__.TestXML) ... ok
Testing 0.7
...
Testing 0.6
...
Passed  : 0.7-post, 0.7, 0.6, 0.5, 0.4
Skipped : 0.3, 0.2, 0.1
Failed  : 

Here, 0.7-post is the latest subversion checkout of 0publish-gui, and the others are released versions. Versions before 0.4 didn't have this test, so they're skipped. So, we now know that every recent version of 0publish-gui works with my svn copy of ROX-Lib!

Actually, 0publish-gui has a rather poor test suite so this doesn't prove much, but it's just an example. I should probably try using the Linux Desktop Testing Project to automate the GUI tests, but it seems rather hard to set up.

If you're interesting in testing your Zero Install programs, you might also be interested in the new FeedLint program, which checks that all your links are still working (Zero Install caches everything, so if your download server eats your archives you might not notice until your users start complaining).