Headless Testing for Continuous Integration with Git and Jasmine

January 14, 2011 | Ariya Hidayat

Continuous integration typically relies on an automated build and test system, with a central build server that verifies the code repository on a regular basis by running a predefined set of tests. However, it's often a good idea to supplement these tests, with pre-commit testing. At the very least, we'd like to see every developer run general regression tests before he or she commits. The practice of pre-commit testing is now common enough that there are a number of test frameworks, e.g. Selenium, that accommodate this practice well. In this blog post, I'll show you a simple WebKit-based application which executes and runs the Jasmine-based SpecRunner tool headlessly. Since it runs completely from the command line, I'll show you how to combine this little tool with a git pre-commit hook to prevent broken code from being checked in. Note: I'm going to use QtWebKit so that all popular platforms (Windows, Mac, Linux) are covered. If you want to follow along, you'll need Qt libraries version 4.7 or later before you can continue. (If you're on Linux, there's no need to download the libraries, almost every distribution offers a ready-to-use binary package, usually under the name libqt4-devel or libqt4-dev.)

The Code

Let’s start by checking out the code from the Sencha Labs github, which is as easy as:

git clone git://github.com/senchalabs/examples.git
cd examples/specrunner

To compile the program, just run:

qmake && make

(On Mac, you might need to run qmake -spec macx-g++ instead). Alternatively, you could open specrunner.pro file from an IDE, e.g. Qt Creator and compile it from there. If everthing goes well, you should have a specrunner executable (on Mac OS X, it should appear under specrunner.app/Contents/MacOS subdirectory). (Tip: copy the specrunner executable to your PATH so it is easily accessible.) And now do a trial run by entering:

specrunner SpecRunner.html

If you've installed it correctly, you'll get a simple response, e.g.

specrunner SpecRunner.html 
1661 specs, 0 failures in 2.71s

If something has gone wrong then you'll get something like:

specrunner SpecRunner.html 
FAIL: 1661 specs, 1 failure in 2.45s

Note: Since often SpecRunner include other JavaScript source files with relative paths, I recommend that you run specrunner from a suitable working directory.

Combining specrunner with Git pre-commit hook

Git, a popular version control system, has powerful support for hooks, i.e. scripts which get executed before or after certain actions. One of them is pre-commit hook, executed right before a git commit. The pre-commit hook is useful for integrated testing because git will not continue with the commit if the pre-commit hook returns non-zero. Let's go through an example to see how we can combine this with the above specrunner tool. First create a dummy repository:

mkdir foobar
cd foobar
git init

Now, create a simple Jasmine-based SpecRunner.html with the following content:

 
<html>
<head>
  <link rel="stylesheet" type="text/css" href="lib/jasmine-1.0.1/jasmine.css">
  <script type="text/javascript" src="lib/jasmine-1.0.1/jasmine.js"></script>
  <script type="text/javascript" src="lib/jasmine-1.0.1/jasmine-html.js"></script>
 
  <script>
  describe('FooBar', function(){
    it('should increment a variable', function () {
      var foo = 0;
      foo++;
      expect(foo).toEqual(1); 
    });
  });
  </script>
 
</head>
<body>
 
<script type="text/javascript">
  jasmine.getEnv().addReporter(new jasmine.TrivialReporter());
  jasmine.getEnv().execute();
</script>
 
</body>
</html>

Next -- if you don't have it already -- download Jasmine and place it under the lib subdirectory. The test above is very simple: it's a function which increments a value. It should pass, in fact you can verify it by running our headless specrunner tool:

specrunner SpecRunner.html 
1 spec, 0 failures in 0.011s

Since this passes, commit the code by entering:

git add *
git commit -a -m import

In order to create a pre-commit hook, create a file .git/hooks/pre-commit with the following content:

specrunner SpecRunner.html
exit $?

This runs Specrunner and reuses its return value, which will be passed to git when it executes this commit hook. Now, make sure that the pre-commit is executable:

chmod +x .git/hooks/pre-commit

And here comes the fun part. Let's try to break the tests, for example by changing foo++ in SpecRunner.html with foo += 2. If you do that and try to commit, you'll get the following error:

git commit -a -m "Test"
FAIL: 1 spec, 1 failure in 0.013s
    FooBar
      should increment a variable
        Expected 2 to equal 1.

The failure message comes from Specrunner tool. Because the return value is non-zero, the commit will fail. You can check this by running git diff: your repository is still dirty. The technique outlined above is not a substitute for a comprehensive, multiple-browser test system. However, this example shows one good purpose: a very fast sanity check to prevent a potential regression quietly sneaking into the code repository. Feel free to extend and customize this example to suit your needs and pass it along!

There are 7 responses. Add yours.

Awesome Bob

1 year ago

I’m 100% behind this approach. I think I’ll update my SVN Repository so a prehook would require that phpunit is run on my repository before new changes are commited. Great article!

Henry P

Very impressive. We are still on SVN, any chance you could give some pointers on setting this up on SVN?

Chris Johnson

This is very easy to do in SVN, as SVN also supports a wide variety of powerful hooks.

Log into the machine which hosts your SVN repository and look for the directory /hooks. In that directory, there should be example hook scripts, usually named like pre-commit.tmpl. Copy that example, or start a new sh script in a file named pre-commit.

You can use the exact script code given above:

An exit value of 0 allows SVN to commit, while non-zero values block commit, just as with git.

Ariya Hidayat

@Chris Thanks for the additional info!

Brian Murrell

Hi Ariya,

You mention here that one should have QtWebKit 4.7—did you find any issues with earlier versions? I have QtWebKit 4.6.1 and the simplest jasmine match fails 90% of the time (expect(foo.bar).toEqual(‘bar’) fails with “Expected ‘bar’ to equal ‘bar’”.) If I re-run the test multiple times, it succeeds about 10% of the time. I suspect a QtWebkit bug…

I cannot find any google hits on “qtwebkit jasmine”. If you have any experience, thanks for offering….

Brian

Ariya

@Brian The code works only for Qt 4.7. I’m sure it’s possible to make it work for 4.6 as well, however I can’t promise. Let me put this in our radar and just watch the repository in case we make any update.

@Ariya, others who might encounter this problem—I upgraded my Qt to 4.7.1 and it indeed solves my problem. So, I think it is safe to say that a bug in QtWebKit 4.6 prevents jasmine from working.

Next task (or blog topic? is to get QtWebKit+Jasmine working as a slave to JsTestDriver http://code.google.com/p/js-test-driver/—I saw that there is a jasmine plugin for JsTestDriver, but I have not looked yet to see what type of capability exists for for making QtWebKit a slave to JsTestDriver…

Thanks,
Brian

Comments are Gravatar enabled. Your email address will not be shown.

Commenting is not available in this channel entry.

Blog