Adding unit tests to Ext JS core

[rflosi]

Download and unzip http://farmdev.com/src/ext-js-unit-tests.zip

This file contains a copy of ext-2.0 with an added 'tests' directory.
This file also includes a copy of JsUnit and Mock4JS.
There are some initial tests setup under 'tests/unit/'.

I am hoping this will be a starting point for adding unit tests to Ext JS. Please let me know your thoughts and if you will support this testing effort. Thanks.

Below is the README.txt file included in the zip file. The README.txt is in the 'tests' directory.

***

Unit Testing Ext 2.0 with JsSUnit and Mock4JS
--------------------------------------------

There is a new directory called 'tests'.
This README.txt exists at 'tests/README.txt'.
Under 'tests/unit/' you will find a copy of JsUnit and Mock4JS
along with some initial test HTML files.


Running the tests manually
--------------------------

1. Open the file 'tests/unit/jsunit/testRunner.html' in a web browser.
You can open it with the file:// protocol or use a local web server.
This is the JsUnit Test Runner. This provides the interface for
running the test.
2. Enter the filename of the Test Page to be run -- it must be prefixed with
the protocol (http:// or file://). Or click Browse and select a file. To run
the entire suite, select 'tests/unit/test_suite.html'. Test Suite files are
a way of organizing Test Page files.
3. Click Run.
This will run the tests defined in the Test Suite. The Status, Progress
and Runs will give you information on the tests as the run. Any errors
that occur will be listed below that. If an error occurs you can highlight
the error and click "Show selected."


Running the test automatically
------------------------------

In addition to running tests by manually loading a file you can construct
a url to pass in the Test Page information and a flag to run the tests.

testpage::
Add a 'testpage' variable to your query string with the path to your
Test Page or Test Suite just like you would have done manually in step 2 above.

autoRun::
Add 'autoRun=true' to your query string to tell the test runner to run the
tests when the page loads.

Example url::
file:///ext-2.0/tests/unit/jsunit/testRunner.html?testpage=file:///ext-2.0/tests/unit/test_suite.html&autoRun=true


Continuous Integration Test Automation
--------------------------------------

Using a url that can run the tests automatically you can build a call to this
test into your continuous integration testing environment. More info and
examples to come.


Supporting software
-------------------

JsUnit::
http://www.jsunit.net/

Mock4JS::
http://mock4js.sourceforge.net/

[kumar303]

Hello.

Here is a little more background to follow up on Rick's post. Rick and I are part of a team that is building a CMS that relies heavily on Ext JS. In order to minimize bugs and add features quickly, we develop everything with automated unit and functional tests. We noticed there aren't any unit tests for Ext itself so we'd like to offer some help so that Ext may have its own tests maintained by core developers, new tests added as new features get added.

Our main concern will be writing tests for our own application but directly behind that will be helping out with the Ext tests. Are there any core developers out there who have some time to review what Rick has put together here to see if it can be included in Ext directly? If we were to get a work branch in the repository we can continue to improve upon this test suite, otherwise we will maintain it separately and continue submitting revisions.

By the way ... Ext has been extremely stable for us. This is by no means suggesting that it needs testing to fix something "broken." We just feel, from experience, that any large project can only continue to grow larger if it has a suite of automated tests.

-Kumar

[rflosi]

We are currently analyzing the ext library to prioritize a list of what should be unit tested first. If anyone wants to chime in with suggestions, please do.

[rflosi]

Here's the list of things I came up with for a first set of tests:

Ext.util.Observable
Abstract base class that provides a common interface for publishing events.
Ext.Component
Base class for all Ext components.
Ext.BoxComponent
Base class for any visual Ext.Component that uses a box container.
Ext.Container
Base class for any Ext.BoxComponent that can contain other components.
Ext.Panel
Panel is a container that has specific functionality and structural components that make it the perfect building block for application-oriented user interfaces.

[brian.moeskau]

We appreciate the effort. We actually already had some plans to look into a unit testing approach, at least for the core lib, but anything that you already have that can help would be great. I don't have time to look closely at this right this moment, but I will when I get a chance.

[kumar303]

Hi Brian. We'll keep hacking away when we have time, posting back to the thread. This is certainly only one approach and it could be done differently --- we chose JsUnit because it seems to work and has documentation.

[dantheman]

I am up to my ass in alligators at the moment,
but this is one project I will contribute to when
things clear up a bit (~2 months).

Many thanks for kickstarting this.
BTW, I've used JSunit before and it's quite serviceable.

A suggestion for how to prioritize testing:
newest back to oldest. At first blush I'd say
the data classes are big candidates, but they've
been "stable" for much longer than the newer
stuff (component lifecycle, etc). Just one opinion.

I am in a constant state of slack-jawed amazement
that Ext could be built w/o unit tests . . .
I can't seem to get anything working w/o them :D

--dan

[mystix]

stickied this thread (cos i think it's useful ;))

throw some pies at me if you disagree.

[kumar303]

We added a few more tests and simplified a few existing tests, uploaded over the old zip:
http://farmdev.com/src/ext-js-unit-tests.zip

Not sure that we'll get much time to hack over the holidays, but wanted to post some progress.

[brian.moeskau]

Hey guys, I'm going to check this out. One suggestion, it's probably better if you only include the tests folder, not all of Ext, in your download. For example, your included version is already out of date as of yesterday. ;) Better to just let people drop the tests package into the local install of Ext that they already have without having to re-download it.

[kumar303]

yep, that's a good point. However, one may write a unit test based on an old version of the code and that would be confusing if you were trying to run it with the wrong version. Ideally, this should be maintained in the main Ext JS repository, either in trunk or in a branch, that would solve it.

Also, our project has sped up and we probably won't have much time to continue with the unit tests. Thus, I'd like to put it in the hands of your team! Do you want to schedule an IRC session so I can go over what's there and how to add future unit tests? I'd be happy to. This may be helpful especially for core Ext JS developers who are not familiar with unit testing.

I have a few more changes to post and was hoping to get you guys some more tests, but this'll have to do for now.

-Kumar

[psenger]

Hi kumar303,

I have been tasked with exploring JavaScript unit testing frameworks. Outside of the scope of this thread, I talked to brian.moeskau via the premium support, he told me the ExtJS team is working on something for unit testing. However, it is not done. I suspect it will be based on JsUnit and or MockJs. Maybe we can keep up the pressure with this thread?

So, I'm looking at the Zip file from your post. I must be missing something, in ext-js-unit-tests\tests\unit\ there are only 6 html JsUnit Tests. I was expecting something.... more :D. Like a Grid Panel or Data Store example or something else. Have you made anymore progress on this? I would like to expand on it. Your work is Much Appreciated.

Ive been experimenting with JsUnit and MockJS, although I would prefer something more like Selnium's Point-And-Click Scripting, these frameworks will work for our tests. I just hate the idea of coding more JavaScript to test JavaScript.... its kind of ironic.

Thanks,
Phil

[kumar303]

Hi Phil.

You are correct, there are only 6 files and they are very basic tests. Unfortunately we are hip deep in our own Ext JS application, building out custom functionality and testing along the way. We had a few cycles to burn and I wanted to try and kickstart a framework for testing Ext itself. What Rick and I put together is something that runs tests out of the box and is easy to add new tests to. You just create a test class and test methods like the examples already there. We won't have time to maintain this and a test suite for Ext JS will only succeed if *all* Ext JS developers **write tests as they write code**. No one will be able to simply come in a week later and write tests for the latest code. Obviously, to get the test suite started it will need some retrofitting.

I guarantee that once developers start writing tests alongside their code they will be hooked. It's a great morale booster -- getting a test to pass after it has been failing feels great. Also, it provides a lot of confidence when you have regressions for all functionality. You can change *anything* you want and run the entire test suite for immediate feedback on what your change might have broken.

There are caveats of course that will take time and guidance to learn. For example, it is very easy to write an ineffective test or one that is fragile. The last thing you want is to have to update tests even though something isn't broken. It is a fine art to create such a robust test suite, one that only comes with practice. I'd be happy to check in every once in a while and provide feedback to how the test suite is implemented.

If getting all Ext developers on board with writing tests seems crazy, I'd also be happy to hold an IRC overview of the test framework and provide pointers for how to go about writing tests. I think this would be really helpful and I'm sure a lot of developers don't know where to start so it would give them that starting point.

As for your idea of record/playback testing, I'd advise against it. There are tools for this but they are only good for regression testing, and aren't even very good at that. Once you have a stream of recorded data that can be played back, this data is often unruly to maintain and as soon as it breaks no one wants to try and fix it. Also, I don't see the irony in testing javascript with javascript. What better way is there to test it? :) Ext is comprised of code that performs specific functions and you want to exercise those functions in a runtime environment as close to the real thing as possible.

[ibrandt]

Ha, what a hoot... I'm working on an Ext extension, and a couple minor naming conventions aside I coded up exactly the same solution with JSUnit. It was only in searching for a mocking framework for JS that I stumbled onto this thread. DOH!

Influenced by my JUnit experience I put my tests in a "package" (folder) structure under test/unit that mirrors the original sources. So I'd have test/unit/util/ObservableTest.html instead of test/unit/test_util_observable.html. I went camel case to align with the convention in witnessed in the Ext sources. I also put an all-tests.html suite into each package, with a root one in test/unit that calls each of them. Perhaps this organization might scale to large numbers of test files better?

Finally I had a launcher for convenience, test/unit/run-all-tests.html:


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>JSUnit Redirect for All Unit Tests</title>
</head>
<body>
<p>
Redirecting...
</p>
<script type="text/javascript">
// A little path munging to ensure JSUnit can resolve the testPage regardless
// of whether we're opened from a file or HTTP server...
var path = window.location.pathname;
var dir = path.substr(0, path.lastIndexOf('/')+1);
var testPage = dir + "all-tests.html";
window.location = "../jsunit/testRunner.html?testPage=" + testPage + "&autoRun=true";
</script>
</body>
</html>


Anyway, Mock4JS it is, and count this as my vote to integrate this into the main distribution.

[kumar303]

hey, cool! I like your way better ;) I think running all files dynamically is much nicer too as one typically forgets to add hard-coded test files otherwise. And, I agree, the camel case and mirrored directory structure seems more logical. Can you perhaps post a zip file with your tests so far? And if you feel inclined, could you include an updated README file similar to the one we created? Otherwise, I might be able to find time to merge the two but I'm off to the Python conference this week and out of town the next.

-Kumar

[kumar303]

oh, nm, I see you still redirect to an all-tests.html file. I think the only way to get files dynamically is to use php or some server lang ... which adds a bit more complexity (apache + php has to be running). we could include a python (or whatever) script to auto-generate an all-tests.html file and that is probably a better way to go since python comes with most dists (minus Win) these days. Ideally this would hook into ext's own build system but I don't know what that is.

[ibrandt]

oh, nm, I see you still redirect to an all-tests.html file.

Right, test files are still all hard coded in suite functions. Dynamically generated suites would be really nice. I'm picking up JS as I go here, but best I can tell file system access is not readily available when running in a browser? So no way to scrape up all the *Test.html's in process. As for pre-generating the suite files is there a lightweight server-side JS interpreter with a file system API that we could just include in the overlay? Otherwise Python sounds as good a choice as any, but for the dependency perhaps KISS says hard-coded suites aren't really all that bad.

All my tests so far are related to my in progress additions to the framework, so nothing that would be of any use to anyone yet. Still I was going to merge your tests into my suites anyway, so I'll be sure to post an updated overlay in the next day or two.

[rflosi]

We could setup something that could send out an Ajax Json request to the server passing a varible with the calculated directory path which would return a Json list of files in that given directory. Although this won't help us in the the file:// case since there would be no server.

[kumar303]

a great summary here for why projects are more likely to succeed with automated tests:
http://ivory.idyll.org/blog/mar-08/software-quality-death-spiral

[ecommand]

Having unit tests for Ext seems a very good idea to me. Wouldn't it be better to settle on a testing framework and start writing tests for new code, before trying to cover all the old code?

I'm not sure Selenium and JsUnit target the same kind of test. Selenium seems more aimed at simulating mouse input, while JsUnit tests core logic, or am I wrong?

One framework that looks quite versatile, yet simple is Test.AnotherWay. This framework is for instance used in OpenLayers that has a thorough suite of tests with it. It's GUI isn't flashy, but very user friendly.

Several other javascript frameworks have unit testing frameworks Yahoo! UI has YUI test, Dojo has D.O.H. unit testing.

Maybe it's possible to reuse some of this via the Ext adapter interface. I haven't used any of them, so I unfortunately I don't know what their strengths and weaknesses are.