PDA

View Full Version : [FIXED] STC should exit with non-zero exit (errorlevel) code when tests fail, and 0 for pass



warrenpostma
28 Sep 2016, 12:41 PM
All CI (build systems) like Jenkins, GitlabCI, and others, expect to run commands, and then examine their errorlevel (shell exit code) to see if the tests passed or not. This allows us to interrupt the flow of a build, and mark it red.

This is not working for us. STC will always exit(0) at the end and does not exit(1) or any other value when a test fails.

This needs to be fixed.


1. Write failing test.
2. Execute it with STC
3. On DOS command prompt type echo %ERRORLEVEL%
4. Expected: not 0
5. Actual 0

Corrollary is that if you write a PASSING test we expect 0 error code at exit.


As a secondary bug I would like to point out that when a test fails, I expect to see a line number of the test where the failure occurred, so that I don't have to guess which expect failed out of ten of them that are comparing a bool to another bool value. The message "Expected false to equal true" (an unfortunate practice inherited from Jasmine it seems) is useless, so unless we can add a .because('reason') at least I might expect the failed test to output:


chrome52 - Chrome 53.0.2785.116 / Windows 10.0
WorklistControllerSpec.js
onShowHideWorklistClick function: be able to show/hide Worklist view
Expected false to equal true (line 182)

Instead all we get is:

chrome52 - Chrome 53.0.2785.116 / Windows 10.0
WorklistControllerSpec.js
onShowHideWorklistClick function: be able to show/hide Worklist view
Expected false to equal true

marcelofarias
28 Sep 2016, 8:29 PM
Thanks for the report! I have opened a bug in our bug tracker.