diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bad1b9c0ac1cb600929a05d0348afdbee7204c38..0cc8bad5348f4af0737f804c10c1dd398b4cfb2b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,4 +1,92 @@ +## Using the Console Test Runner + +The console test runner is used to test browserless implementations of ECMAScript, e.g., [v8](http://en.wikipedia.org/wiki/V8_(JavaScript_engine), [node](http://en.wikipedia.org/wiki/Node.js), or [js24](http://packages.ubuntu.com/trusty/libmozjs-24-bin) + +### Requirements + +To use the `test262.py` runner, you must have the following: + + * a checkout of the [test262 project](https://github.com/tc39/test262/) + * Python 2.7 + * the Python YAML library [PyYAML](http://www.pyyaml.org) + * the javascript engine you intend to test (node, v8, etc.) + +### Quick Start + +To confirm the console test runner is working on a UNIX-like system + +``` +test262$ ./tools/packaging/test262.py --command "node" 7.2_A1.1_T1 +ch07/7.2/S7.2_A1.1_T1 passed in non-strict mode + +test262$ +``` + +On a Windows system: + +``` +Z:\test262>tools\packaging\test262.py --command="node" 7.2_A1.1_T1 +ch07\7.2\S7.2_A1.1_T1 passed in non-strict mode + + +Z:\test262> +``` + +### Options + +Name | Action +-----|------- +-h, --help | displays a brief help message +--command=COMMAND | **required** command which invokes javascript engine to be tested +--tests=TESTS | path to the test suite; default is current directory +--cat | don't execute tests, just print code that would be run +--summary | generate a summary at end of execution +--full-summary | generate a longer summary with details of test failures +--strict_only | run only tests that are marked **@onlyStrict** +--non_strict_only | run only tests that are marked **@noStrict** +--unmarked_default=MODE | mode to use for tests that are not marked **@onlyStrict** or **@noStrict** ; MODE can be `strict` or `non_strict` or `both` +--logname=LOGNAME | write output to file (in addition to stdout) +--junitname=JUNITNAME | write test results to file in JUnit XML format +--loglevel=LOGLEVEL | set log level, primarily useful for debugging `test262.py` +--print-handle=FUNC | enable async test logging via javascript function e.g., `console.log` + +### Usage Notes + +Non-option arguments are used as filters to match test names. If no filters are found, the whole test suite is run. + +Example | Result +-|- +test262.py --command="node" | run all tests +test262.py --command="node" ch07 ch11 | run tests from chapters 7 and 11 +test262.py --command="node" 4.4 | run all tests with "4.4" in the name + +The COMMAND argument can be a quoted string. This is useful when testing ECMAScript6 features in node, because node requires the command-line argument `--harmony` to enable ES6: + +``` +$ test262.py --command="node --harmony" es6 +``` + + ## Test262 Authoring Guidelines + +### Test Case Names + +These are guidelines. There is a substantial amount of variation in existing test names. + +Test cases are often named by the portion of the spec or draft spec they reference. A test for ECMAScript 5.1 section [8.7.2 PutValue](http://www.ecma-international.org/ecma-262/5.1/#sec-8.7.2) should be placed in the directory `test262/suite/ch08/8.7/8.7.2/` and have a filename which contains the substring `8.7.2`. A test for ECMAScript 6.0 draft section [25.4.4.1 Promise.all](https://people.mozilla.org/~jorendorff/es6-draft.html#sec-promise.all) should be placed in directory `test262/suite/es6/ch25/25.4/25.4.4/25.4.4.1` and have a filename that contains the substring `25.4.4.1`. + +One common convention is to break down a section into a number of testable assertions (A) and to test each assertion with one or more test cases (T). For example, section 25.4.4.1 of ES6 could be broken down like this: + + * A1 Promise.all is a function of one argument + - A1.1 Promise.all is callable + - A1.2 Promise.all expects 1 argument + * A2 Promise.all([]) is a Promise immediately resolved with [] + - A2.1 Promise.all([]) returns a Promise + - A2.2 Promise.all([]) is resolved immediately + - A2.3 Promise.all([]) is resolved with a new empty array + +with test case `S25.4.4.1_A1.1_T1.js` being the first test for assertion A1.1, and `S25.4.4.1_A2.3_T3.js` being the third test for assertion A2.3. + ### Test Case Style A test file has the following style format: ```javascript @@ -11,23 +99,30 @@ A test file has the following style format: [Test Code] ``` + ### Tags Test262 officially supports the following tags: **@description**, **@negative**, **@path** & **@author** + ##### @description This tag is used to describe the purpose of a particular testcase. Eg: Insert <LS> between chunks of one string + ##### @negative [.] \(not supported by Python console harness) Negative means the test will throw an error and given the error there will be a string comparison on the error message. It has an optional second parameter (it's a *regex-dot* if not provided) that will be used to compare. For best practices on how to use negative tag please see Handling Errors and Negative Test Cases. + ##### @path -This tag is used by the JSON packaging. It is not needed to manually enter this tag. +This tag is used by the JSON packaging. Do not manually enter this tag. + ##### @author This tag is used to identify the author of a test case. It's optional. -There are also couple other tags which are used only by the Python runner: **@onlyStrict**, **@noStrict** +Some tags which are used only by the Python runner: **@onlyStrict**, **@noStrict** + ##### @onlyStrict -Will run the test only in strict mode +Will only run the test in strict mode + ##### @noStrict -Will not run the test in strict mode +Will only not run the test in non-strict mode ### Handling Errors and Negative Test Cases The following patterns are considered the best practice: @@ -64,7 +159,60 @@ The example uses ReferenceError however it's also possible to use any of the fol throw "NotEarlyError"; [test code] ``` + There are *very* few cases where a syntax error is **not** an early error. In those cases use the Runtime Error pattern but wrap the test code in an eval statement. Be careful, eval code is not global code! + +### Writing Asynchronous Tests + +An asynchronous test is any test that includes the string `$DONE` anywhere in the test file. The test runner checks for the presence of this string; if it is found, the runner expects that the `$DONE()` function will be called to signal test completion. + + * If the argument to `$DONE` is omitted, is `undefined`, or is any other falsy value, the test is considered to have passed. + + * If the argument to `$DONE` is a truthy value, the test is considered to have failed and the argument is displayed as the failure reason. + +A common idiom when writing asynchronous tests is the following: + +```js +var p = new Promise(function () { /* some test code */ }); + +p.then(function checkAssertions(arg) { + if (!expected_condition) { + $ERROR("failure message"); + } + +}).then($DONE, $DONE); +``` + +Function `checkAssertions` implicitly returns `undefined` if the expected condition is observed. The return value of function `checkAssertions` is then passed to the first function of the final `then` call, resulting in a call to `$DONE(undefined)`, which signals a passing test. + +If the expected condition is not observed, function `checkAssertions` throws a `Test262Error` via function $ERROR. This is caught by the Promise and passed to the second function in the call -- which is also `$DONE` -- resulting in a call to `$DONE(error_object)`, which signals +a failing test. + +#### Checking Exception Type and Message in Asynchronous Tests + +This idiom can be extended to check for specific exception types or messages: + +```js +p.then(function () { + // some code that is expected to throw a TypeError + + return "Expected exception to be thrown"; +}).then($DONE, function (e) { + if (!e instanceof TypeError) { + $ERROR("Expected TypeError but got " + e); + } + + if (!/expected message/.test(e.message)) { + $ERROR("Expected message to contain 'expected message' but found " + e.message); + } + +}).then($DONE, $DONE); + +``` + + ### Other Features + #### $INCLUDE(fileName) method -$INCLUDE (not supported by Python console harness) will load an external Javascript file in the same context before executing a test. In most cases usage of this method should be avoided. It's a good practice only when a large amount of tests need a special check that's not provided by the default harness. If that's the case, please explain the use case in detail. \ No newline at end of file + +$INCLUDE will load an external Javascript file in the same context before executing a test. In most cases usage of this method should be avoided. It's a good practice only when a large amount of tests need a special check that's not provided by the default harness. If that's the case, please explain the use case in detail.