For asynchronous tests, the contract between test file and test runner
is implicit: runners are expected to inspect the source code for
references to a global `$DONE` identifier.

Promote a more explicit contract between test file and test runner by
introducing a new frontmatter "tag", `async`. This brings asynchronous
test configuration in-line with other configuration mechanisms and also
provides a more natural means of test filtering.

The modifications to test files was made programatically using the
`grep` and `sed` utilities:

    $ grep "\$DONE" test/ -r --files-with-match --null | \
        xargs -0 sed -i 's/^\(flags:\s*\)\[/\1[async, /g'
    $ grep "\$DONE" test/ -rl --null | \
        xargs -0 grep -E '^flags:' --files-without-match --null | \
        xargs -0 sed -i 's/^---\*\//flags: [async]\n---*\//'

Adds a `-j`/`--workers-count` parameter to `tools/packaging/test262.py`, defaulting to `[number of cores] - 1`.

Speeds up running the test suite by about ~3x on my 4-core machine, with the SpiderMonkey shell. This could certainly be optimized more by just appending test results to per-thread lists and merging them at the end, but it's better than nothing.

Some tests involving the directive prologue are invalidated by source
text transformations that insert executable code in the beginning of the
script. Implement a `raw` flag that allows these tests to opt-out of
this transformation. Update the relevant tests to use this flag (and
remove references to globals only available when code is injected).

Update the Python runner accordingly:

- Do not run tests marked as "raw" in strict mode
- Reject invalid test configurations

Update the browser runner accordingly:

- Do not modify the script body of tests marked as "raw"

The "monkeyYaml" parser is intended to serve as a lightweight fallback
to Python's standard YAML parser in contexts where the latter is not
available. Any intentionally-simplified implementation will necessarily
exhibit non-standard behavior for different input, so not all input
accepted by the standard parser will be accepted by "monkeyYaml". If
loaded exclusively in fallback situations, these edge cases can only be
identified (and debugged) in the environments that require the fallback.
This has allowed developers to unknowingly author tests that cause
errors.

Update the test runner to use "monkeyYaml" in all cases, ensuring more
consistent behavior across contexts and precluding this class of
regression.

Fixes #345

Some JavaScript source files are only relevant in the context of the
Test262 website. They should not be explicitly included by individual
tests, so their presence in the `harness/` directory alongside "include"
files is misleading.

Move the scripts to a location within the `website/` directory to
better-reflect their intended use. Update the relevant HTML templates
with the new locations.

Although test files once expressed dependencies on external files using
a global `$INCLUDE` function, that pattern was removed in favor of
declarative meta-data [1].

Remove the associated logic from the Python runner and the browser.

[1] See commit d4354d14.

Since the Python runner was updated to include `assert.js` in all tests
unconditionally, a number of tests have been written that implicitly
rely on its presence. The browser runner does not currently provide this
file's contents to these tests, so they fail unconditionally.

Update the browser runner to inject that file's contents into every test
context.

Note: the existing approach to file retrieval (namely loading via
synchronous XHR requests) is inefficient and deprecated in some
browsers. It is honored here for the sake of consistency and to minimize
the changeset necessary to fix the browser runner.

monkeyYaml didn't split lines correctly leading to \r in resulting
values.

Fixes #295

Test262 now accepts community contributions. Update the copy on the
website to reflect this.

additional test, tests should have unique names
improve python style thx to arv

Jenkins. Wrap XML output correctly and
post-process failure messages to avoid invalid XML characters.

Presumably since Python is whitespace-sensitive, this is actually quite important?

add test to repro #114
fix issue #114
factor out PercentFormat function
new fn write; start factoring summary output
fix tests to match
consolidate "Failed Tests" message
add test for successful run
consolidate all_succeeded message
consolidate summary line
consolidate ntests
failing negative tests
squeeze whitespace

use imp to import monkeyYaml locally
make monkeyYaml the backup again

parseTestRecord.py:
- recover from ImportError when YAML not defined
- use monkeyYaml as backup

monkeyYaml.py:
- parser for subset of YAML used in test262 frontmatter

test_monkeyYaml.py:
- unit tests

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

--- changes suggested by @anba

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)
correct test descriptions to expect ReferenceError only
enforce parsing as block statement by adding `;`
remove needless include of $FAIL.js

effectively, a rebase of all changes onto master

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Merge branch 'saved-bestPractice-negative' into negative-addErrorName-work

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    incorporate changes made on PR branch

    re-remove added NotEarlyError

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    Conflicts:
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch12/12.1/S12.1_A4_T1.js
    	test/suite/ch12/12.5/S12.5_A2.js

    ch15: correct negative regexes

    ch10: avoid use of not-NotEarlyError regex

    ch14 error regexps

    ch13 error regexps

    ch12: negative error matching

    Merge branch 'console-runner-checkError' into negative-addErrorName-work

    test262.py: check negative tests with regex

    implement checking of negative tests =>

    negative: SyntaxError

    means that /SyntaxError/ must match stderr
    or test is reported as failure

    Fixes #78

    expect real errors

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    --- changes suggested by @anba

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)
    correct test descriptions to expect ReferenceError only
    enforce parsing as block statement by adding `;`
    remove needless include of $FAIL.js

    Merge branch 'negative-addErrorName-work' into negative-addErrorName-bestPractice

    resolved Conflicts:
    	test/suite/ch07/7.9/S7.9_A5.7_T1.js
    	test/suite/ch11/11.13/11.13.1/S11.13.1_A2.1_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T1.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T10.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T11.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T2.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T3.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T4.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T5.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T6.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T7.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T8.js
    	test/suite/ch11/11.13/11.13.2/S11.13.2_A2.2_T9.js
    	test/suite/ch11/11.3/11.3.1/S11.3.1_A2.1_T3.js
    	test/suite/ch11/11.3/11.3.2/S11.3.2_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.4/S11.4.4_A2.1_T3.js
    	test/suite/ch11/11.4/11.4.5/S11.4.5_A2.1_T3.js
    	test/suite/ch12/12.5/S12.5_A2.js
    	test/suite/ch12/12.6/12.6.3/S12.6.3_A4_T2.js

    negative: second half of @anba notes

    correct test descriptions to expect ReferenceError only
    enforce parse as block statement by adding `;`
    remove needless include of $FAIL.js

    ch12: specify type

    negative: @anba fixes

    bestPractice - remove added NotEarlyError
    ch07,ch11 - expect ReferenceError (req'd by ES6)

    bestPractice: supply error names to negative

    pilot directory for replacing flags: [negative] with
    negative: errorname

    ch07-7.2,7.3: add error names
    ch07-rest: insert error names rest of ch07
    ch08: add expected error name
    ch11: add error name to negative
    ch12: put error name in negative

    ch13: add error name to negative
    only one test, did it manually

    ch15: add error name to negative:

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    expect a SyntaxError

    ch07: add error name to negative

    these tests used flags: \n - negative and so
    were not caught by the earlier naive grep for flags: [negative]

    ch15: add error name to negative:

    ch13: add error name to negative

    only one test, did it manually

    ch12: put error name in negative

    ch11: add error name to negative

    ch08: add expected error name
    ch07-rest: insert error names rest of ch07

    ch07-7.2,7.3: add error names

    negative-errorname: pilot

    pilot directory for replacing flags: [negative] with
    negative: errorname

    bestPractice: supply error names to negative

bestPractice: supply error names to negative

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names
ch07-rest: insert error names rest of ch07
ch08: add expected error name
ch11: add error name to negative
ch12: put error name in negative

ch13: add error name to negative
only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

bestPractice: supply error names to negative

negative-errorname: pilot

pilot directory for replacing flags: [negative] with
negative: errorname

ch07-7.2,7.3: add error names

ch07-rest: insert error names rest of ch07

ch08: add expected error name

ch11: add error name to negative

ch12: put error name in negative

ch13: add error name to negative

only one test, did it manually

ch15: add error name to negative:

ch07: add error name to negative

these tests used flags: \n - negative and so
were not caught by the earlier naive grep for flags: [negative]

expect a SyntaxError

negative: @anba fixes

bestPractice - remove added NotEarlyError
ch07,ch11 - expect ReferenceError (req'd by ES6)

ch12: specify type

negative: second half of @anba notes

correct test descriptions to expect ReferenceError only
enforce parse as block statement by adding `;`
remove needless include of $FAIL.js

expect real errors

ch12: negative error matching

ch13 error regexps

ch14 error regexps

add "description" header back

remove now-unused includes

remove needless include

sta.js:
slight change to Test262Error() semantics; message property now always set (default "")
make $ERROR a var
set $ERROR to function $ERROR so it can be overridden if needed
remove 2009 copyright in favor of 2012 copyright
REVERTED: remove never-used fn testFailed
testFailed actually used by $FAIL
remove obsolete fn $INCLUDE

ed.js:
remove commented-out obsolete code

test262.py:
remove always-included harness file that provides no functions used by any extant test
add comment line to nonstrict

cth.js: define `print` for node, cscript

use cth (console test harness) to define functions
for console runner.  V8, Spidermonkey (js) and JavaScriptCore (jsc)
provide a function `print`.  Provide a default `print` for node
and cscript

set print_handle to 'print' by default (can still override)
for cscript: wrap tests in try/catch so we get syntax errors

parseTestRecord: add support for YAML frontmatter
parseTestRecord: initial unit test for test record parser
parseTestRecord: refactor for testing

factor old parsing; add YAML parsing

runner: support "includes" from YAML frontmatter

support frontmatter "includes" in python runner
use test.includes if present instead of scanning test code with regex

harness: factor individual functions out into files

tools: handle YAML errors

tolerate missing keys in dictionary (flags, includes)
report filename when empty frontmatter block
new option --list-includes to test262.py

harness: factor helper functions into separate files

sth: remove extra close-paren (syntax error)

test_common: TDD; failing parse of YAML

common: use parseTestRecord (YAML-aware)

test262.py: only supply async helper scripts when test is async
sth.js: factor out function isAsyncTest()
timer.js: improve workaround for async tests when Promise is defined but setTimeout is noot

timer.js emulates setTimeout using Promise by doing a busy loop that checks
if `timeout` milliseconds have elapsed.  Modified check to (timeLeft > 0) instead
of (!timeLeft) to prevent infinite loop when check does not happen to run
at precise millisecond timeout expires.

Because test262.py did not support the $INCLUDE directive, some helper
scripts were added to every test -- notably testIntl, timer, and donePrintHandle
Now that $INCLUDE is supported, these can be dropped, speeding overall test run time

packager.py cannot run due to syntax errors in a few script files

packagerConfig: use git instead of hg
test262: use pop instead of delete to avoid throw if property missing
S22.1.2.1_T*: fix docString header comment: s/b /**
S22.1.2.1_T3: fix docString header comment: s/b /**, fix end of docstring * / => */

doneprintHandle.js: make $DONE accept any falsy argument as meaning 'pass'
PromiseHelper.js: checkSequence: new helper fn for async tests
.gitignore: port .hgignore to .gitignore syntax
test262.py: support $INCLUDE directive in python test runner

S25.4.4.1*: tests to cover Section 25.4.4.1, Promise.all( iterable )
A1.1: Promise.all is callable
A1.2: Promise.all expects 1 argument
A2.1: Promise.all([]) is a Promise
A2.2: Promise.all([]) is resolved immediately
A2.3: Promise.all([]) is resolved with a new empty array
A3.1: Promise.all expects an iterable argument

Fixed a property access in 6.4_c so that the error message shows the correct expected value.

- Removed a few old test cases that were redundant with new, more comprehensive ones.
- Added testIntl.js as standard include for all console tests in test262.py – see related bug 574.
- Added .jshintrc file for settings for the JSHint tool.

Added tests for requirements imposed on the built-in objects of the ECMAScript Internationalization API Specification by the introduction of chapter 15 of the ECMAScript Language Specification.
- Removed some old tests that were redundant with the new tests.
- Added testBuiltInObject.js as standard include for all console tests in test262.py – see related bug 574.

- Removed LocaleList tests; updated other tests so they don't depend on LocaleList.
- Updated tests so they no longer assume that the prototype object of a constructor is an instance of that constructor.
- Updated tests so that jshint is happy.
- Removed @path attributes from test files; updated comment in packager.py explaining why they're unnecessary.
- Removed "use strict" statements, which interfere with strict/non-strict testing.
- Removed testcase functions, which are unnecessary.