Skip to content
Snippets Groups Projects
Commit ba32580b authored by jugglinmike's avatar jugglinmike Committed by Tom Care
Browse files

Remove Python-based test runner (#748) 

TC-39 agreed to remove this tool from the Test262 repository in May of
2016 [1]. It has since been re-published as a standalone project to
facilitate independent development by interested parties [2]. Remove it
from the Test262 repository.

[1] https://github.com/tc39/tc39-notes/blob/master/es7/2016-05/may-25.md
[2] https://github.com/test262-utils/test262-harness-py
parent cac55f29
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
tools/CONSOLE-RUNNER.md deleted 100644 → 0
+ 0
− 126
View file @ cac55f29
## Using the Console Test Runner
**Note** This utility has been deprecated. It will not be updated with new
features; future changes will be limited to bug fixes only.
The console test runner is used to test browserless implementations of ECMAScript, e.g., [v8](http://en.wikipedia.org/wiki/V8_(JavaScript_engine)), [node](http://en.wikipedia.org/wiki/Node.js), or [js24](http://packages.ubuntu.com/trusty/libmozjs-24-bin)
### Requirements
To use the `test262.py` runner, you must have the following:
* a checkout of the [test262 project](https://github.com/tc39/test262/)
* Python 2.7
* the Python YAML library [PyYAML](http://www.pyyaml.org)
* the javascript engine you intend to test (node, v8, etc.)
### Quick Start
To confirm the console test runner is working on a UNIX-like system
```
test262$ ./tools/packaging/test262.py --command "node" 7.2_A1.1_T1
ch07/7.2/S7.2_A1.1_T1 passed in non-strict mode
test262$
```
On a Windows system:
```
Z:\test262>tools\packaging\test262.py --command="node" 7.2_A1.1_T1
ch07\7.2\S7.2_A1.1_T1 passed in non-strict mode
Z:\test262>
```
If this does not work, see Troubleshooting (below)
### Options
Name | Action
-----|-------
-h, --help | displays a brief help message
--command=COMMAND | **required** command which invokes javascript engine to be tested
--tests=TESTS | path to the test suite; default is current directory
--cat | don't execute tests, just print code that would be run
--summary | generate a summary at end of execution
--full-summary | generate a longer summary with details of test failures
--strict_only | run only tests that are marked **onlyStrict**
--non_strict_only | run only tests that are marked **noStrict**
--unmarked_default=MODE | mode to use for tests that are not marked **onlyStrict** or **noStrict** ; MODE can be `strict` or `non_strict` or `both`
--logname=LOGNAME | write output to file (in addition to stdout)
--junitname=JUNITNAME | write test results to file in JUnit XML format
--loglevel=LOGLEVEL | set log level, primarily useful for debugging `test262.py`
--print-handle=FUNC | enable async test logging via javascript function e.g., `console.log`
### Usage Notes
Non-option arguments are used as filters to match test names. If no filters are found, the whole test suite is run.
Example | Result
-|-
test262.py --command="node" | run all tests
test262.py --command="node" ch07 ch11 | run tests from chapters 7 and 11
test262.py --command="node" 4.4 | run all tests with "4.4" in the name
The COMMAND argument can be a quoted string. This is useful when testing ECMAScript6 features in node, because node requires the command-line argument `--harmony` to enable ES6:
```
$ test262.py --command="node --harmony" es6
```
#### Async Tests
Async tests require a 'print' function to be supplied to the test runner. Here are some good defaults:
Engine | Filename | Print Function
-------|----------|---------------
V8/Node | node | console.log
V8/shell | shell | print
SpiderMonkey<sup>1</sup> | js | print
JavaScriptCore<sup>2</sup> | jsc | print
***Notes:***
1. As of 2014-Jul-23, SpiderMonkey does not support Promise in the `js` executable ([see bug 911216](https://bugzilla.mozilla.org/show_bug.cgi?id=911216) )
2. As of 2014-Jul-23, JavaScriptCore does not support Promise in the `jsc` executable
### Troubleshooting
#### ImportError: No module named yaml
On Windows, the message "No module named yaml" can mean that the PyYAML library is installed but not found. If you have this problem, you may be able to use `yaml` interactively from python:
```
Z:\Code\github\test262>python
ActivePython 2.7.5.6 (ActiveState Software Inc.) based on
Python 2.7.5 (default, Sep 16 2013, 23:16:52) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml
<module 'yaml' from 'C:\Python27\lib\site-packages\yaml\__init__.pyc'>
```
If you can load `yaml` interactively but get the `ImportError` when running `test262.py`:
```
Z:\Code\github\test262>tools\packaging\test262.py --command="node --harmony" --p
rint-handle="console.log" ch25
Traceback (most recent call last):
File "Z:\Code\github\test262\tools\packaging\test262.py", line 31, in <module>
from parseTestRecord import parseTestRecord, stripHeader
File "Z:\Code\github\test262\tools\packaging\parseTestRecord.py", line 20, in
<module>
import yaml
ImportError: No module named yaml
```
Then the fix is to explicitly set the PYTHONPATH environment variable. The location may vary from system to system, but it is typically `'C:\Python27\lib\site-packages`.
```
set PYTHONPATH=C:\Python27\lib\site-packages
```
tools/packaging/test/test_test262.py deleted 100644 → 0
+ 0
− 274
View file @ cac55f29
#!/usr/bin/env python
# Copyright 2014 by Sam Mikes. All rights reserved.
# This code is governed by the BSD license found in the LICENSE file.
import unittest
import sys
import os
import cStringIO
from functools import wraps
sys.path.insert(0, "..")
import test262
class TestTest262(unittest.TestCase):
def test_that_tests_run(self):
self.assertEqual(1 + 2, 3)
class MockTest(object):
def __init__(self, name, negative):
self.name = name
self.negative = negative if negative else False
self.strict_mode = False
def GetName(self):
return self.name
def IsNegative(self):
return self.negative
def GetMode(self):
if self.strict_mode:
return "strict mode"
return "non-strict mode"
class MockResult(object):
def __init__(self, case):
self.case = case
class TestTestSuite(unittest.TestCase):
def test_that_tests_run(self):
self.assertEqual(1 + 2, 3)
def test_create_test_suite(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
self.assertNotEqual(test_suite, None)
def test_summary(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
progress = test262.ProgressIndicator(100)
progress.succeeded = 98
progress.failed = 2
result = mute(True)(test_suite.PrintSummary)(progress, None)
self.assertEqual("""
=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
""", result)
def test_summary_logfile(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
progress = test262.ProgressIndicator(100)
progress.succeeded = 98
progress.failed = 2
fake_log = cStringIO.StringIO()
test_suite.logf = fake_log
result = mute(True)(test_suite.PrintSummary)(progress, True)
expected_out = """
=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
"""
expected_log = """=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
"""
self.assertEqual(expected_out, result)
self.assertEqual(expected_log, fake_log.getvalue())
def test_summary_withfails(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
progress = test262.ProgressIndicator(100)
progress.succeeded = 98
progress.failed = 2
progress.failed_tests = [
MockResult(MockTest("foo", False)),
MockResult(MockTest("bar", True))
]
result = mute(True)(test_suite.PrintSummary)(progress, None)
self.assertEqual("""
=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
Failed Tests
foo in non-strict mode
Expected to fail but passed ---
bar in non-strict mode
""", result)
def test_summary_withfails_andlog(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
progress = test262.ProgressIndicator(100)
progress.succeeded = 98
progress.failed = 2
progress.failed_tests = [
MockResult(MockTest("foo", False)),
MockResult(MockTest("bar", True))
]
fake_log = cStringIO.StringIO()
test_suite.logf = fake_log
expected_out = """
=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
Failed Tests
foo in non-strict mode
Expected to fail but passed ---
bar in non-strict mode
"""
expected_log = """=== Summary ===
- Ran 100 tests
- Passed 98 tests (98.0%)
- Failed 2 tests (2.0%)
Failed Tests
foo in non-strict mode
Expected to fail but passed ---
bar in non-strict mode
"""
result = mute(True)(test_suite.PrintSummary)(progress, True)
self.assertEqual(expected_out, result)
self.assertEqual(expected_log, fake_log.getvalue())
def test_summary_success_logfile(self):
test_suite = test262.TestSuite(".",
False,
False,
False,
None)
progress = test262.ProgressIndicator(100)
progress.succeeded = 100
progress.failed = 0
fake_log = cStringIO.StringIO()
test_suite.logf = fake_log
result = mute(True)(test_suite.PrintSummary)(progress, True)
expected_out = """
=== Summary ===
- Ran 100 tests
- All tests succeeded
"""
expected_log = """=== Summary ===
- Ran 100 tests
- All tests succeeded
"""
self.assertEqual(expected_out, result)
self.assertEqual(expected_log, fake_log.getvalue())
def test_percent_format(self):
self.assertEqual(test262.PercentFormat(1, 100), "1 test (1.0%)")
self.assertEqual(test262.PercentFormat(0, 100), "0 tests (0.0%)")
self.assertEqual(test262.PercentFormat(99, 100), "99 tests (99.0%)")
# module level utility functions
# copied from https://stackoverflow.com/questions/2828953/silence-the-stdout-of-a-function-in-python-without-trashing-sys-stdout-and-resto
def mute(returns_output=False):
"""
Decorate a function that prints to stdout, intercepting the output.
If "returns_output" is True, the function will return a generator
yielding the printed lines instead of the return values.
The decorator litterally hijack sys.stdout during each function
execution for ALL THE THREADS, so be careful with what you apply it to
and in which context.
>>> def numbers():
print "42"
print "1984"
...
>>> numbers()
42
1984
>>> mute()(numbers)()
>>> list(mute(True)(numbers)())
['42', '1984']
"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
saved_stdout = sys.stdout
sys.stdout = cStringIO.StringIO()
try:
out = func(*args, **kwargs)
if returns_output:
out = sys.stdout.getvalue()
finally:
sys.stdout = saved_stdout
return out
return wrapper
return decorator
if __name__ == '__main__':
unittest.main()
tools/packaging/test262.py deleted 100755 → 0
+ 0
− 646
View file @ cac55f29
This diff is collapsed.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment