zope-testrunner - Man Page

legacy_module_filter

DEPRECATED: Prefer to use --module.

legacy_test_filter

DEPRECATED: Prefer to use --test.

-h,  --help

show this help message and exit

Options in this group are used to define which tests to run.

--package, --dir, -s PACKAGE

Search the given package's directories for tests. This can be specified more than once to run tests in multiple parts of the source tree. For example, if refactoring interfaces, you don't want to see the way you have broken setups for tests in other packages. You *just* want to run the interface tests. Packages are supplied as dotted names. For compatibility with the old test runner, forward and backward slashed in package names are converted to dots. (In the special case of packages spread over multiple directories, only directories within the test search path are searched. See the --path option.)

--module, -m MODULE

Specify a test-module filter as a regular expression. This is a case-sensitive regular expression, used in search (not match) mode, to limit which test modules are searched for tests. The regular expressions are checked against dotted module names. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can be specified multiple test-module filters. Test modules matching any of the test filters are searched. If no test-module filter is specified, then all test modules are used.

--test, -t TEST

Specify a test filter as a regular expression. This is a case-sensitive regular expression, used in search (not match) mode, to limit which tests are run. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The option can be specified multiple test filters. Tests matching any of the test filters are included. If no test filter is specified, then all tests are run.

--unit, -u

Run only unit tests, ignoring any layer options.

--non-unit, -f

Run tests other than unit tests.

--layer LAYER

Specify a test layer to run. The option can be given multiple times to specify more than one layer. If not specified, all layers are run. It is common for the running script to provide default values for this option. Layers are specified regular expressions, used in search mode, for dotted names of objects that define a layer. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). The layer named 'zope.testrunner.layer.UnitTests' is reserved for unit tests, however, take note of the --unit and non-unit options.

-a, --at-level AT_LEVEL

Run the tests at the given level. Any test at a level at or below this is run, any test at a level above this is not run. Level <= 0 runs all tests.

--all

Run tests at all levels.

--list-tests

List all tests that matched your filters. Do not run any tests.

--require-unique

Require that all test IDs be unique and raise an error if duplicates are encountered.

Reporting options control basic aspects of test-runner output

--verbose,  -v

Make output more verbose. Increment the verbosity level.

--quiet,  -q

Make the output minimal, overriding any verbosity options.

--progress,  -p

Output progress status

--no-progress

Do not output progress status. This is the default, but can be used to counter a previous use of --progress or -p.

--auto-progress

Output progress status, but only when stdout is a terminal.

--color,  -c

Colorize the output.

--no-color,  -C

Do not colorize the output. This is the default, but can be used to counter a previous use of --color or -c.

--auto-color

Colorize the output, but only when stdout is a terminal.

--subunit

Use subunit v1 output. Will not be colorized.

--subunit-v2

Use subunit v2 output. Will not be colorized.

--slow-test N

With -c and -vvv, highlight tests that take longer than N seconds (default: 10).

-1,  --hide-secondary-failures

Report only the first failure in a doctest. (Examples after the failure are still executed, in case they do any cleanup.)

--show-secondary-failures

Report all failures in a doctest. This is the default, but can be used to counter a default use of -1 or --hide-secondary-failures.

--ndiff

When there is a doctest failure, show it as a diff using the ndiff.py utility.

--udiff

When there is a doctest failure, show it as a unified diff.

--cdiff

When there is a doctest failure, show it as a context diff.

--ignore-new-thread REGEXP

If a thread with this name is left behind, don't report this at the end. This is a case-sensitive regular expression, used in match mode. This option can be used multiple times. If a thread name matches any of them, it will be ignored.

--buffer

Buffer the standard output and standard error streams during each test. Output during a passing test is discarded. Output during failing or erroring tests is echoed. This option is enabled by default if --subunit or --subunit-v2 is in use, to avoid corrupting the subunit stream.

--xml XMLOUTPUT

If given, XML reports will be written to the specified directory.

Analysis options provide tools for analysing test output.

--stop-on-error,  --stop,  -x

Stop running tests after first test failure or error.

--post-mortem,  --pdb,  -D

Enable post-mortem debugging of test failures

--gc,  -g GC

Set the garbage collector generation threshold. This can be used to stress memory and gc correctness. Some crashes are only reproducible when the threshold is set to 1 (aggressive garbage collection). Do "--gc 0" to disable garbage collection altogether. The --gc option can be used up to 3 times to specify up to 3 of the 3 Python gc_threshold settings.

--gc-option,  -G {DEBUG_INSTANCES,DEBUG_SAVEALL,DEBUG_COLLECTABLE,DEBUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_STATS,DEBUG_OBJECTS}

Set a Python gc-module debug flag. This option can be used more than once to set multiple flags.

--gc-after-test

After each test, call 'gc.collect' and record the return value *rv*; when *rv* is non-zero, output '!' on verbosity level 1 and '[*rv*]' on higher verbosity levels. On verbosity level 4 or higher output detailed cycle information.

--repeat,  -N REPEAT

Repeat the tests the given number of times. This option is used to make sure that tests leave their environment in the state they found it and, with the --report-refcounts option to look for memory leaks.

--report-refcounts,  -r

After each run of the tests, output a report summarizing changes in refcounts by object type. This option that requires that Python was built with the --with-pydebug option to configure.

--coverage COVERAGE

Perform code-coverage analysis, saving trace data to the directory with the given name. A code coverage summary is printed to standard out.

--profile {cProfile}

Run the tests under cProfiler and display the top 50 stats, sorted by cumulative time and number of calls.

--profile-directory PROF_DIR

Directory for temporary profiler files. All files named tests_profile.*.prof in this directory will be removed. If you intend to run multiple instances of the test runner in parallel, be sure to tell them to use different directories, so they won't step on each other's toes.

Setup options are normally supplied by the testrunner script, although they can be overridden by users.

--path PATH

Specify a path to be added to Python's search path. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run. This option also specifies directories to be searched for tests. See the search_directory.

--test-path TEST_PATH

Specify a path to be searched for tests, but not added to the Python search path. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run.

--package-path ARG ARG

Specify a path to be searched for tests, but not added to the Python search path. Also specify a package for files found in this path. This is used to deal with directories that are stitched into packages that are not otherwise searched for tests. This option takes 2 arguments specifying the path and the package. This option can be used multiple times to specify multiple search paths. The path is usually specified by the test-runner script itself, rather than by users of the script, although it can be overridden by users. Only tests found in the path will be run.

--tests-pattern TESTS_PATTERN

The test runner looks for modules containing tests. It uses this pattern to identify these modules. The modules may be either packages or python files. If a test module is a package, it uses the value given by the test-file-pattern to identify python files within the package containing tests.

--suite-name SUITE_NAME

Specify the name of the object in each test_module that contains the module's test suite.

--test-file-pattern TEST_FILE_PATTERN

Specify a pattern for identifying python files within a tests package. See the documentation for the --tests-pattern option.

--ignore_dir IGNORE_DIR

Specifies the name of a directory to ignore when looking for tests.

--shuffle

Shuffles the order in which tests are ran.

--shuffle-seed SHUFFLE_SEED

Value used to initialize the tests shuffler. Specify a value to create repeatable random ordered tests.

Other options

--version

Print the version of the testrunner, and exit.

-j PROCESSES

Use up to given number of parallel processes to execute tests. May decrease test run time substantially. Defaults to 1.

--keepbytecode,  -k

Normally, the test runner scans the test paths and the test directories looking for and deleting pyc or pyo files without corresponding py files. This is to prevent spurious test failures due to finding compiled modules where source modules have been deleted. This scan can be time consuming. Using this option disables this scan. If you know you haven't removed any modules since last running the tests, can make the test run go much faster.

--usecompiled

Normally, a package must contain an __init__.py file, and only .py files can contain test code. When this option is specified, compiled Python files (.pyc and .pyo) can be used instead: a directory containing __init__.pyc or __init__.pyo is also considered to be a package, and if file XYZ.py contains tests but is absent while XYZ.pyc or XYZ.pyo exists then the compiled files will be used. This is necessary when running tests against a tree where the .py files have been removed after compilation to .pyc/.pyo. Use of this option implies --keepbytecode.

--exit-with-status

DEPRECATED: The test runner will always exit with a status.