Testing Parallelization Improvements

1

I changed some of the testing infrastructure in #3620:

  • pytest and pytest-xdist are now dependencies.

  • There are 3 gradle tasks: test, testPython, doctest that now take a parallelism parameter given by -Dtest.parallelism=N. The default value for N is 2. Locally, I was able to use -Dtest.parallelism=4.

  • testPython is run via pytest. See pytest -h for a number of useful options such as to run single tests matching regexes (-k) and limit to N failures (--maxfail N). There’s also an option to enable the Python debugger to kick in on failure with --pdb (although I couldn’t get it to work for the doctests).

  • doctest is no longer run through Sphinx. Instead, it also uses pytest, so the same options for testPython also apply. I wrote a script python/hail/docs/doctest.sh which passes any extra arguments to the pytest command such as -k or --maxfail. This is useful for local debugging to not have to run the entire test suite.

  • the building of the docs has been consolidated into one file python/hail/docs/makeDocs.sh. makeDocs will test the tutorial while makeDocsNoTest will skip executing the tutorial, but will build the website.

New gotchas

  • When adding new .py files to python/hail/docs and its subdirectories, you must specifically add an option to ignore the file in makeDocs.sh with --ignore.

  • Instead of .. testsetup::, you have to add the setup to init in python/hail/conftest.py. Be aware that this is a global namespace for all tests.

  • Each method is executed in its own scope. So initializing variables in the class docstring won’t be visible in the rest of the methods. You have to add the variables you want to init in python/hail/conftest.py. Be aware that this is a global namespace for all tests.

  • The directives .. testsetup:: and .. doctest:: no longer work because Sphinx is no longer running doctest. I will have a future PR to remove these from the docstrings.

2

If you’re getting an error about -n not being recognized, then you probably need to install the pytest-xdist module. pip install pytest-xdist