Testing

We maintain a series of tests in Trident to make sure the code gives consistent results and to catch accidental breakages in our source code and dependencies. These tests are run by Travis automatically and regularly to assure consistency in functionality, but you can run them locally too (see below). The tests consist of a mix of unit tests (tests to assure Trident functions don’t actively fail) and answer tests (tests comparing newly generated results against some old established results to assure consistency).

Running the Test Suite

Running the test suite requires a version of Trident installed from source (see Step 2: Install Trident).

The tests are run using the pytest Python module. This can be installed with conda or pip.

$ conda install pytest

The test suite requires a number of datasets for testing functionality. Trident comes with a helper script that will download all the datasets and untar them. Before running this, make sure you have the answer_test_data_dir variable set in your config file (see Step 3: Get Ionization Table and Verify Installation). This variable should point to a directory where these datasets will be stored. The helper script is located in the tests directory of the Trident source.

$ cd tests
$ python download_test_data.py

If this is your first time running the tests, then you need to generate a “gold standard” for the answer tests. Follow Generating Gold Standard Answer Test Results for Comparison before continuing with running the tests, otherwise your answer tests will fail.

Make sure you’re on the desired version of yt and trident that you want to test and use (usually the tip of the development branch i.e., main).

$ export TRIDENT_GENERATE_TEST_RESULTS=0
$ cd /path/to/yt/
$ git checkout main
$ pip install -e .
$ cd /path/to/trident
$ git checkout main
$ pip install -e .

The test suite is run by calling py.test from within the tests directory.

$ cd tests
$ py.test
============================= test session starts ==============================
platform darwin -- Python 3.6.0, pytest-3.0.7, py-1.4.32, pluggy-0.4.0
rootdir: /Users/britton/Documents/work/yt/extensions/trident/trident, inifile:
collected 52 items

test_absorption_spectrum.py ..........
test_download.py .
test_generate.py .
test_instrument.py .
test_ion_balance.py ............
test_light_ray.py .....
test_line_database.py .......
test_lsf.py ....
test_pipelines.py ...
test_plotting.py .
test_ray_generator.py .
test_spectrum_generator.py ......

========================= 52 passed in 117.32 seconds ==========================

If a test fails for some reason, you will be given a detailed traceback and reason for it failing. You can use this to identify what is wrong with your source or perhaps a change in the code of your dependencies. The tests should take around ten minutes to run.

Generating Gold Standard Answer Test Results for Comparison

In order to assure the Trident codebase gives consistent results over time, we compare the outputs of tests of new versions of Trident against an older, vetted version of the code we think gives accurate results. To create this “gold standard” result from the older version of the code, you must roll back the Trident and yt source back to the older “trusted” versions of the code. You can find the tags for the most recent trusted versions of the code by running gold_standard_versions.py and then rebuilding yt and Trident with these versions of the code. Lastly, set the TRIDENT_GENERATE_TEST_RESULTS environment variable to 1 and run the tests:

$ cd tests
$ python gold_standard_versions.py

Latest Gold Standard Commit Tags
yt = 953248239966
Trident = test-standard-v2

To update to them, `git checkout <tag>` in appropriate repository

$ cd /path/to/yt
$ git checkout 953248239966
$ pip install -e .
$ cd /path/to/trident
$ git checkout test-standard-v2
$ pip install -e .
$ export TRIDENT_GENERATE_TEST_RESULTS=1
$ cd tests
$ py.test

The test results should now be stored in the answer_test_data_dir that you specified in your Trident configuration file. You may now run the actual tests (see Running the Test Suite) with your current version of yt and Trident comparing against these gold standard results.

The Tests Failed – What Do I Do?

If the tests have failed (either locally, or through the automatically generated test from Travis), you want to figure out what caused the breakage. It was either a change in trident or a change in one of Trident’s dependencies (e.g., yt). So first examine the output from py.test to see if you can deduce what went wrong.

Sometimes it isn’t obvious what caused the break, in which case you may need to use git bisect to track down the change, either in Trident or in yt. First, start with the tip of yt, and bisect the changes in Trident since its gold standard version (see below). If that doesn’t ID the bad changeset, then do the same with yt back to its gold standard version. Once you have identified the specific commit that caused the tests to break, you have to identify if it was a good or bad change. If the unit tests failed and some functionality no longer works, then it was a bad, and you’ll want to change the code that caused the break. On the other hand, if the answer tests changed, and they did so because of an improvement to the code, then you’ll simply want to go about Updating the Testing Gold Standard.

Updating the Testing Gold Standard

Periodically, the gold standard for our answer tests must be updated as bugs are caught or new more accurate behavior is enabled that causes the answer tests to fail. The first thing to do is to identify the most accurate version of the code (e.g., changesets for yt and trident that give the desired behavior). Tag the Trident changeset with the next gold standard iteration. You can see the current iteration by looking in the .travis.yml file at the TRIDENT_GOLD entry–increment this and tag the changeset. Update the .travis.yml file so that the YT_GOLD and TRIDENT_GOLD entries point to your desired changeset and tag. You have to explicitly push the new tag (hereafter test-standard-v3) to your repository (here: origin. Issue a pull request.

$ git tag test-standard-v3 <trident-changeset>
$ ... edit .travis.yml files to update YT_GOLD=<yt changeset>
$ ... and TRIDENT_GOLD=test-standard-v3
$ git add .travis.yml
$ git commit
$ git push origin test-standard-v3
$ <MAKE PULL REQUEST>

Once the pull request has been accepted, someone with admin access to the main trident repository (here upstream) will have to push the gold standard tag.

$ git push upstream test-standard-v3

Lastly, that person will have to also clear Travis’ cache, so that it regenerates new answer test results. This can be done manually here: https://travis-ci.org/trident-project/trident/caches .