Writing tests#
To write a test for a new feature, copy one of the existing parameter files in
the tests/
folder in the ASPECT source
directory, or simply any other parameter file, modify it to use the new
feature, check that the new feature does what it is supposed to do, and then
just add the parameter file to the tests directory. You will then need to add
another folder to that directory that is named exactly like the parameter
file, and add the model output files that prove that the feature is working
(usually, these are the log file and the statistics file, and you will have to
rename log.txt
to screen-output
for historical reasons). The test and
output files should be as small and quick to run as possible. If you need to
include graphical output to test your feature, you will have to use the
gnuplot output format, so that the tester can compare the actual numbers (in
the vtu format, the output files are compressed, and can not be compared using
Numdiff). An easy way to create all of the files you need is to copy the
folder of an existing test and rename it to the name of your parameter file.
To actually run the test, you have to go to your ASPECT build directory and run
make setup_tests
so that your new test is added to the test suite. Then you can run it by executing
ctest -R name_of_your_test -V
and you will get an output telling you if the test has passed or failed (and
why it failed). If you have just copied the output files of a different test
in the tests/
directory to make your test, you of course expect your test to
fail. In this case, the output you see should contain a line that starts with
******* Check
and then just shows two paths. Those two paths are the one
where the output files of the test are located (the ones you just created by
running the test) and where the reference output of the test is located (the
one you created by copying an existing test). So you can copy this whole line
and replace ******* Check
by cp
to copy the output you just created over
the reference output. Of course, you should only do that after you have made
sure that these output files show that the feature you want to test is working
as expected.
When you make a new test part of a pull request on GitHub, then as explained
above that will lead to a run of all tests – including your new one
– on a “reference machine.” The reference machine that runs
the tests may of course produce slightly different results than the machine on
which a pull request was developed and from which the output was taken. If
this has been confirmed to be the source of a failed test run, a file that
contains the differences between the test output you submitted as part of your
pull request and the “reference” tester output will be available
from GitHub (You will have to click on the link labelled “Details”
next to the line that tells you if tests have failed; for the jenkins tester
that will bring you to a new page, where you have to go to the
“Artifacts” tab in the top right corner. Depending on the tester,
the file might be called changes-gcc.diff
or changes-test-results.diff
).
To use this file to update your test output, you will have to download it and
put it into your top-level ASPECT directory.
There you can apply the .diff file using:
git apply changes-test-results.diff
This will update your test output so that it matches the results from the official tester.
On the other hand, if a change leads to even a single existing test failing on that system, then we know that some more investigation as to the causes is necessary.