DISCUS-suite macro tests#
The folders 00.Introduction … 15.Timing hold example and test macros.
Most of them check their own result and set LTEST_MAC = TRUE when it is
correct. check.mac evaluates that flag.
There are two ways to run the tests.
1. Interactively: test_all.mac#
discus_suite -macro test_all.mac
Runs all checked macros in one session. The first failure prints
Failure, macro … did not succeed and stops. This is unchanged.
2. Unattended: tools/run_legacy_tests.py#
For CI and for checking a build. It runs every test listed in
manifest.toml in its own fresh copy of the test folder and its own
discus_suite process. It reports PASS / FAIL / ABORTED / TIMEOUT per test
and exits with code 1 if any test did not pass.
python3 tools/run_legacy_tests.py --exe /path/to/discus_suite -j 4
Useful options:
Option |
Meaning |
|---|---|
|
the |
|
run N tests in parallel (tests tagged |
|
only tests with one of these tags ( |
|
skip tests with these tags (default: |
|
only tests whose id matches (repeatable) |
|
machine-readable reports |
|
where copies and logs go; keep passing tests’ copies too |
|
tests tagged |
|
re-run a failed test tagged |
|
show what would run |
|
check that |
Without an X display it runs each test under xvfb-run (KUPLOT needs one).
On Debian/Ubuntu install xvfb and pgplot5.
Results of a run with -j 4 on 4 cores: 30 PASS, 1 RAN (03.pdf, no check
yet), 2 SKIPPED (04.sro2, 09.znse_pdf_mpi), about 80 s wall time.
How batch mode works#
The runner writes a small driver macro that sets LTEST_BATCH = TRUE and
LTEST_FILE before calling the test macro. In batch mode check.mac appends
PASS <name> or FAIL <name> to LTEST_FILE instead of stopping. The driver
writes END as its last line.
Result file |
Status |
|---|---|
expected number of |
PASS |
any |
FAIL |
no |
ABORTED (a command error ended the macro run; the first |
process killed after |
TIMEOUT |
|
RAN (the macro finished, nothing was verified) |
check.mac declares LTEST_BATCH itself, so a test macro run by hand still
behaves as before.
pytest#
tests/python/test_legacy_macros.py turns each manifest entry into a pytest
test, and also checks the manifest against test_all.mac:
DISCUS_SUITE=/path/to/discus_suite pytest tests/python -m "not slow"
Without DISCUS_SUITE only the manifest checks run.
3. Python: tests/python/test_suite_python.py, test_diffuse_api.py#
Run DISCUS in-process through the f2py module suite_python and check the
results as NumPy arrays, against the legacy RESULTS/ files and against
golden data in tests/golden/<id>/. test_diffuse_api.py tests the
diffuse package (python/diffuse/, see python/README.md) and the demo
script python/examples/powder_demo.py (needs matplotlib). Build with -DDIFFUSE_PYTHON=ON, then:
pip install -r tests/requirements.txt # gemmi, for independent references
DIFFUSE_PYTHON_PATH=/path/to/build/python pytest tests/python
Besides the API tests, test_wp3_structures.py and test_wp4_fourier.py
are the safety net for the WP3 (structures, I/O) and WP4 (single-crystal
Fourier) work: they compare DISCUS with gemmi’s space groups and form
factors and with physical rules, and hold golden data. Tests marked
xfail(strict=True) document known DISCUS bugs; they turn red when the bug
is fixed, so the mark can be removed. Without gemmi the cross-checks are
skipped.
Variable |
Meaning |
|---|---|
|
folder that holds |
|
fail instead of skip if the module cannot be imported (used in CI) |
|
write the golden files instead of comparing; review the diff before committing |
To add a test, use the suite fixture (the session from diffuse.session()) or
the diffuse functions directly. suite.run(commands, folder) runs
macro text with folder as working directory and raises MacroError if a
command fails; suite.value("n[1]") evaluates an expression;
suite.dataset(1) / suite.dataset_2d(1) return KUPLOT data set 1. Load
results into KUPLOT (kuplot, rese, load ...) to read them back.
compare.check_golden(id, name, r_max=..., axis..., result) handles golden
data. The Fortran state is global and shared by all tests in one pytest run,
so start each calculation with the relevant reset.
CI#
.github/workflows/tests.yml builds with the Python module and runs the
pytest tests, then the legacy tests tagged fast and mc, on every push to
claude-python-app (and claude/** branches) and on pull requests into
claude-python-app.
Adding a test#
Write the macro so that it sets
LTEST_MAC = TRUEwhen its result is correct (see02.Calculation_Scattering/1d.macfor a comparison againstRESULTS/).Add it to
test_all.mac(@macro argsfollowed by@../check.mac macro).Add a
[[test]]entry tomanifest.toml.Run
python3 tools/run_legacy_tests.py --check-test-alland the test itself with--id.
Plot macros must not need an interactive viewer: use program cif and a
plain run instead of program jmol and run plot:inter.