WP0 – Regression test baseline#
Part of the Python migration plan (DiffuseCode_Python_Migration_Plan.md, option B).
WP0 is the safety net: before any Fortran is wrapped or refactored, we need a test
suite that runs unattended, reports pass/fail reliably, and stores reference
(“golden”) results to compare against.
This document has three parts:
Where we are – what the existing
tests/folder covers and what happened when it was run headless on Linux.WP0 work plan – the tasks to turn it into an automated regression baseline.
Progress log – what has been done so far.
mixscat is deprecated and is left out of WP0 (and of the migration).
1. Where we are#
1.1 What is in tests/#
Folder |
Topic |
|
Macros with a self-check |
Reference data ( |
Run by |
|---|---|---|---|---|---|
00.Introduction |
Command-language basics |
5 |
0 |
– |
no |
01.Simulation |
Building structures, symmetry, shear |
19 |
6 |
– |
4 tests |
02.Calculation_Scattering |
1D/2D Fourier, lots, powder |
29 |
5 |
yes |
5 tests |
03.Pair_Distribution_Function |
11 |
0 |
– |
runs, no check |
|
04.SRO |
Short-range order via MMC |
138 |
6 |
yes |
4 tests (SRO_2 commented out) |
05.Stack |
Stacking faults |
48 |
4 |
yes |
4 tests |
06.Domains |
Domains |
32 |
4 |
yes |
4 tests |
07.Refinement |
DIFFEV/REFINE |
50 |
1 |
– |
1 test |
08.Nanoparticle_example |
Nanoparticles, powder, PDF |
41 |
3 |
yes |
2 tests |
09.Nanoparticle_refinement |
MPI refinement |
364 |
0 |
– |
1 shell script, does not run (see 1.3) |
11.Properties |
Atom properties |
7 |
1 |
– |
1 test |
12.Waves |
Modulation waves |
8 |
3 |
yes |
2 tests |
13.Analyze |
Neighbour/bond analysis |
10 |
1 |
yes |
1 test |
14.Decoration |
Surface decoration |
46 |
10 |
– |
2 tests |
15.Timing |
OpenMP benchmarks |
15 |
0 |
– |
no |
How a test works. test_all.mac changes into each folder, runs a macro with
parameters, then calls check.mac. Each macro sets LTEST_MAC = TRUE if its own
check passes. A check is either:
a geometric value (
blen,bang, atom counts), ora KUPLOT comparison against a stored file:
loadthe new result andRESULTS/<file>, thenrval(R-value) below a threshold, e.g.< 0.001for Fourier and powder,< 0.25for the Monte-Carlo SRO examples.
Total: 30 test invocations, about 3.5 minutes on 4 cores.
1.2 Headless run results (2026-11-04)#
Build: Ubuntu 24.04, gfortran 13.3, CMake 3.28, FFTW, HDF5, pgplot5 from the
Ubuntu repository, run under xvfb-run (virtual X display), no human input.
Run |
Result |
|---|---|
|
Stopped in 05.Stack after 4 sections: |
Each test run in isolation |
22 PASS, 7 ABORTED (all the same JMOL error), 1 ran without a check |
With JMOL lines disabled (this branch) |
All 13 sections SUCCESS, 200 s, exit code 0. CIF output files are still written; |
JMOL fix in this branch. program jmol first runs which jmol. If JMOL is
not installed it raises error -152, which aborts the whole macro run.
run plot:inter also raises -152 for anything other than JMOL. In the 9 plot
macros that test_all.mac reaches, both lines are now commented out and replaced
with program cif + run. These write the same CIF file through the same code
path (discus_plot.f90:1448), just without opening the viewer.
Files changed:
05.Stack/STACK/plot.mac, 05.Stack/STACK_FILE/plot.mac,
06.Domains/CUBE/plot.cif.mac, 08.Nanoparticle_example/{SIMPLE,ELLIPSOID,Q}/plot.cif.mac,
14.Decoration/Deco_01/plot.mac, 14.Decoration/Deco_02/plot.mac,
14.Decoration/Deco_02/clean_surface.mac.
Other tutorial macros still call JMOL interactively, but the suite does not run
them: 01.Simulation/plotcif.mac, 04.SRO/SRO_MOLECULES/plot_jmol.mac,
05.Stack/{STACK_DISTRIBUTION,WURTZITE}, 06.Domains/Molecules,
08.Nanoparticle_example/COMPLEX/main.mac, 11.Properties/{prop_ex_2,plot.cif}.mac,
15.Timing/OpenMP/plot.mac.
1.3 Problems found#
Test-harness problems
Errors are invisible to automation. A macro error ends
discus_suitewith exit code 0, and so does a failing check (stopincheck.mac). The only pass/fail signal is text on screen (“SUCCESS in …” / “Failure, macro …”). With macro echo on, the log also contains the source of both branches of everyif, so grepping for “Failure” finds false hits.One failure hides all later tests.
test_all.macis a single sequential macro, and the first error ends the run.Some sections report SUCCESS without testing anything:
03.Pair_Distribution_Function –
pdf.macruns but has no check.09.Nanoparticle_refinement –
system ./TEST.shfails with Permission denied (no execute bit in git), and needsmpiexecandDISCUS_NCPU. The section still prints SUCCESS.14.Decoration/Deco_01 –
all.machas no aggregated check.
Randomness is not controlled. The suite has a
seedcommand (seed <value>orseed <a>,<b>,…, group:<n>, seelib_f90/lib_f90.hlp). Only 12 macros use it (mostlydiffev_best.mac), and none of the tests intest_all.macdo. The checks therefore use loose tolerances (SRO correlation ±0.08, R < 0.25).seeddoes not currently make MMC runs reproducible. A quick check ran04.SRO/SRO_0/sro.0.mac 0.7,1.5twice withseed 1234, both at the suite prompt and directly beforemmc, withOMP_NUM_THREADS=1and 4. Every time the two runs gave different MC acceptance counts, and about half the lines ofexample0.strudiffered. This needs debugging (task 0.5a) before golden MC data can be exact.Generated output is committed next to inputs. Example:
02.Calculation_Scattering/example3.intesits besideRESULTS/example3.inte.tests/is 109 MB / 2,053 files, and it is hard to tell inputs, references and leftovers apart.Coverage gaps. A rough grep finds no use of these DISCUS commands in the test macros:
anis, append, asym, bragg, change, define, copy, demolecularize, interpret, inverse, kick, molecularize, patterson, perioditize, rmc, spacegroup, storage, super, switch, wyckoff. Some may be reached indirectly or under abbreviations; this needs confirming.
Build problems
DIFFUSE_GRAPHICS=OFFdoes not work:suite/prog/CMakeLists.txt:5always callsfind_package(PGPLOT REQUIRED).make installassumes PGPLOT’sgrfont.datandpgxwin_serverare inPGPLOT_DIR. On Debian/Ubuntu they are in/usr/lib/pgplot5and/usr/bin. The binary works fine from the build tree.The build requires an
FCenvironment variable even for re-running CMake.
Found while doing 0.2 / 0.3
Start-up race (fixed). On every start
discus_suitepings GitHub (check_githubinlib_f90/appl_unix.f90) and wrote the answer to a fixed file,/tmp/check_github.txt. Two processes starting at the same time could delete each other’s file, and one crashed with a Fortran runtime error. The file name now includes the process ID, like the other temporary files. The ping also means every start tries to reach the network; in CI it simply fails quickly.Macros override the thread count.
04.SRO/SRO_1/sro.1.macandSRO_3/sro.3.maccallset parallel, useomp:use, nthreads:all, which overridesOMP_NUM_THREADS. Next to other tests this over-commits the CPUs, and the OpenMP threads slow down by up to ~100× (2.4 s → 130–340 s). These tests are taggedexclusiveand the runner runs them alone.Endless prompt loop. When
check.maccallsstop(interactive mode) and stdin is at end of file,discus_suiteprintssuite >prompts forever (18 MB of log in 20 s). Batch mode never callsstop, and the runner’s timeout would catch it, but the input loop should end on EOF.
2. WP0 work plan#
Goal: one command (and one CI job) that builds DiffuseCode, runs every test in isolation, writes machine-readable results, and fails when a test fails. It also stores golden numerical outputs that later Python-API tests (WP3–WP8) can compare against.
Proposed layout#
tests/ # existing macro tests stay where they are
check.mac # [done] batch mode writes PASS/FAIL to a result file
test_all.mac # kept for interactive use (unchanged)
manifest.toml # [done] list of tests (TOML: read by Python's standard library)
README.md # [done] how to run the tests
golden/ # 0.6: [started] reference outputs, one folder per test id
python/ # pytest harness
conftest.py # [done] registers the manifest tags as markers
test_legacy_macros.py # [done] one test per manifest entry + manifest checks
compare.py # [started] R-value, golden .npz read/write (0.6)
test_suite_python.py # [done] in-process Python tests against RESULTS/ and golden/
tools/
run_legacy_tests.py # [done] runner -> console table, JUnit XML, JSON
.github/workflows/
tests.yml # [done] 0.9: build + pytest + legacy tests on claude-python-app
Tasks#
# |
Task |
Details |
Done when |
Effort |
|---|---|---|---|---|
0.1 |
Headless run |
JMOL disabled in the test plot macros (done in this branch). Run under |
|
done |
0.2 ✅ |
Machine-readable results |
Extend |
CI shows per-test pass/fail; process exits non-zero on any failure or abort |
2–3 d |
0.3 ✅ |
Test manifest |
Move the list of tests out of |
Harness and |
1–2 d |
0.4 |
Close the silent gaps |
Add a check to 03 PDF (compare G(r) against a new |
No section prints SUCCESS without a check |
2–3 d |
0.5a |
Debug |
Find out why |
Two runs with the same seed give identical |
1–3 d |
0.5 |
Reproducibility |
Once 0.5a is fixed, add |
Same seed gives the same output on the same build |
2 d |
0.6 |
Golden data capture |
For each test, copy key outputs (intensities, powder, PDF, 3D-PDF, final structure/CIF, histograms) into |
Every test has at least one golden numeric output |
3–4 d |
0.7 |
Coverage expansion |
Confirm the uncovered-command list from 1.3, then add short, deterministic tests for the commands that WP3–WP8 will wrap first: space-group expansion / |
All commands planned for WP3–WP8 have at least one test |
5–8 d |
0.8 |
Build fixes |
Make |
|
1–2 d |
0.9 ✅ |
CI |
GitHub Actions on |
Red/green status on every push to |
1–2 d |
0.10 |
Repo hygiene |
Run |
|
1 d |
Total: about 4–6 weeks for one developer. That fits the “months 0–3” slot in the plan, with room for coverage work alongside WP1.
Suggested order#
0.2 + 0.3 – reliable pass/fail first. Everything else depends on it.
0.9 – CI right away, so regressions are seen from day one.
0.4 + 0.5a + 0.5 – remove false SUCCESS, fix
seed, and make results reproducible.0.6 – golden data, the input for the Python-API work.
0.8 + 0.10 – build and repo clean-up, then turn CI to the no-graphics build.
0.7 – coverage growth, continuing in parallel with WP1–WP3.
Open questions for the team#
Is changing
check.mac’s behaviour (write a result instead ofstop) OK for interactive users oftest_all.mac? AnLTEST_BATCHvariable could switch between the two.Which tolerances count as “same physics” for MC-based tests, and who signs off on them?
Should golden data live in the repo (Git LFS) or in a separate data repository or release asset?
Is MPI coverage needed in CI, or is a nightly or manual run enough?
Reproducing the headless run#
sudo apt-get install -y gfortran cmake libfftw3-dev libhdf5-dev libreadline-dev \
libpng-dev libx11-dev zlib1g-dev pgplot5 xvfb
mkdir -p ../DiffuseBuild && cd ../DiffuseBuild && \
FC=$(which gfortran) cmake ../DiffuseCode -DCMAKE_BUILD_TYPE=Release -DPGPLOT_DIR=/usr/lib/pgplot5 && \
make -j"$(nproc)"
Unattended, with per-test results (from the DiffuseCode folder; the runner
copies each test, so tests/ stays clean):
python3 tools/run_legacy_tests.py --exe ../DiffuseBuild/suite/prog/discus_suite -j 4 --junit results.xml
Or the classic way, on a copy because the macros write output files into
tests/:
cp -r tests /tmp/dc_tests && cd /tmp/dc_tests && \
PGPLOT_DIR=/usr/lib/pgplot5 xvfb-run -a "$OLDPWD/../DiffuseBuild/suite/prog/discus_suite" -macro test_all.mac < /dev/null
3. Progress log#
0.2 + 0.3 – runner, batch mode and manifest#
tests/check.machas a batch mode. WithLTEST_BATCH = TRUEit appendsPASS <name>/FAIL <name>to the file inLTEST_FILEand carries on, judging each check on its own. Interactive behaviour is unchanged (test_all.macstill passes all 13 sections).check.macdeclaresLTEST_BATCHitself, so test macros run by hand still work.tests/manifest.tomllists 33 tests: the 31 macros run bytest_all.mac, plus04.sro2and09.znse_pdf_mpi, which are disabled with a reason.--check-test-allconfirms that it matchestest_all.mac.14.deco01_allnow counts its 7 internal checks; before, it had no overall check.tools/run_legacy_tests.py(standard library only):Runs each test in a fresh copy, in its own process group, under xvfb when there is no display. Runs tests in parallel (
-j) and runs tests taggedexclusivealone.Kills the whole process group on timeout, retries on another display if Xvfb cannot start, and writes JUnit XML and JSON.
Exits with 1 on FAIL, ABORTED or TIMEOUT.
tests/python/gives the same tests as pytest tests, plus the manifest checks (which run without a binary).tests/README.mdexplains how to run and add tests.lib_f90/appl_unix.f90: fixed the start-up race (finding 10).
Checked:
Run |
Result |
|---|---|
Full manifest, |
30 PASS, 1 RAN ( |
Full manifest, |
same result; 210 s |
Deliberate failures (wrong result, bad parameter, too few checks, 5 s timeout) |
FAIL, ABORTED (first error line reported), ABORTED, TIMEOUT; exit code 1; no processes left behind |
Display already taken |
retried on another display, PASS |
Runner and pytest at the same time |
all pass; no display clashes |
|
all 13 sections SUCCESS |
0.9 (and start of 0.6) – CI and Python-interface tests#
Build fix:
python/CMakeLists.txtlinkedexperi_allafterdiscus_all.experi_kdocallsexp3_pdf, so the f2py module built but failed on import withundefined symbol: __exp3_pdf_menu_MOD_exp3_pdf.experi_allnow comes first.A test helper wraps
suite_python(nowdiffuse.session(), see below).run()writes a driver macro thatcds into the test folder (the suite keeps its own working directory, soos.chdiris not enough) and setsr[499]on its last line. A command error ends the whole macro run, so a flag that is not set raisesMacroError.dataset()/dataset_2d()return KUPLOT data sets as NumPy arrays.tests/python/test_suite_python.py: expressions, error detection, crystal build, 1-D Fourier, 2-D Fourier and Debye powder, run in-process. The 1-D Fourier and powder results are compared with the legacyRESULTS/files (R < 1e-3; actual about 1e-6) and all three with golden data.tests/python/compare.pyandtests/golden/<id>/(.npz+meta.json): first golden data for02.1d,02.2d,02.powder. Rewrite withDIFFUSE_UPDATE_GOLDEN=1..github/workflows/tests.yml: Ubuntu 24.04, apt dependencies, build with-DDIFFUSE_PYTHON=ON, then pytest (DIFFUSE_REQUIRE_PYTHON=1, so a broken module fails instead of skipping), then the legacy runner with--tags fast,mc. JUnit XML and logs are uploaded.
Limits of the f2py module that the tests work around (to fix in WP2):
execute_macro reports errors only as text; set_value only works on
user-defined variables, not r[]; data come back as single precision.
Checked (gfortran 13, 4 cores):
Run |
Result |
|---|---|
|
9 passed (33 legacy macro tests skipped, no |
Same build, |
29 PASS, 1 RAN, 2 SKIPPED |
Demo API: python/diffuse/ (ahead of WP3/WP5)#
A first Python package for the demo, built on suite_python:
read_structure() (CIF, .cell, .stru, with supercells), Crystal.atoms(),
powder() returning NumPy arrays, PowderPattern.plot() with matplotlib,
and run_macro() for everything else. See python/README.md.
The test helper moved into the package (
diffuse/_session.py), and thesuitefixture now returnsdiffuse.session(). The session also captures DISCUS output (stdout/stderr go to a temporary file during each call, withGFORTRAN_UNBUFFERED_PRECONNECTED=yso nothing is flushed late), andMacroErrorcarries the***error lines.n[]andlat[]are only defined inside thediscussection; the read macro copies them tor[490]–r[496], which are reserved for the package (r[499]is the completion flag).Reading a CIF makes DISCUS write a
.strufile next to the input, so inputs are copied to a scratch folder first.python/examples/powder_demo.py: anatase fromanatase.cif(COD 1010942), X-ray vs neutron and crystal-size broadening, saved as a PNG. Peak positions were checked against the allowed I4₁/amd reflections ((101) strongest for X-rays at Q = 1.81 Å⁻¹).tests/python/test_diffuse_api.py: 12 tests (reading, supercells, errors, powder grid and (101) position, X-ray/neutron golden data, reloading an olderCrystal, plotting, the demo script). CI installs matplotlib.
Checked: pytest tests/python gives 21 passed (33 legacy macro tests skipped
without DISCUS_SUITE), about 2 s, with OMP_NUM_THREADS 1 and 4.
Flaky Monte-Carlo test in CI#
CI run 2 failed on 04.sro3 (FAIL from its own check) with exactly the code
that passed in run 1; locally it passed 8 times in a row. MMC does not use
the value set with seed (known bug, task 0.5a), so tests tagged mc are
random and checked against statistical limits (R < 0.38, correlations
within ±0.08), and they can fail by chance.
Until 0.5a is fixed, run_legacy_tests.py --retry-mc N re-runs a failed
mc test up to N times. A pass on retry is counted as PASS, with the
message flaky: passed on retry k; first run ... in the summary and the
JUnit report. A real regression fails every attempt, and the run still
fails. CI uses --retry-mc 2. Once seed works, the mc tests should get
fixed seeds and the retry should be removed.
0.5a – seed (first half fixed)#
Cause 1 (fixed): seed set only part of the generator state. DISCUS uses
the Fortran intrinsic random_number. gfortran’s state has 8 integers
(random_seed(size=...)). ini_ran_ix (lib_f90/random_state_mod.f90)
copied only the values the user gave (seed 1234 gives one) and left the
other words at their clock-based values from start-up, so every run was
different. The remaining words are now derived from the user’s values.
Cause 2 (open): OpenMP. With threads, MMC is still not reproducible.
random_number keeps one state per thread, and random_seed(put=...) only
resets the calling thread’s; the worker threads keep their own. The threads
also change the shared crystal while others evaluate their moves, so the
result depends on timing. A fix needs per-thread streams seeded from the
seed value, plus a fixed order for applying the moves.
Checked with 04.SRO/SRO_0/sro.0.mac 0.7,1.5, seed directly before the
macro, comparing example0.stru:
Setting |
Same seed twice |
Different seed |
|---|---|---|
before the fix, |
different |
– |
after the fix, |
identical |
different |
after the fix, |
different (cause 2) |
different |
The earlier test (section 1.3, finding 4) used OMP_NUM_THREADS=1, but the
SRO macros call set parallel, nthreads:all, which overrides it, so that
run was threaded. After the fix, pytest tests/python and the legacy runner
(--tags fast,mc) pass as before.
0.5 – seeded Monte-Carlo tests#
run_legacy_tests.py now runs every test tagged mc with seed 1234 (option
--seed, or a test’s own seed in the manifest) and with
OMP_THREAD_LIMIT=1. The limit also caps set parallel, nthreads:all in the
SRO macros (DISCUS reports “maximum threads 1”). --no-seed gives the old
behaviour. Seeded tests run on one thread, so the exclusive tests no longer
need to run alone.
Checked by running all 16 enabled mc tests twice (--keep) and comparing
every output file: byte-identical in both runs, apart from the generated
driver macro (it contains the work-folder path) and one 04.sro3 run, see
below. All pass with seed 1234.
New finding: rare broken Fourier result in 04.sro3. In one of the two
runs, 04.sro3 aborted. Its MC structure (example3.stru) was identical
to the other run’s, but the Fourier step (fourier.mac: random lots
plus set technique:nufft) wrote 1530 Infinity values and blocks of zeros
to example3.inte, and NaN to example3.3DPDF; kex3.mac then failed on
zmax[3]. 14 more seeded runs (6 serial, 8 with 4 at a time) all gave the
same correct result, so it is rare (1 in about 17 so far). It may also
explain the original CI failure of 04.sro3 (run 2). Needs its own
investigation: uninitialised memory or a race in the lots/NUFFT path are
candidates. Until then, CI keeps --retry-mc 2.
Demo API: single-crystal Fourier and PDF#
diffuse.fourier() (line or plane in reciprocal space, optional removal of
the average structure for diffuse scattering) and diffuse.pdf() (G(r),
periodic crystal or finite particle), with result objects FourierMap and
PDFPattern, plots, and two gallery examples (fourier_demo.py,
pdf_demo.py). Tests: tests/python/test_fourier_pdf.py.
Checked against independent references:
Check |
Result |
|---|---|
|
identical (R ≤ 1e-9). The reference files come from |
|
R = 1.7e-7 |
|
identical |
Orientation: a row of atoms along a, neutrons, no ADPs |
intensity varies only along the first index (h), exactly constant along k |
Finite particle ( |
G(r) = 0 beyond the particle; a periodic crystal keeps its peaks |
Bug found and fixed: DISCUS’s cd changes the working directory of the
whole Python process, so after any diffuse call, relative paths in the
user’s script pointed somewhere else. Session.run() now restores the
caller’s working directory (test test_working_directory_kept).
0.6 + 0.7 for WP3 and WP4 – structures and single-crystal Fourier#
Golden data and coverage for the first two work packages that will be
rewired, checked against independent references wherever possible (new
tests/requirements.txt: gemmi).
tests/python/test_wp3_structures.py (26 tests):
Space-group expansion of 10 structures (monoclinic C2/c to cubic, origin choice 1, hexagonal, a CIF, one with molecules) agrees site by site with gemmi’s symmetry operations; the CIF cell parameters too.
Supercells contain every unit-cell site exactly once per cell.
save→read stru, CIF and RMCProfile export → import round trips give the same atoms; DISCUS’s CIF read by gemmi gives the same cell; VASP export checked by parsing (metric, counts, coordinates).Known failure (strict xfail): SHELX export/import round trip.
tests/python/test_wp4_fourier.py (16 tests):
fcc extinctions and equal intensities; Debye-Waller factor exp(−B|d*|²/2) to 1e-3; X-ray form factors within 1 % of IT92 (gemmi).
Anomalous dispersion: Friedel pairs in ZnSe differ by the ratio the structure factor predicts (0.5 %); centrosymmetric nickel stays symmetric; without dispersion pairs are equal; tabulated
moa1values (golden).Golden maps for X-rays, neutrons and electrons;
energy=wvle.lotswithseed: repeats exactly with 1 and 4 threads; golden.NUFFT agrees with the standard technique for the 50×50 layer on the
04.sro3grid; known failure (strict xfail) for small crystals.
Findings (details in WP0_Test_Coverage_Gaps.md):
NUFFT gives wrong maps for small crystals on many grids (e.g. Ni 4×4×4, hk0 −3…3: R = 1.7), but is exact for others (10×10 cells, the 50×50 layer, a 9×9 grid over 0…4). The default technique is unaffected. Probably related to the rare Infinity/NaN map in
04.sro3, which uses NUFFT. Needs fixing before WP4 wraps the technique option.SHELX export writes every atom of the cell together with the space-group symmetry (and occupancy 0.5): re-reading CuO gives 32 atoms instead of 8.
disp ondoes not switch on anomalous dispersion; onlydisp anomdoes. Any macro usingdisp onsilently gets no dispersion.Fractions in the legacy atom format are silently misread.
.cellfiles have two atom formats. The comma format with a column header (atoms x, y, z, Biso/co 1./3., 2./3., 0.0, 1.0) runs the values through the interpreter, so1./3.works. The legacy format (bareatoms, values separated by blanks) uses a list-directed read, which stops at/:1./3.becomes 1., without a warning. Mixing the two (bareatomswith commas, or a header with blanks) gives an error.01.Simulation/calculate.celluses the legacy format and gets Co at (0,0,0) instead of (1/3,2/3,0).export.mac: its failures are mostly outdated reference files and typos; two checks pass falsely becausecompare_files.mactreats a missing file as “no differences”.Crystal.atoms()(Python) failed on structures with molecules; fixed.
Checked: pytest tests/python 80 passed, 2 known failures, the same with
OMP_NUM_THREADS 1 and 4.